mjhjmtu

I have a big file to which I am doing various operations, and this error just came up. I tried googling it but there didn't find any result with this.

sed: regex input buffer length larger than INT_MAX

My purpose is to quote every line, appending a comma,
and subsequently enclose the entirety of the file with square brackets
(as a single line). 
For example, an input of

The quick brown fox
jumps over
the lazy dog.

should yield a result of

["The quick brown fox","jumps over","the lazy dog.",]

Assume that the input file doesn’t contain any quotes.

The code I run is this:

cat "$FILE" | sed -e 's/.*/"&",/' | sponge "$FILE"

truncate --size=-1 "$FILE"

cat "$FILE" | sed -z 's/.*/[&]/' | tr --delete 'n' | sponge "$FILE"

sed version:

sed --version
sed (GNU sed) 4.5

Any thoughts?

Your question is strange. 
You say “… this error just came up. 
I tried googling it but there didn't find any result with this.”,
making it sound like you have no idea what’s happening. 
But you do understand it, don’t you? 
When you say sed -z, you’re telling sed to read the input,
treating NUL as record (line) separators instead of newline. 
But text files typically don’t have NUL characters in them,
so, in practical terms,
this means that you want sed to read the entire file
and treat it as one line. 
You obviously understand this; your 's/.*/[&]/' command,
to “enclose the entirety of the file with square brackets”,
doesn’t make sense unless you expect the entirety of the file
to be treated as a single line.

So why are you so surprised that your big file
is too big to be handled as a single line?

You say that your script works sometimes —
presumably when the size of the file
is below the maximum line size permitted by sed. 
This script should do the same thing, regardless of the size of the file:

cat "$FILE" | sed -e 's/.*/"&",/' -e '1s/^/[/' -e '$s/$/]/' | tr --delete 'n'

Of course this can still choke if any individual line in the input
is absurdly long.

Notes:

• You don’t need the and ; "$FILE" is fine.


• Following the suggestion made by steeldriver,
  this inserts a [ at the beginning of the first line
  and appends a ] at the end of the last line.


• I left off the sponge for illustration purposes. 
  Overwriting your input file may be operationally necessary,
  but it’s a bad thing to do while you’re still debugging. 
  Add the sponge command back when you’re sure it’s doing what you want.

This duplicates your script, so an input of

The quick brown fox
jumps over
the lazy dog.

will yield a result of

["The quick brown fox","jumps over","the lazy dog.",]

with an extra comma before the ]. 
If that’s really what you want, OK, that’s fine with me. 
If you don’t want the comma at the end, do

cat "$FILE" | sed -e 's/.*/"&",/' -e '1s/^/[/' -e '$s/,$/]/' | tr --delete 'n'

where the '$s/,$/]/' command
removes the comma at the end of the file when it appends the ].

Note also that all of the commands discussed so far
will leave you with a file with no newline characters,
not even one at the end. 
This is a malformed text file,
and some commands my fail to process it properly. 
If that’s really what you want, OK, that’s fine with me. 
Otherwise, add

echo >> "$FILE"

or

printf 'n' >> "$FILE"

at the end of your script.

If you didn't require sed, awk can do this, IMHO a bit more clearly:

Edit: original method (fixed by G-Man, tnx), which I based on looking at the sample output in the Q WITH comma after last string:

 awk <"$FILE" -vORS= -vq=" 'BEGINprint "[" print q $0 q "," ENDprint "]n"' | sponge "$FILE"

• as G-Man said, leave off the sponge part for debugging


• if you don't want the newline at the end, leave out the n
  

Add: modified method, based on the request to remove the last comma before adding the brackets:

 awk <"$FILE" -vORS= -vq=" 'BEGINprint "[" print sep q $0 q; sep="," ENDprint "]n"' | sponge "$FILE"

(In awk an uninitialized variable in string context is guaranteed to yield an empty string, but if you prefer to be explicit add -vsep= to the options or ;sep="" to the BEGIN block to initialize it.)