mjhjmtu

I have a CSV file with vertical bars (|) as the delimiter, like below, for which I need to apply merging technique in Unix. The file contains hundreds of thousands of records (four fields), but I gave only five records for ease of reading.

field1 |field2 | field3 |field4|
1|abc|def|ghi|
4|ijk|
|lmn|
5||opq|rst|
8|
uvw||xyz|
10|hjg|jsh|nbm|

And I want the output result as

field1|field2|field3|field4|
1|abc|def|ghi|
4|ijk||lmn|
5||opq|rst|
8|uvw||xyz|
10|hjg|jsh|nbm|

Can someone help me how to do the same?

I'm assuming you don't want all those blank lines.

$ cat file
1|abc|def|ghi|
4|ijk|
|lmn|
5||opq|rst|
8|
uvw||xyz|
10|hjg|jsh|nbm|

$ awk -F'|' 'while (NF < 5) getline nextline; $0 = $0 nextline1' file
1|abc|def|ghi|
4|ijk||lmn|
5||opq|rst|
8|uvw||xyz|
10|hjg|jsh|nbm|

Update for the question edit: remove whitespace around the field separator

awk -F'[[:blank:]]*[|][[:blank:]]*' -v OFS='|' '
 while (NF < 5) getline nextline; $0 = $0 nextline; $1=$1; print 
' file

With GNU sed:

sed ':loop /(.*|)4.*/ !N; s/n//; b loop; s/ *| */|/g' file

The command dissected:

:loop

The : signals a label that we can use for branches. "loop" is just the name that I chose for the label.

/(.*|)4.*/

Is a line selector regex that matches lines that contain 4 pipe symbols, each allowed to be preceded by zero or more arbitrary characters (.*|), with zero or more arbitrary characters allowed to follow the last pipe.

! ...

Applies the commands in the brackets to any line that did not match the previous regex.

N; s/n//; b loop

N concatenes the current line in pattern space with a newline symbol and the next line from the source file, then s/n// removes the newline symbol and b loop branches back to the label we have defined in the start, so the concatenated line will be compared against the regex again.

Lastly

s/ *| */|/g

will be applied to any line in pattern space before it is output. This removes any spaces around pipe symbols.

If using Vim is an option:

vim -Nesc 'g!/(.*|)4$/j!' -cwq input.txt

• 
  -Nes runs Vim in script mode, making it easier to automate


• 
  -c ... runs Vim commands after opening the file


• 
  g!/(.*|)4$/j! - on every line :g, that doesn't ! match /(.*|)4$/ (a regex matching 4 pipes separated by anything), join the next line to it (:j).


• 
  wq - save and quit.