mjhjmtu

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy.

source.txt contains:

a$$a$$a
b**b**
c%%cc%%
d##d##d##
e^^e^^e^^

target.txt contains:

a$$a$$a
ee$$ee$$
ff__ff__
gg@@gg@@
zzxxzzxx
bb..bb..bb
e^^e^^e^^
hh;;hh;;hh

on this scenario, I assuming that there only 3 entries will be copied to target.txt which are:

b**b**
c%%cc%%
d##d##d##

My test code is:

#!/bin/bash
echo "started"
programpath=/home/mysite/www/copyfiles

var str input ; cat "$programpath/source.txt" > $input 
var str target ; cat "$programpath/target.txt" > $target 

cat $input >> $target

uniq -u "$target"

echo "finished"
 exit 1
fi

Why using bash? The grep command can do the job clean.

grep -Fxvf target.txt source.txt #>> target.txt

This will return those lines which are exist only in source.txt, then you can append these lines to your target.txt with just uncomment #>> target.txt.

You may also needs to unique the source.txt before, to prevent appending duplicated entries if in source.txt file which awk also does the same at next.

grep -Fxvf target.txt <(sort -u source.txt) #>> target.txt

• The -F option is telling grep that match pattern as a string instead of regex.


• With -x option we are telling whole line is my pattern.


• The -v is the reverse match that if you miss it, that will output the lines which are exist in the both files.


• And -f is telling grep read my patterns from a file which is target.txt here.

Or you could use awk instead.

awk 'NR==FNRseen[$0]=1;next !seen[$0]++' target.txt source.txt #>> target.txt

• Add whole target.txt file into the array called seen with the key of whole line seen[$0], and do next to read next line.




• With !seen[$0]++ we are looking a line from source.txt which is not exist in array, then print it. Also add source.txt file lines into the array to prevent printing duplicated lines if exist in source.txt_.