mjhjmtu

First set of example data:

$ awk -vOFS='t' ' r=""; delete t; for (i=1;i<=NF;++i) if (!t[$i]++) r = r ? r OFS $i : $i print r ' file
A B1 C1
B B2 D2
C C12 C13
D D3 D5 D9
G F2

Second set of example data (same awk script):

$ awk -vOFS='t' ' r=""; delete t; for (i=1;i<=NF;++i) if (!t[$i]++) r = r ? r OFS $i : $i print r ' file
A CD274 PDCD1LG2 CD276
B NEK2 NEK6 NEK10 NEKL-4
C TNFAIP3 OTUD7B
D DUSP16 DUSP4 DUSP8 VHP-1
E AGO2

The script reads the input file file line by line, and for each line it goes through each field, building up the output line, r. If the value in a field has already been added to the output line (determined by a lookup table, t, of used field values), then the field is ignored, otherwise it's added.

When all the fields of an input line have been processed, the constructed line is outputted.

The output field delimiter is set to tab through -vOFS='t' on the command line.

The awk script unravelled:

 r = ""
 delete t

 for (i = 1; i <= NF; ++i) 
 if (!t[$i]++) 
 r = r ? r OFS $i : $i
 
 

 print r

sed/tr, uniq and paste

while read -r l; do sed 's/t/n/g' <<< "$l" | uniq | paste -s; done < test

or POSIX compliant:

while read -r l; do echo "$l" | tr 't' 'n' | uniq | paste -s -; done < test

For the file test this will line by line replace all Tab characters with linebreaks, run uniq to delete dupes and replace the linebreaks with Tab characters again.

$ cat test
A B1 B1 C1
B B2 D2
C C12 C13 C13
D D3 D5 D9
G F2 F2

$ while read -r l; do sed 's/t/n/g' <<< "$l" | uniq | paste -s; done < test
A B1 C1
B B2 D2
C C12 C13
D D3 D5 D9
G F2

NB: This solution will not work for duplicates over multiple rows, e.g. C1 in

A B1 B1 C1
C1 B B2 D2

Maybe something like:

gawk -vRS='\s*\S*' -vORS= '$0=RT;$1!=prev;prev=$1'

The RS=pattern...$0=RT trick lets you process records defined as the parts that match the pattern.

So here, we're slicing the input into <whitespace><non-whitespace> $0 records, <non-whitespace> goes in $1 (the first and only field). We're printing the records whose $1 is not equal to the previous one.

On an input like:

A B1 B1 C1
B B2 D2 
C C12 C13 C13
D D3 D5 D9
G F2 F2

The records are:

[A][ B1][ B1][ C1][
B][ B2][ D2][ 
C][ C12][ C13][ C13][
D][ D3][ D5][ D9][
G][ F2][ F2][
]

Doesn't work for your second example though and note that it could remove some newline characters.

This is more of a code-golf / freak challenge solution:

xargs -L1 -I echo '; ' < ./test.txt | 
 xargs -n1 | 
 uniq | 
 xargs | 
 sed -e 's/; /n/g' -e 's/ +/t/g'

But it avoids using loops and all other heavy machinery seen in other answers.

It also builds on an assumption your data doesn't contain ; character.

With perl:

unique words on each line:

perl -MList::Util=uniq -lape '$_ = join "t", uniq @F'

unique words globally:

perl -lape '$_ = join "t", grep !$count$_++ @F'

Or to only consider words of each line starting with the 2^nd one:

perl -lape '$_ = join "t", shift(@F), grep !$count$_++ @F'

With bash v4.3 (if you don't mind the order of fields as it's sorted except first)

while IFS='n' read -r line; 
 do aline=( $line );
 echo $aline[0] $(sort -u <(printf "%sn" $aline[@]:1));
done < infile

Explanation:

• 
  aline=( $line ) this make the line save into an array 'aline'


• 
  $aline[0] prints first element of an array 'aline' (array index is starting with zero in bash)


• 
  printf "%sn" $aline[@]:1 prints each element of array 'aline' in separate lines and ignore first element; Then


• 
  sort -u sorts each line and remove duplicates entries


• 
  

  echo this also combine splited line elements after sort into one linear.

  
  
  

  Please see below example to have better view of this step:

  
  
  

  printf "Cn4nBnC" |sort -u 
  4
  B
  C
  echo $(printf "Cn4nBnC" |sort -u)
  4 B C
  

  
  

This will give output as:

A CD274 CD276 PDCD1LG2
B NEK10 NEK2 NEK6 NEKL-4
C OTUD7B TNFAIP3
D DUSP16 DUSP4 DUSP8 VHP-1
E AGO2

sed substitution with back reference

sed -re 's/s+$//; s/(t[^t]+)1+$/1/'

(s/s+$// gets rid of trailing white-space like in your example.)