mjhjmtu

I have two files that look like this:

file1 (unique IDs):

 C84610112
 C96209347
 C84774620
 C84774691
 C85594749
 C89372772
 C89651687
 C89845500
 C89914896
 C91269765
 C91526663
 C92210411
 C92254517
 C93709504
 C94303303
 C95100561
 C95100609
 C95417520
 C95696352
 C96045246
 C96045496
 C96060727
 C96076986

and file2:

 1 C95696352 score: -69.785 nathvy = 38 nconfs = 888
 2 C98230482 score: -57.431 nathvy = 47 nconfs = 575
 3 C96209347 score: -57.128 nathvy = 24 nconfs = 1188
 4 C36510773 score: -56.502 nathvy = 38 nconfs = 7595
 5 C04355288 score: -56.400 nathvy = 41 nconfs = 50502
 6 C89372772 score: -55.728 nathvy = 22 nconfs = 3228
 7 C96209347 score: -54.713 nathvy = 24 nconfs = 162
 8 C96209347 score: -53.901 nathvy = 24 nconfs = 159
 9 C06169346 score: -53.438 nathvy = 22 nconfs = 105
 10 C95696352 score: -52.848 nathvy = 38 nconfs = 878
 11 C98216318 score: -52.061 nathvy = 52 nconfs = 1092
 12 C04285713 score: -52.009 nathvy = 38 nconfs = 1355
 13 C96209347 score: -51.477 nathvy = 24 nconfs = 1375
 14 C98222837 score: -50.730 nathvy = 34 nconfs = 588
 15 C98216318 score: -50.694 nathvy = 52 nconfs = 1136
 16 C32832068 score: -50.546 nathvy = 22 nconfs = 548
 17 C95696352 score: -50.475 nathvy = 38 nconfs = 3220
 18 C32832068 score: -50.457 nathvy = 22 nconfs = 16235
 19 C95696352 score: -50.234 nathvy = 38 nconfs = 3048
 20 C85594749 score: -49.780 nathvy = 44 nconfs = 4536
 21 C72332782 score: -49.676 nathvy = 41 nconfs = 3942
 22 C97970648 score: -49.616 nathvy = 45 nconfs = 17640
 23 C04285713 score: -49.594 nathvy = 38 nconfs = 14038
 24 C98043133 score: -49.370 nathvy = 43 nconfs = 1236
 25 C89372772 score: -49.308 nathvy = 22 nconfs = 471
 26 C97970648 score: -49.297 nathvy = 45 nconfs = 17850
 27 C85594749 score: -49.122 nathvy = 44 nconfs = 4158
 28 C70006381 score: -49.092 nathvy = 24 nconfs = 880

I would like to match IDs from file1 with IDs in file2 (second column) and for those that are matching to print them. Also, in file2 some IDs are repeating, such as C96209347 (although whole lines are not identical). I would like to grep those lines that are appearing for the first time only and others to skip. So in this specific example with C96209347 only third line from file2 should be printed. Anybody can help?

Try this,

grep -f file1 file2 | awk '!_[$2]++'

 1 C95696352 score: -69.785 nathvy = 38 nconfs = 888
 3 C96209347 score: -57.128 nathvy = 24 nconfs = 1188
 6 C89372772 score: -55.728 nathvy = 22 nconfs = 3228
20 C85594749 score: -49.780 nathvy = 44 nconfs = 4536

Explanation

• 
  grep -f file1 file2: search in file2 for matches of patterns obtained from file1


• 
  awk '!_[$2]++': Don't print anything if field $2 has been seen before (via)
  
  
  • 
    _ is the array name (can be anything, e.g. "seen")
  
  
  • 
    _[$2]++ will create an array entry with the key being the content of field $2 and add 1
  
  
  • If _[$2] was not (!) already set, print the line. The printcommand is the default action that is made by awk when the condition matches.
  
  

With awk alone:

$ awk 'NR==FNR a[$1]=1; next $2 in a print; delete a[$2]' file1 file2
 1 C95696352 score: -69.785 nathvy = 38 nconfs = 888
 3 C96209347 score: -57.128 nathvy = 24 nconfs = 1188
 6 C89372772 score: -55.728 nathvy = 22 nconfs = 3228
 20 C85594749 score: -49.780 nathvy = 44 nconfs = 4536