mjhjmtu

I'm trying to make a script that looks through each line of one file and if a line fails to match anywhere in any line of another text file then remove that line from the original file.

An example of and input and output desired from this script would be:

example input: file 1 (groups file),

hello
hi hello
hi
great
interesting

 file 2: 
this is a hi you see
this is great don't ya think
sometimes hello is a good expansion of its more commonly used shortening hi
interesting how brilliant coding can be just wish i could get the hang of it

Example script output - file 1 changed to:

hello
hi
great
interesting

So its removed hi hello, because its not present in the second file

here is the script, it seems to work to the point of making the variables.

#take first line from stability.contigs.groups
echo | head -n1 ~/test_folder/stability.contigs.groups > ~/test_folder/ErrorFix.txt
#remove the last 5 character
sed -i -r '$ s/.5$//' ~/test_folder/ErrorFix.txt 

#find match of the word string in errorfix.txt in stability.trim.contigs.fasta if not found then delete the line containing the string in stability.contigs.groups
STRING=$(cat ~/test_folder/MothurErrorFix.txt)
FILE=~/test_folder/stability.trim.contigs.fasta
if [ ! -z $(grep "$STRING" "$FILE") ]
 then
 perl -e 's/.*$VARs*n//' ~/test_folder/stability.contigs.groups
fi

If you have gnu grep you could run:

grep -oFf file1 file2 | sort | uniq | grep -Ff - file1

remove the last grep if don't need to preserve the order of the lines in file1.

If you don't have access to gnu grep, with awk:

awk 'NR==FNRz[$0]++;next;for (l in z)if (index($0, l)) y[l]++
ENDfor (i in y) print i' file1 file2

Go for don_crissti's (accepted) answer if you have GNU grep. Just in case you don't (e.g. on a standard Mac OS X, where that won't work), you could alternatively save this snippet to a bash script, e.g. myconvert.sh

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
 if ! grep -Fq "$line" $2
 then
 sed -i '' "/$(echo $line | sed -e 's//$*.^|/\&/g')/d" $1
 fi
done < "$1"

an call it with the two files as arguments

./myconvert.sh file1 file2

However, please note don_crissti's knowledgeable comments below regarding the usage of while/read and the obvious performance drawbacks of invoking sed.