mjhjmtu

I have a very large dataset which is supposed to consist of emails. However, there are a large amount of invalid emails that need to be removed from the file completely.

Here are some examples:

89 is @msn .com
89!3@nomail.com
89%@yahoo.com
89%azn@yahoo.com
89&#39:s@msn.com
89'Mustang@yahoo.com
89's@msn.com
89&main@yahoo.com
89+475asdjkl:jkl@aol.com
89+475asdjkl;jkl@aol.com
89+ggg@hotmail.com

Is there a simple approach available to remove lines which contain invalid emails from the file?

EDIT: As pointed out by @Ivanivan, we could just use this regex in a grep instead of scripting anything:

grep "^[a-z0-9!#$%&'*+/=?^_`~-]+(.[a-z0-9!#$%&'*+/=?^_`~-]+)*@([a-z0-9]([a-z0-9-]*[a-z0-9])?.)+[a-z0-9]([a-z0-9-]*[a-z0-9])?$" my_email_list.txt >> my_valid_emails.txt

A simple script can sort this for you. As commented above by @ilkkachu and @Mark Plotnick, some of those examples are perfectly valid email addresses.

email_validate.sh:

#!/bin/bash

# email regex check
email_valid="^[a-z0-9!#$%&'*+/=?^_`~-]+(.[a-z0-9!#$%&'*+/=?^_`~-]+)*@([a-z0-9]([a-z0-9-]*[a-z0-9])?.)+[a-z0-9]([a-z0-9-]*[a-z0-9])?$"

# set field separator to new lines
IFS=$'n' 
# for loop checking line against regex above
for line in $(cat my_email_list.txt); do
 if [[ $line =~ $email_valid ]]; then
 echo "$line is valid"
 else
 echo "$line is invalid"
 fi
done

example output:

┌─[root@Fedora]─[~]─[03:27 pm]
└─[$]› ./email_validate.sh
89 is @msn .com is invalid
89!3@nomail.com is valid
89%@yahoo.com is valid
89%azn@yahoo.com is valid
89&#39:s@msn.com is invalid
89'Mustang@yahoo.com is invalid
89's@msn.com is invalid
89&main@yahoo.com is valid
89+475asdjkl:jkl@aol.com is invalid
89+475asdjkl;jkl@aol.com is invalid
89+ggg@hotmail.com is valid

if you need them deleting from the file as it runs through, just add a sed '/$line/d' to the if statement. Though I would personally recommend moving valid emails to a new file instead, in case you need to refer to the old

 if [[ $line =~ $email_valid ]]; then
 echo "$line is valid"
 echo "$line" >> my_valid_emails.txt
 else
 echo "$line is invalid - deleting"
 fi

Which will return something like this:

┌─[root@Fedora]─[~]─[03:34 pm]
└─[$]› cat my_valid_emails.txt
89!3@nomail.com
89%@yahoo.com
89%azn@yahoo.com
89&main@yahoo.com
89+ggg@hotmail.com

hope that helps.