mjhjmtu

I want to remove four lines of a fastq file. For example, usually the file looks like this: (for each sample correspond four lines)

@M04241:303:000000000-BR896:1:1102:21438:12389 1:N:0:TATGGCAC
TGTCAGCCGCCGCGGTAATACGGAGGGTCCGAGCGTTATCCGGAATTATTGGGTTTAAAGGGTCCGCAGGCGGGCTTATAAGTCAGGGGTGGAATGGTGCGGCTCAACCGTAGCACTGCCCTTGATACTGTTAGTCTTGAGTTATGGTGGAGTGGCCGGAATATGTAGTGTAGCGGTGAAATGCATAGATATTACATAGAACACCGATCGCGAAGGCAGGTCACTAACCATTTGACTGACGCTGATGGACGAAAGCGTGGGGAGCGAACAGGATTAGATACCCTGGTAGTCCACGCCGGAAACGATGGATACTAGCTGTCGGGCACTTGTGCTCGGCGGCCAAGCGAAAGTGATAAGTATCCCACCTGGGGAGTACGTGCGCAAGAATGAAACTCAAATGAATTGACGG
+
EGGGGGGGGGGGGGGGGGGGGGGGDE@FFGEEEGGGGDGFEFGGGGGGGGGGGGGGGGGGGGGGGDGEFFGGGCGGFDF<DGGFGGGGGGGG7FFG?FDF:FGGGFCGGGGFEGGGF:>GGGG>F>DE@GG6@GGG@G9<EGGGG9FGGGGGG7FGGDDEFGGGGGGGGGGGGGGGGCEFGGGGFG?EFFCFGGGGGGFGG?GGGGGGGG=EGEGGGGGGGGGGGFGCGGFGGGGCFFF6CD7DDFFFFFED9:BFCBEE@DEF:@EGCFCF@FFFD?=A:CFEF0<C<A>FB>@6+C,@GFFGFDGGF<AFEFB+FEECGFF9FDFAC6@+:@FC:GFC,CFC,EFGE,9FFCGFF<@;6:,FD,D:FGGFFGF7@8+7,,CF<<6CF<CC-CA@<GEGFE@6@A,CB
@M04241:303:000000000-BR896:1:1103:11464:7575 1:N:0:TATGGCAC
GTCAATTTCTTTGCGTTTCAATCTTGCGATCGTACTCCCCAGGTGGGATACTTATCACTTTCGCTTAGTCACTGAGATAAATCCCAACAACTAGTGTCCATCGTTTACGGCGTGGACTACCAGGGTATCTAATCCTGTTCGCTCCCCACGCTTTCGTCCATCAGCGTCAGTATATGGTTAGTGACCTGCCTTCGCGATCGGTGTTCTATGTAATATCTATGCATTTCACCGCTACACTACATATTCCGGCCACTCCACCATAACTCAAGACTAACAGTATCAAAGGCAGTGCTACGGTTGAGCCGCACCATTTCACCCCTGACTTATCAGCCCGCCTGCGGACCCTTTAAACCCAATAATTCCGGATAACGCTCGGACCCTCCGTATTACCGCGGCTGCTGGC
+
CCCCCGGGGGGGG-FCFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGFFGGDFFGFGFGGGGGGGGGGGGGGGGGGGGGGGGGEGGEGGGGDGGG4FFGGGGGGGGGGGGGGGGGGGGGEGGGGGGFGGGFFGGGGGGGGGGGGGGGFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFFFGFFFGFGGGGGGGGGGGGGGGGGGGFGGFFGGGGGGGGGGGGGGGGGGGCDGGGGGGGGFCFGGGFGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGGGGGGGGGFGGGGGCGEFFGGEGGGGGGGGGGGGGGGGGDGGGGFFCGGGGGGGGGGGGFGGGDGGGGGGGGGGGGFGGGGGGGGGGGGGGGGG
@M04241:303:000000000-BR896:1:1103:23291:21403 1:N:0:TATGGCAC
CTGCGGCACCGCAGGGCAAGCCCCCCGACGCCTAGCCCACATCGTTTAGGGCGTGGACTACCAGGGTATCTAATCCTGTTTGCTCCCCACGCTTTCGCGCCTCAGCGTCAGTGCCGGACCAGAGAGCCGCTTTCGCCACCGGTGTTCCACCCAATATCTACGAATTTCACCTCTACACTGGGTATTCCACCCTCCTCTTCCGGACTCGAGCACCGCAGTCTCGGCTGCACCTCCGGGGTTGAGCCCCGGGCTTTCACAGCCGACTTGCGACGCCGCCTACGCGCCCTTTACGCCCAGTGATTCCGAACAACGCTAGCACCCTCCGTCTTACCGCGGCGGCTGAC
+
CCCCCGGGGGG>FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@@FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

But I found one sample that have empty two of the four lines empty like this:

@M04241:303:000000000-BR896:1:1103:11464:7575 1:N:0:TATGGCAC

+

@M04241:303:000000000-BR896:1:1103:23291:21403 1:N:0:TATGGCAC
CTGCGGCACCGCAGGGCAAGCCCCCCGACGCCTAGCCCACATCGTTTAGGGCGTGGACTACCAGGGTATCTAATCCTGTTTGCTCCCCACGCTTTCGCGCCTCAGCGTCAGTGCCGGACCAGAGAGCCGCTTTCGCCACCGGTGTTCCACCCAATATCTACGAATTTCACCTCTACACTGGGTATTCCACCCTCCTCTTCCGGACTCGAGCACCGCAGTCTCGGCTGCACCTCCGGGGTTGAGCCCCGGGCTTTCACAGCCGACTTGCGACGCCGCCTACGCGCCCTTTACGCCCAGTGATTCCGAACAACGCTAGCACCCTCCGTCTTACCGCGGCGGCTGAC
+
CCCCCGGGGGG>FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@@FGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
@M04241:303:000000000-BR896:1:1103:26180:21941 1:N:0:TATGGCAC
CCGCCAATTTCTTTGAGTTTCAGCCTTGCGACCATACTCCCCAGGCGGGGTACTTAACACTTTTGATTCGGCAGTGCACCCATGTTAGTCCACTACCTAGTACCCATCGTTTAGGGCTAGGACTACCGGGGTATCTAATCCCGTTCGCTACCCTAGCTTTCGCGCCTCAGCGTCAGAAGAGGTCCAGCACGTCGCTTTCGCCACCGGCGTTCCTTCCGATCTCTACGCATTTCACCGCTCCACCGGAAGTTCCACATGCCCCTACCTCCCTCGAGATTGGCAGTTTCGAAGGCAGTTCTACAGTTGAGCTGCAGGATTTCACCTCCGACTGACCTATCCGCCTACGCGCCCTTTAAGCCCAGTGATTCCGAACAACGTTCGC
+
CCCCCGEGGGGGGGGGGEGGGGGGGGGGDFGGGGGGGGGGGGGEGGGGGGEFGGGFFFFGGGGGG,CEFGGGGGGGGGG?GGGGGG9FFGGGGGGGCGGGGGGGGGCFGGGG@GGGGGFGGGGGGGGGCGGFGGGGGGGGGGGGGGGGGGGGGGGFFGGGGGGGGGDEGGGGGGGDGGGGFGFFGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGEGGGDGEFGGEEGGGGFGGGGGGGGGGGGGGGGGGGGGEF?GGGEGGEEFEFFDFFGFGGFGGGGGGFFFGFGGGGGGGGGFGGGGFCGGGGGGGGGFFGGGGGGGGGGGGGGGFF@7GGGGGGGGGGGGGGGFDFCGGGGFEFGGFGGGGGGGGFGFEGGGG
@M04241:303:000000000-BR896:1:1102:21438:12389 1:N:0:TATGGCAC
TGTCAGCCGCCGCGGTAATACGGAGGGTCCGAGCGTTATCCGGAATTATTGGGTTTAAAGGGTCCGCAGGCGGGCTTATAAGTCAGGGGTGGAATGGTGCGGCTCAACCGTAGCACTGCCCTTGATACTGTTAGTCTTGAGTTATGGTGGAGTGGCCGGAATATGTAGTGTAGCGGTGAAATGCATAGATATTACATAGAACACCGATCGCGAAGGCAGGTCACTAACCATTTGACTGACGCTGATGGACGAAAGCGTGGGGAGCGAACAGGATTAGATACCCTGGTAGTCCACGCCGGAAACGATGGATACTAGCTGTCGGGCACTTGTGCTCGGCGGCCAAGCGAAAGTGATAAGTATCCCACCTGGGGAGTACGTGCGCAAGAATGAAACTCAAATGAATTGACGG
+
EGGGGGGGGGGGGGGGGGGGGGGGDE@FFGEEEGGGGDGFEFGGGGGGGGGGGGGGGGGGGGGGGDGEFFGGGCGGFDF<DGGFGGGGGGGG7FFG?FDF:FGGGFCGGGGFEGGGF:>GGGG>F>DE@GG6@GGG@G9<EGGGG9FGGGGGG7FGGDDEFGGGGGGGGGGGGGGGGCEFGGGGFG?EFFCFGGGGGGFGG?GGGGGGGG=EGEGGGGGGGGGGGFGCGGFGGGGCFFF6CD7DDFFFFFED9:BFCBEE@DEF:@EGCFCF@FFFD?=A:CFEF0<C<A>FB>@6+C,@GFFGFDGGF<AFEFB+FEECGFF9FDFAC6@+:@FC:GFC,CFC,EFGE,9FFCGFF<@;6:,FD,D:FGGFFGF7@8+7,,CF<<6CF<CC-CA@<GEGFE@6@A,CB

How can I detect this empty lines and remove from a fastq file? I know the number of the line, however it is a huge file I cannot open normally so I need a command to detect that the two lines are empty and to remove the four lines associated to that sample.

Thanks!!

sed 'N;N;N;/nn/d' file.fastq >new-file.fastq

This would read in the four lines of a FastQ record and then check whether it contains two consecutive newline characters. If it does, the whole record is ignored. If it doesn't, it is printed. This would repeat for all entries in the file. All printed records goes to a new file (here new-file.fastq).

The sed script, with annotations:

 # (implicit: read a line)
N; # read a second line, append it to the pattern space with embedded n in-between
N; # read a third line
N; # read a fourth line
/nn/d # if there are two consecutive newlines, delete and continue from top
 # (implicit: print)

Comment from a colleague:

Fastq records are often paired, and software tends to throw a fit when pair mates are not found without explicitly telling it mates are missing. Several tools have a minimum length option like trimmomatic that will keep pairing, and separate orphaned records.

This means that if the reads in your file are paired, and one of the pairs is empty, you would mess up the pairing by just removing the empty record.

Removing the mate of the empty read would be a lot more complicated, unless using already existing bioinformatics tools. With the tools in the standard Unix tool chest, it would probably involve saving the empty reads to a separate file, and then, by using their FastQ headers, scan for and delete the corresponding mates.

The data shown in the question seems to only be unpaired reads though.