mjhjmtu

I have a big file containing 27 columns and nearly 6 million rows. The following is a little example of my file

head data
0.65 0.722222 1.0 0.75 0
0.35 0.277778 0.0 0.25 0
0 0.666667 0.75 0.5 0.5625
0 0.333333 0.25 0.5 0.4375 

Rows are samples, and I have "2 rows per sample" (one for observation "a", and the other observation "b"). In the example above I showed data for 2 samples (rows 1 and 2 correspond to sample1 and rows 3 and 4 correspond to sample 2). I want to check if for each sample, both observations are 0, replace them with 9.
this is my desired output:

head desired
0.65 0.722222 1.0 0.75 9
0.35 0.277778 0.0 0.25 9
9 0.666667 0.75 0.5 0.5625
9 0.333333 0.25 0.5 0.4375 

Any perl or python or bash (if reliable for such a big file )solution how to do it?
In the past I was just splitting the file for each sample and running the following code for each file

awk 'NR==1 split($0,a);next; NR==2 split($0,b);for(i=1;i<= NF;i++) printf("%s%s",(i==1?"":"t"),a[i]==0 && b[i]==0?9:a[i]);
printf("n");;for(i=1;i<= NF;i++) printf("%s%s",(i==1?"":"t"),a[i]==0 && b[i]==0?9:b[i]);printf("n"); ' 

but now I want to do it for the entire file, do not want to split it.

Thanks.

Thanks.

Here is how I would do this in Python:

#!/usr/bin/env python3

firstLineZero = False

# Open the file for reading
with open("biodata2", "r") as inFile:
 for line in inFile:
 # Check if last value in line is 0
 if not firstLineZero and line.split()[-1] == "0":
 # Save this line, and set a boolean
 firstLineZero = True
 prevLine = line
 elif firstLineZero and line.split()[-1] == "0":
 # Now we know that both lines end with 0.
 # Change the final value to 9 in both lines...
 prevLineSplit = prevLine.split()
 thisLineSplit = line.split()
 prevLineSplit[-1] = "9" 
 thisLineSplit[-1] = "9" 
 prevLine = "t".join(prevLineSplit)
 thisLine = "t".join(thisLineSplit)
 print(prevLine)
 print(thisLine)
 # Reset boolean
 firstLineZero = False
 # Reset prevLine
 prevLine = ""
 else:
 print(line, end="")

# If we have a 'trailing' saved line, print that
if prevLine is not None:
 print(prevLine, end="")

Execution example, with a couple of more lines to provide a POC.

Data:

cat biodata2 
0.65 0.722222 1.0 0.75 0
0.35 0.277778 0.0 0.25 0
0 0.666667 0.75 0.5 0.5625
0 0.333333 0.25 0.5 0.4375
0 0.333333 0.25 0.5 1
0 0.333333 0.25 0.5 0

Execution:

./readBioData.py
0.65 0.722222 1.0 0.75 9
0.35 0.277778 0.0 0.25 9
0 0.666667 0.75 0.5 0.5625
0 0.333333 0.25 0.5 0.4375
0 0.333333 0.25 0.5 1
0 0.333333 0.25 0.5 0

Obviously, if you want to save this to a file instead of printing to stdout, you will have to alter the print statements into write and set a file for writing.

Like so:

#!/usr/bin/env python3

firstLineZero = False
outFile = open("bioDataOut.txt", "w")

# Open the file for reading
with open("biodata2", "r") as inFile:
 for line in inFile:
 # Check if last value in line is 0
 if not firstLineZero and line.split()[-1] == "0":
 # Save this line, and set a boolean
 firstLineZero = True
 prevLine = line
 elif firstLineZero and line.split()[-1] == "0":
 # Now we know that both lines end with 0.
 # Change the final value to 9 in both lines...
 prevLineSplit = prevLine.split()
 thisLineSplit = line.split()
 prevLineSplit[-1] = "9" 
 thisLineSplit[-1] = "9" 
 prevLine = "t".join(prevLineSplit)
 thisLine = "t".join(thisLineSplit)
 outFile.write(prevLine + "n")
 outFile.write(thisLine + "n")
 # Reset boolean
 firstLineZero = False
 # Reset prevLine
 prevLine = ""
 else:
 outFile.write(line)

# If we have a 'trailing' saved line, print that
if prevLine is not None:
 outFile.write(prevLine)

outFile.close()

Then you can do:

./readBioDataSaveToFile.py
cat bioDataOut.txt 
0.65 0.722222 1.0 0.75 9
0.35 0.277778 0.0 0.25 9
0 0.666667 0.75 0.5 0.5625
0 0.333333 0.25 0.5 0.4375
0 0.333333 0.25 0.5 1
0 0.333333 0.25 0.5 0

The trick to handle pairs of lines is to merge them:

paste - - < paired_file

Then you can test/manipulate the fields with awk ($1==0 && $6==0, etc.)