mjhjmtu

This is an SOS question. My professor asked me to obtain output from a long-running simulation code bequeathed to us by a former post-doc (who had explained to me its workings).

I had done a few small-scale trial runs and everything went fine. Then I started the full simulation about a month back and has been running continuously since then. But just a few minutes ago, due to some memory issues, the program crashed before it could write the formatted tabular output to disk.

Luckily, I had enabled terminal echoing of the intermediate results, and had set my scroll-back history to a large value. I managed to salvage partial output by entering scroll-back mode and copying the entire terminal dump into a text file (and also made backup copies of it).

Now, this terminal output is quite verbose (intentionally set so for debug purposes). The following is a snapshot from the salvaged terminal output text file (let us call it terminal_output.txt)

1 Linear search iteration no. 1 begins: Attempting to blah blah with 1 ...
2 blah blah 
3 blah 
4 blah blah blah
5 lorem ipsum 
.........
........
75 Success with 128 blah ....
76 blah blah
77 blah blah
78 result_flag: 1, exit_reason: 6
79 blah
80 Completed optimal computation with T_init = 25.00 degC & T_sink = 35.00 degC

And then this exact pattern repeats. e.g.,

81 Linear search iteration no. 2 begins: Attempting to blah blah with 1 ...
82 blah
......
95 Success with 307 blah ....
......
......
100 Completed optimal computation with T_init = 30.00 degC & T_sink = 40.00 degC

My requirement is to extract the following information to produce a tabular output like:

25 35 128
30 40 307
...........
...........

i.e. the 1st & 2nd columns are from the numerical values corresponding to T_init and T_sink respectively, from those lines begin with Completed. The 3rd column is the numerical value from the line that begins with Success (which is always 5 rows before Completed if that helps). Any separator between the columns is acceptable-be it spaces, tabs or comma.

I wish to do this natively using standard *nix utilities such as grep, sed and awk or even vi/vim. Either piped one-liners strung together or bash scripts is fine. If necessary, I am open to using python, perl or other scripting languages also.

It's essentially a matter of capturing the parts you want, and discarding the parts you don't. For example using sed, you could capture the integer Success value and copy it to the hold space (h), retrieving and appending it (G) to the captured digits of the Completed line:

sed -nE 
 -e '/Success/ s/.* ([0-9]+).*/1/; h;' 
 -e '/Completed/G; s/.*T_init = ([0-9]+).00 degC & T_sink = ([0-9]+).*n/1 2 /; p;
' terminal_output.txt

Perl provides a somewhat more expressive syntax, which IMHO is more readable:

perl -lne '
 our $a = $1 if /Success.*?(d+)/; print join " ", /(d+).d+/g, $a if /Completed/
' terminal_output.txt

produces the desired output

25 35 128
30 40 307

POSIX-compatible sed:

grep -e 'Success' -e 'Completed' your_file | sed 'N;s/Success with ([[:digit:]]+).*T_init = ([^[:space:]]+).*T_sink = ([^[:space:]]+).*/2 3 1/;s/.00//g'

GNU sed: (in which . does not match n at least in 4.2.2 on CentOS)

grep -e 'Success' -e 'Completed' your_file | sed 'N;s/Success with ([[:digit:]]+).*n.*T_init = ([^[:space:]]+).*T_sink = ([^[:space:]]+).*/2 3 1/;s/.00//g'

Grabs lines containing Success and Completed, then operating on two lines (way more explicitly than necessary) pulls the three fields you care about and orders them into one line.

This only truncates .00 off of any numbers, leaving any significant fractional component alone (including something like 12.20, there would still be the single trailing zero).

Caveat where it won't work if some of those ... lines contain Completed or Success

A quick awk command should get you started:

awk '$2 ~ /Success/a=$4;next; $2 ~ /Completed/b=$8;c=$13;print a,b,c' terminal_output.txt

This wont work if you have multiple Success lines before a Completed line etc.