mjhjmtu

If I have a lot of commands I've issued and I want the exit status of a process that exited say 100 commands ago. Is there a struct, file location, or variable in linux I can access all of exited processes and see information about them?

If BSD process accounting was enabled (accton on was issued), with GNU acct 6.6.3 or above, you can get that information from lastcomm --debug or dump-acct /var/log/account/pact (or wherever the process accounting data is stored on your system).

$ perl -e 'exit 123'
$ lastcomm --debug | grep perl
CURRENT REC: perl |v3| 0.00| 0.00| 0.00| 1000| 1000| 26328.00| 0.00| 332| 8530| | 123|pts/1 |Wed Sep 19 20:21:26 2018
$ dump-acct /var/log/account/pacct | grep perl
perl |v3| 0.00| 0.00| 0.00| 1000| 1000| 26328.00| 0.00| 332| 8530| | 123|pts/1 |Wed Sep 19 20:21:26 2018

You get the exit code in the 3rd last field and whether it was killed or not (but not the signal number, see @mosvy's answer for that) in the 4th last one.

If you're on debian, you can install the acct package in order to enable process accounting, but notice that neither lastcomm --debug nor dump-acct show anything like the exit status or the signal that terminated a process.

If order to get that data, you can use a script like this:

$ cat pacct.pl
#! /usr/bin/perl
use strict;
use Config;
printf "%-7s %6s %6s %8s %8s %sn",
 'STATUS', 'UID', 'PID', 'BTIME', 'ETIME', 'COMMAND';
my @sig = split ' ', $Configsig_name;
$/ = 64;
while(<>)
 my @f = unpack 'CCSL6fS8A*', $_;
 my ($flag, $version, $tty, $exitcode, $uid, $gid, $pid, $ppid,
 $btime, $etime, $utime, $stime, $mem, $io, $rw,
 $minflt, $majflt, $swaps, $cmd) = @f;
 my $s = $exitcode & 0x7f;
 my $status = $s ? "SIG$sig[$s]" : $exitcode >> 8;
 printf "%-7s %6d %6d %02d:%02d:%02d %8.2f %-16sn",
 $status, $uid, $pid,
 (localtime $btime)[2,1,0],
 $etime / 100,
 $cmd;

$ perl pacct.pl /var/log/account/pacct

This assumes the version 3 of the log file format -- see acct.h.

Notice however that this isn't that useful, because only the process/thread name in included in the log file (ie the basename of the executable, truncated to 15 bytes, and which could be easily faked with prctl(PR_SET_NAME)), not the path of the executable or the arguments it was invoked with.

If you want to extend that script to also display the stime, utime, etc fields, this may be useful:

# translate comp_t to float
# utime, stime, mem, minflt, majflt are in the comp_t format
# io, rw, swaps are never set; they're purely decorative
sub comp2f 
 my $m = $_[0] & 0x1fff; my $e = $_[0] >> 13; $m * 8 ** $e;

The

 set -o pipefail 

is pretty close to what you want; it will

A pipeline will not complete until all components of the pipeline have
completed, and the return value will be the value of the last non-zero command
to fail or zero if no command has failed.

So if you have 40 commands piped together, and the third command gives a return code of 8, and the rest complete successfully, then the overall return code would be 8. To figure out which command gave the bad return code would be tricky.