I wrote a script that listens for commands from a FIFO and collects couple of files for debugging on my raspberry pi, all working fine until i decided to write a systemd unit file for it. running systemctl start crashcollector will start the script but it never reads anything from the pipe it's running but not doing anything, systemctl stop crashcollector takes forever to stop the service so i'm guessing something wrong with my unit file even though i wrote many that work just fine.

bash script:

#!/bin/bash

collectCrashReport() 
 # teleport to cache dir
 cd /mnt/cache
 # remove any previous report
 rm -rf crash/crashreport_*
 # create report directory
 directory=crashreport_`cat /etc/hostname`_$(date +%Y%m%d%H%M%S)
 mkdir $directory
 cd $directory
 # snatch all the logs
 cp -r /var/log/ .
 ps --no-headers -ax > processes.dump
 # dump dmesg to file 
 dmesg > dmesg.dump
 # create crash dir
 mkdir crash
 # zip the whole thing
 zip -qr /mnt/cache/crash/"$directory.zip" .
 # remove collection dir
 rm -rf /mnt/cache/crashreport_*
 # done and done


# check if already running and kill
PIDFILE=/var/run/crashcollector.pid
# check if already running
if [ -f $PIDFILE ]; then
 kill `cat $PIDFILE`
fi
echo $$ > $PIDFILE

cleanup() 
 rm -f $PIDFILE
 exit 0

# trap them signals 
trap "cleanup" INT TERM EXIT

that_awesome_pipe="CCPipe"
[ -p $that_awesome_pipe ] || mkfifo $that_awesome_pipe
chmod 766 $that_awesome_pipe
while :; do
 if read line; then
 case $line in
 "collect")
 collectCrashReport
 ;;
 *)
 log "Received bad command $line, ignoring..."
 ;;
 esac 
 fi
done <"$that_awesome_pipe"

Systemd unit file:

[Unit]
Description=Crash report collector
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/crashCollector
ExecStop=/usr/bin/kill -9 `cat /var/run/crashcollector.pid`
KillMode=process
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

not sure what exactly i'm missing here removing that if read line block makes everything work just fine.

There are so many things wrong, there.

• Do not treat ExecStart and ExecStop as if they are shell command lines. They are not. The systemd manual does warn against this. Shell expansions such as command substitution are not available in these service unit file settings. It is not the shell language.


• Do not create a PID file mechanism when the place that you are starting from is having an actual service manager to hand. PID files are a mechanism that we have known to be broken since the 1980s, and that proper service management completely does away with the need for. You have a service manager. It will ensure that at most only one instance of the service runs at any time. It will track the process ID. It will deal with sending termination signals to the service process when the service is shut down.


• Do not put basic server-side stuff into the server process, such as creating the FIFO and opening its server end.

Do things this way:

• Create a socket unit to go alongside the service unit.


• Take all of the explicit handling of the FIFO out of crashCollector.


• Use ListenFIFO in the socket unit. This will of course need an absolute pathname.


• Use StandardInput=socket in the service unit.


• Make your crashcollector script just read from its standard input.


• Take all of the PID file mucking about out of crashCollector.


• Take all of the signal trapping and cleanup() stuff out of crashCollector.


• Take all of the explicit killing with ExecStop, and the consequent bodge that you tried to do with KillMode, out of the service unit.


• Use StandardError=journal in the service unit.


• Replace log with a simple echo 1>&2 to standard error.

Further reading

• Jonathan de Boyne Pollard (2001). Mistakes to avoid when designing Unix dæmon programs. Frequently Given Answers.


• Lennart Poettering et al.. systemd.socket. systemd manual pages. Freedesktop.org.


• How make service reading from FIFO socket

Everything was fixed after I moved the fifo to /tmp; not sure what that changes, but things work smoothly now.