mjhjmtu

I have a service running software that generates some configuration files if they don't exist, and read them if they do exist. The problem I have been facing is that these files sometimes get corrupt, making the software unable to start, and thus making the service fail. In this case I would like to remove these files and restart the service.

I tried creating a service that should get executed in case of failure, by doing this:

[Service]
ExecStart=/bin/run_program
OnFailure=software-fail.service

where this service is:

[Service]
ExecStart=/bin/rm /file/to/delete
ExecStop=systemctl --user start software.service

The problem, however, is that this service doesn't start, even when the service has failed.

I tried doing

systemctl --user enable software-fail.service

but then it starts every time the system starts, just like any other service.

My temporary solution is to use

ExecStopPost=/bin/rm /file/to/delete

but this is not a satisfying way of solving it, as it will always delete the file upon stopping the service, no matter if it was because of failure or not.

Output when failing:

● software.service - Software
 Loaded: loaded (/home/trippelganger/.config/systemd/user/software.service; enabled; vendor preset: enabled)
 Active: failed (Result: exit-code) since Fri 2018-05-04 09:05:26 CEST; 5s ago
 Process: 1839 ExecStart=/bin/run_program (code=exited, status=1/FAILURE)
 Main PID: 1839 (code=exited, status=1/FAILURE)



May 04 09:05:26 trippelganger systemd[595]: software.service: Main process exited, code=exited, status=1/FAILURE
May 04 09:05:26 trippelganger systemd[595]: software.service: Unit entered failed state.
May 04 09:05:26 trippelganger systemd[595]: software.service: Failed with result 'exit-code'.

Output of systemctl --user status software-fail.service
is:

● software-fail.service - Delete corrupt files
 Loaded: loaded (/home/trippelganger/.config/systemd/user/software-fail.service; disabled; vendor preset: enabled)
 Active: inactive (dead)

NOTE: You probably want to use ExecStopPost= instead of OnFailure= here (see my other answer), but this is trying to address why your OnFailure= setup is not working.

The problem with OnFailure= not starting the unit might be because it's in the wrong section, it needs to be in the [Unit] section and not [Service].

You can try this instead:

# software.service
[Unit]
Description=Software
OnFailure=software-fail.service

[Service]
ExecStart=/bin/run_program

And:

# software-fail.service
[Unit]
Description=Delete corrupt files

[Service]
ExecStart=/bin/rm /file/to/delete
ExecStop=/bin/systemctl --user start software.service

I can make it work with this setup.

But note that using OnFailure= is not ideal here, since you can't really tell why the program failed, and chaining another start of it in ExecStop= by calling /bin/systemctl start directly is pretty hacky... The solution using ExecStopPost= and looking at the exit status is definitely superior.

If you define OnFailure= inside [Service], systemd (at least version 234 from Fedora 27) complains with:

software.service:6: Unknown lvalue 'OnFailure' in section 'Service'

Not sure if you're seeing that in your logs or not... (Maybe this was added in recent systemd?) That should be a hint of what is going on there... I hope this helps.

In order to perform some cleanup if the service fails, you can use ExecStopPost=, which is executed whether the service succeeds or not.

In the code you run at ExecStopPost=, you can use one of $SERVICE_RESULT, $EXIT_CODE or $EXIT_STATUS to determine the failure condition and act accordingly. See the documentation on those environment variables to check which one is appropriate for you.

Then you can use Restart=on-failure so that systemd tries to restart your unit when it fails.

Putting it all together, this is what it would look like. Assuming that run_program will exit with status 2 whenever the files are corrupted (hopefully you can adapt this to other failure scenarios from the documentation above), this should work:

[Service]
ExecStart=/bin/run_program
ExecStopPost=/bin/sh -c 'if [ "$$EXIT_STATUS" = 2 ]; then rm /file/to/delete; fi'
Restart=on-failure

(NOTE: The double dollar-sign $$ is to escape this to systemd, so the shell sees $EXIT_STATUS and accesses that variable. Using a single dollar-sign would also work, but then systemd would do that replacement instead and the shell would see [ "2" = 2 ], which arguably also works... Anyways, you can bypass most of that by putting all this logic into a shell script and calling it by its full path in ExecStopPost=, that would be probably better and you could also easily add more commands to the script, such as logging the action taken to recover from the error condition.)

Hopefully this will give you enough pointers to figure out how to configure this correctly given your particular situation!