Configure buggy systemd service to terminate via SIGKILL
Clash Royale CLAN TAG#URR8PPP
up vote
16
down vote
favorite
Background
I've been asked to create a systemd script for a new service, foo_daemon, that sometimes gets into a "bad state", and won't die via SIGTERM (likely due to custom signal handler). This is problematic for developers, as they are instructed to start/stop/restart the service via:
systemctl start foo_daemon.servicesystemctl stop foo_daemon.servicesystemctl restart foo_daemon.service
Problem
Sometimes, due to foo_daemon getting into a bad state, we have to forcibly kill it via:
systemctl kill -s KILL foo_daemon.service
Question
How can I setup my systemd script for foo_daemon so that, whenever a user attempts to stop/restart the service, systemd will:
- Attempt a graceful shutdown of
foo_daemonviaSIGTERM. - Give up to 2 seconds for shutdown/termination of
foo_daemonto complete. - Attempt a forced shutdown of
foo_daemonviaSIGKILLif the process is still alive (so we don't have a risk of the PID being recycled andsystemdissuesSIGKILLagainst the wrong PID). The device we're testing spawns/forks numerous processes rapidly, so there is a rare but very real concern about PID recycling causing a problem. - If, in practise, I'm just being paranoid about PID recycling, I'm OK with the script just issuing
SIGKILLagainst the process' PID without being concerned about killing a recycled PID.
linux service systemd signals
asked Aug 28 at 18:11
DevNull
208110
add a comment |Â
up vote
16
down vote
favorite
Background
I've been asked to create a systemd script for a new service, foo_daemon, that sometimes gets into a "bad state", and won't die via SIGTERM (likely due to custom signal handler). This is problematic for developers, as they are instructed to start/stop/restart the service via:
systemctl start foo_daemon.servicesystemctl stop foo_daemon.servicesystemctl restart foo_daemon.service
Problem
Sometimes, due to foo_daemon getting into a bad state, we have to forcibly kill it via:
systemctl kill -s KILL foo_daemon.service
Question
How can I setup my systemd script for foo_daemon so that, whenever a user attempts to stop/restart the service, systemd will:
- Attempt a graceful shutdown of
foo_daemonviaSIGTERM. - Give up to 2 seconds for shutdown/termination of
foo_daemonto complete. - Attempt a forced shutdown of
foo_daemonviaSIGKILLif the process is still alive (so we don't have a risk of the PID being recycled andsystemdissuesSIGKILLagainst the wrong PID). The device we're testing spawns/forks numerous processes rapidly, so there is a rare but very real concern about PID recycling causing a problem. - If, in practise, I'm just being paranoid about PID recycling, I'm OK with the script just issuing
SIGKILLagainst the process' PID without being concerned about killing a recycled PID.
linux service systemd signals
asked Aug 28 at 18:11
DevNull
208110
2
Even if you spawn processes rapidly enough to roll over 4 million PIDs in two seconds, systemd does not sit in a loop checking "is this pid still alive? is this pid still alive?" because it doesn't need to; it is already informed about whether its immediate child processes are still alive or not (by means of ordinary SIGCHLD and waitpid()). So if it sees that the process exited after SIGTERM, it will simply mark the service as 'inactive' at that point – it will not bother with checking, waiting, and sending the SIGKILL at all.
– grawity
Aug 29 at 6:33
add a comment |Â
up vote
16
down vote
favorite
up vote
16
down vote
favorite
Background
I've been asked to create a systemd script for a new service, foo_daemon, that sometimes gets into a "bad state", and won't die via SIGTERM (likely due to custom signal handler). This is problematic for developers, as they are instructed to start/stop/restart the service via:
systemctl start foo_daemon.servicesystemctl stop foo_daemon.servicesystemctl restart foo_daemon.service
Problem
Sometimes, due to foo_daemon getting into a bad state, we have to forcibly kill it via:
systemctl kill -s KILL foo_daemon.service
Question
How can I setup my systemd script for foo_daemon so that, whenever a user attempts to stop/restart the service, systemd will:
- Attempt a graceful shutdown of
foo_daemonviaSIGTERM. - Give up to 2 seconds for shutdown/termination of
foo_daemonto complete. - Attempt a forced shutdown of
foo_daemonviaSIGKILLif the process is still alive (so we don't have a risk of the PID being recycled andsystemdissuesSIGKILLagainst the wrong PID). The device we're testing spawns/forks numerous processes rapidly, so there is a rare but very real concern about PID recycling causing a problem. - If, in practise, I'm just being paranoid about PID recycling, I'm OK with the script just issuing
SIGKILLagainst the process' PID without being concerned about killing a recycled PID.
linux service systemd signals
asked Aug 28 at 18:11
DevNull
208110
Background
I've been asked to create a systemd script for a new service, foo_daemon, that sometimes gets into a "bad state", and won't die via SIGTERM (likely due to custom signal handler). This is problematic for developers, as they are instructed to start/stop/restart the service via:
systemctl start foo_daemon.servicesystemctl stop foo_daemon.servicesystemctl restart foo_daemon.service
Problem
Sometimes, due to foo_daemon getting into a bad state, we have to forcibly kill it via:
systemctl kill -s KILL foo_daemon.service
Question
How can I setup my systemd script for foo_daemon so that, whenever a user attempts to stop/restart the service, systemd will:
- Attempt a graceful shutdown of
foo_daemonviaSIGTERM. - Give up to 2 seconds for shutdown/termination of
foo_daemonto complete. - Attempt a forced shutdown of
foo_daemonviaSIGKILLif the process is still alive (so we don't have a risk of the PID being recycled andsystemdissuesSIGKILLagainst the wrong PID). The device we're testing spawns/forks numerous processes rapidly, so there is a rare but very real concern about PID recycling causing a problem. - If, in practise, I'm just being paranoid about PID recycling, I'm OK with the script just issuing
SIGKILLagainst the process' PID without being concerned about killing a recycled PID.
linux service systemd signals
linux service systemd signals
asked Aug 28 at 18:11
DevNull
208110
asked Aug 28 at 18:11
DevNull
208110
asked Aug 28 at 18:11
DevNull
208110
asked Aug 28 at 18:11
DevNull
208110
asked Aug 28 at 18:11
DevNull
208110
208110
2
Even if you spawn processes rapidly enough to roll over 4 million PIDs in two seconds, systemd does not sit in a loop checking "is this pid still alive? is this pid still alive?" because it doesn't need to; it is already informed about whether its immediate child processes are still alive or not (by means of ordinary SIGCHLD and waitpid()). So if it sees that the process exited after SIGTERM, it will simply mark the service as 'inactive' at that point – it will not bother with checking, waiting, and sending the SIGKILL at all.
– grawity
Aug 29 at 6:33
add a comment |Â
2
Even if you spawn processes rapidly enough to roll over 4 million PIDs in two seconds, systemd does not sit in a loop checking "is this pid still alive? is this pid still alive?" because it doesn't need to; it is already informed about whether its immediate child processes are still alive or not (by means of ordinary SIGCHLD and waitpid()). So if it sees that the process exited after SIGTERM, it will simply mark the service as 'inactive' at that point – it will not bother with checking, waiting, and sending the SIGKILL at all.
– grawity
Aug 29 at 6:33
2
2
Even if you spawn processes rapidly enough to roll over 4 million PIDs in two seconds, systemd does not sit in a loop checking "is this pid still alive? is this pid still alive?" because it doesn't need to; it is already informed about whether its immediate child processes are still alive or not (by means of ordinary SIGCHLD and waitpid()). So if it sees that the process exited after SIGTERM, it will simply mark the service as 'inactive' at that point – it will not bother with checking, waiting, and sending the SIGKILL at all.
– grawity
Aug 29 at 6:33
Even if you spawn processes rapidly enough to roll over 4 million PIDs in two seconds, systemd does not sit in a loop checking "is this pid still alive? is this pid still alive?" because it doesn't need to; it is already informed about whether its immediate child processes are still alive or not (by means of ordinary SIGCHLD and waitpid()). So if it sees that the process exited after SIGTERM, it will simply mark the service as 'inactive' at that point – it will not bother with checking, waiting, and sending the SIGKILL at all.
– grawity
Aug 29 at 6:33
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
23
down vote
accepted
systemd already supports this out of the box, and it is enabled by default.
The only thing you might want to customize is the timeout, which you can do with TimeoutStopSec=. For example:
[Service]
TimeoutStopSec=2
Now, systemd will send a SIGTERM, wait two seconds for the service to exit, and if it doesn't, it will send a SIGKILL.
If your service is not systemd-aware, you may need to provide the path to its PID file with PIDFile=.
Finally, you mentioned that your daemon spawns many processes. In this case, you might wish to set KillMode=control-group and systemd will send signals to all of the processes in the cgroup.
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
Thank you. One last question: let's assume the service is not systemd-aware. What could I add to the systemd script for this service so that systemd creates/manages the PID file? Additionally, the service can be multi-instance via template units, so we typically launch it via `systemctl start foo_dameon@1.service", so would that impact the PID file logic in the script?
– DevNull
Aug 28 at 18:41
4
@DevNull systemd does not create or manage PID files. There's no reason for it to do so. If your service doesn't create its own PID file, then if possible configure it to run in the foreground (instead of daemonizing) and setType=simplein the systemd unit.
– Michael Hampton♦
Aug 28 at 18:42
1
If the service has dependants,Type=forkinghas the advantage of (if the service was properly written) informing systemd when it's fully 'ready' which Type=simple cannot do. Daemonizing isn't a problem, even without a PID file – systemd will track down the main process anyway.
– grawity
Aug 29 at 6:29
1
@grawity True enough...though it's been my experience that services daemonize before they are actually ready to begin serving. A systemd-aware service usingType=notifyis best for systemd, and many common services already do this. But probably not this legacy service. In the OP's case, he has a service which spawns many processes. The systemd docs warn about this case.
– Michael Hampton♦
Aug 29 at 13:54
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
23
down vote
accepted
systemd already supports this out of the box, and it is enabled by default.
The only thing you might want to customize is the timeout, which you can do with TimeoutStopSec=. For example:
[Service]
TimeoutStopSec=2
Now, systemd will send a SIGTERM, wait two seconds for the service to exit, and if it doesn't, it will send a SIGKILL.
If your service is not systemd-aware, you may need to provide the path to its PID file with PIDFile=.
Finally, you mentioned that your daemon spawns many processes. In this case, you might wish to set KillMode=control-group and systemd will send signals to all of the processes in the cgroup.
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
Thank you. One last question: let's assume the service is not systemd-aware. What could I add to the systemd script for this service so that systemd creates/manages the PID file? Additionally, the service can be multi-instance via template units, so we typically launch it via `systemctl start foo_dameon@1.service", so would that impact the PID file logic in the script?
– DevNull
Aug 28 at 18:41
4
@DevNull systemd does not create or manage PID files. There's no reason for it to do so. If your service doesn't create its own PID file, then if possible configure it to run in the foreground (instead of daemonizing) and setType=simplein the systemd unit.
– Michael Hampton♦
Aug 28 at 18:42
1
If the service has dependants,Type=forkinghas the advantage of (if the service was properly written) informing systemd when it's fully 'ready' which Type=simple cannot do. Daemonizing isn't a problem, even without a PID file – systemd will track down the main process anyway.
– grawity
Aug 29 at 6:29
1
@grawity True enough...though it's been my experience that services daemonize before they are actually ready to begin serving. A systemd-aware service usingType=notifyis best for systemd, and many common services already do this. But probably not this legacy service. In the OP's case, he has a service which spawns many processes. The systemd docs warn about this case.
– Michael Hampton♦
Aug 29 at 13:54
add a comment |Â
up vote
23
down vote
accepted
systemd already supports this out of the box, and it is enabled by default.
The only thing you might want to customize is the timeout, which you can do with TimeoutStopSec=. For example:
[Service]
TimeoutStopSec=2
Now, systemd will send a SIGTERM, wait two seconds for the service to exit, and if it doesn't, it will send a SIGKILL.
If your service is not systemd-aware, you may need to provide the path to its PID file with PIDFile=.
Finally, you mentioned that your daemon spawns many processes. In this case, you might wish to set KillMode=control-group and systemd will send signals to all of the processes in the cgroup.
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
Thank you. One last question: let's assume the service is not systemd-aware. What could I add to the systemd script for this service so that systemd creates/manages the PID file? Additionally, the service can be multi-instance via template units, so we typically launch it via `systemctl start foo_dameon@1.service", so would that impact the PID file logic in the script?
– DevNull
Aug 28 at 18:41
4
@DevNull systemd does not create or manage PID files. There's no reason for it to do so. If your service doesn't create its own PID file, then if possible configure it to run in the foreground (instead of daemonizing) and setType=simplein the systemd unit.
– Michael Hampton♦
Aug 28 at 18:42
1
If the service has dependants,Type=forkinghas the advantage of (if the service was properly written) informing systemd when it's fully 'ready' which Type=simple cannot do. Daemonizing isn't a problem, even without a PID file – systemd will track down the main process anyway.
– grawity
Aug 29 at 6:29
1
@grawity True enough...though it's been my experience that services daemonize before they are actually ready to begin serving. A systemd-aware service usingType=notifyis best for systemd, and many common services already do this. But probably not this legacy service. In the OP's case, he has a service which spawns many processes. The systemd docs warn about this case.
– Michael Hampton♦
Aug 29 at 13:54
add a comment |Â
up vote
23
down vote
accepted
up vote
23
down vote
accepted
systemd already supports this out of the box, and it is enabled by default.
The only thing you might want to customize is the timeout, which you can do with TimeoutStopSec=. For example:
[Service]
TimeoutStopSec=2
Now, systemd will send a SIGTERM, wait two seconds for the service to exit, and if it doesn't, it will send a SIGKILL.
If your service is not systemd-aware, you may need to provide the path to its PID file with PIDFile=.
Finally, you mentioned that your daemon spawns many processes. In this case, you might wish to set KillMode=control-group and systemd will send signals to all of the processes in the cgroup.
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
systemd already supports this out of the box, and it is enabled by default.
The only thing you might want to customize is the timeout, which you can do with TimeoutStopSec=. For example:
[Service]
TimeoutStopSec=2
Now, systemd will send a SIGTERM, wait two seconds for the service to exit, and if it doesn't, it will send a SIGKILL.
If your service is not systemd-aware, you may need to provide the path to its PID file with PIDFile=.
Finally, you mentioned that your daemon spawns many processes. In this case, you might wish to set KillMode=control-group and systemd will send signals to all of the processes in the cgroup.
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
edited Aug 28 at 18:25
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
answered Aug 28 at 18:20
Michael Hampton♦
157k25291591
157k25291591
Thank you. One last question: let's assume the service is not systemd-aware. What could I add to the systemd script for this service so that systemd creates/manages the PID file? Additionally, the service can be multi-instance via template units, so we typically launch it via `systemctl start foo_dameon@1.service", so would that impact the PID file logic in the script?
– DevNull
Aug 28 at 18:41
4
@DevNull systemd does not create or manage PID files. There's no reason for it to do so. If your service doesn't create its own PID file, then if possible configure it to run in the foreground (instead of daemonizing) and setType=simplein the systemd unit.
– Michael Hampton♦
Aug 28 at 18:42
1
If the service has dependants,Type=forkinghas the advantage of (if the service was properly written) informing systemd when it's fully 'ready' which Type=simple cannot do. Daemonizing isn't a problem, even without a PID file – systemd will track down the main process anyway.
– grawity
Aug 29 at 6:29
1
@grawity True enough...though it's been my experience that services daemonize before they are actually ready to begin serving. A systemd-aware service usingType=notifyis best for systemd, and many common services already do this. But probably not this legacy service. In the OP's case, he has a service which spawns many processes. The systemd docs warn about this case.
– Michael Hampton♦
Aug 29 at 13:54
add a comment |Â
Thank you. One last question: let's assume the service is not systemd-aware. What could I add to the systemd script for this service so that systemd creates/manages the PID file? Additionally, the service can be multi-instance via template units, so we typically launch it via `systemctl start foo_dameon@1.service", so would that impact the PID file logic in the script?
– DevNull
Aug 28 at 18:41
4
@DevNull systemd does not create or manage PID files. There's no reason for it to do so. If your service doesn't create its own PID file, then if possible configure it to run in the foreground (instead of daemonizing) and setType=simplein the systemd unit.
– Michael Hampton♦
Aug 28 at 18:42
1
If the service has dependants,Type=forkinghas the advantage of (if the service was properly written) informing systemd when it's fully 'ready' which Type=simple cannot do. Daemonizing isn't a problem, even without a PID file – systemd will track down the main process anyway.
– grawity
Aug 29 at 6:29
1
@grawity True enough...though it's been my experience that services daemonize before they are actually ready to begin serving. A systemd-aware service usingType=notifyis best for systemd, and many common services already do this. But probably not this legacy service. In the OP's case, he has a service which spawns many processes. The systemd docs warn about this case.
– Michael Hampton♦
Aug 29 at 13:54
Thank you. One last question: let's assume the service is not systemd-aware. What could I add to the systemd script for this service so that systemd creates/manages the PID file? Additionally, the service can be multi-instance via template units, so we typically launch it via `systemctl start foo_dameon@1.service", so would that impact the PID file logic in the script?
– DevNull
Aug 28 at 18:41
Thank you. One last question: let's assume the service is not systemd-aware. What could I add to the systemd script for this service so that systemd creates/manages the PID file? Additionally, the service can be multi-instance via template units, so we typically launch it via `systemctl start foo_dameon@1.service", so would that impact the PID file logic in the script?
– DevNull
Aug 28 at 18:41
4
4
@DevNull systemd does not create or manage PID files. There's no reason for it to do so. If your service doesn't create its own PID file, then if possible configure it to run in the foreground (instead of daemonizing) and set
Type=simple in the systemd unit.– Michael Hampton♦
Aug 28 at 18:42
@DevNull systemd does not create or manage PID files. There's no reason for it to do so. If your service doesn't create its own PID file, then if possible configure it to run in the foreground (instead of daemonizing) and set
Type=simple in the systemd unit.– Michael Hampton♦
Aug 28 at 18:42
1
1
If the service has dependants,
Type=forking has the advantage of (if the service was properly written) informing systemd when it's fully 'ready' which Type=simple cannot do. Daemonizing isn't a problem, even without a PID file – systemd will track down the main process anyway.– grawity
Aug 29 at 6:29
If the service has dependants,
Type=forking has the advantage of (if the service was properly written) informing systemd when it's fully 'ready' which Type=simple cannot do. Daemonizing isn't a problem, even without a PID file – systemd will track down the main process anyway.– grawity
Aug 29 at 6:29
1
1
@grawity True enough...though it's been my experience that services daemonize before they are actually ready to begin serving. A systemd-aware service using
Type=notify is best for systemd, and many common services already do this. But probably not this legacy service. In the OP's case, he has a service which spawns many processes. The systemd docs warn about this case.– Michael Hampton♦
Aug 29 at 13:54
@grawity True enough...though it's been my experience that services daemonize before they are actually ready to begin serving. A systemd-aware service using
Type=notify is best for systemd, and many common services already do this. But probably not this legacy service. In the OP's case, he has a service which spawns many processes. The systemd docs warn about this case.– Michael Hampton♦
Aug 29 at 13:54
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f928376%2fconfigure-buggy-systemd-service-to-terminate-via-sigkill%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
2
Even if you spawn processes rapidly enough to roll over 4 million PIDs in two seconds, systemd does not sit in a loop checking "is this pid still alive? is this pid still alive?" because it doesn't need to; it is already informed about whether its immediate child processes are still alive or not (by means of ordinary SIGCHLD and waitpid()). So if it sees that the process exited after SIGTERM, it will simply mark the service as 'inactive' at that point – it will not bother with checking, waiting, and sending the SIGKILL at all.
– grawity
Aug 29 at 6:33