mjhjmtu

We encountered the following problem in our company. We have multiple Red Hat Enterprise Linux Servers on which "SAP HANA S/4" is running.
We created a systemd Service to automatically start and stop the daemon so that we don't need to manually interact with the system on a reboot or shutdown.

Autostart works well, but there seems to be a problem with stopping the daemon correctly on shutdown. The daemons are running with another user (individual for each server). It seems that systemd starts to kill the user sessions before the actual service is stopped; as a result, the service won't stop properly.

Service

[Unit]
Description=saphana
After=remote-fs.target user.slice sapinit.service multi-user.target
Requires=user.slice

[Service]
KillMode=none
Type=oneshot
ExecStart=/hana/source/scripts/sapHanaControl.pl start
ExecStop=/hana/source/scripts/sapHanaControl.pl stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The script wich is called in ExecStart and ExecStop basically executes the following command.

On Start:

"sudo -u $username csh -c "sapcontrol -nr $instance -function Start"

On Stop:
"sudo -u $username csh -c "sapcontrol -nr $instance -function Stop"

Shutdown log

Output of the Systemd Log shows the following:

Jun 20 16:23:05 host123 systemd[1]: Stopping Session c4 of user **userxy**.
Jun 20 16:23:05 host123sapHanaControl.pl[15003]: sudo -u **userxy** csh -c "sapcontrol -nr 00 -function Stop"
Jun 20 16:23:05 host123 sapHanaControl.pl[15003]: 20.06.2018 16:23:05
Jun 20 16:23:05 host123 sapHanaControl.pl[15003]: Stop
Jun 20 16:23:05 host123 sapHanaControl.pl[15003]: FAIL: NIECONN_REFUSED (Connection refused), NiRawConnect failed in plugin_fopen()

Would really appreciate some help on that one if someone here ever encountered the same problem as we do. Hope that's enough information. Can provide more if needed.

Update

I see the following processes running when the system is running normally:

[root@wsstadt325 ~]# ps -ef | grep sapstartsrv
d61adm 1740 1 0 11:56 ? 00:00:01 /usr/sap/D61/HDB05/exe/sapstartsrv pf=/usr/sap/D61/SYS/profile/D61_HDB05_wsstadt325 -D -u d61adm
sapadm 1741 1 0 11:56 ? 00:00:04 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D
d21adm 1946 1 0 11:56 ? 00:00:02 /usr/sap/D21/ASCS01/exe/sapstartsrv pf=/usr/sap/D21/SYS/profile/D21_ASCS01_wsstadt325 -D -u d21adm
d21adm 2182 1 0 11:56 ? 00:00:02 /usr/sap/D21/D00/exe/sapstartsrv pf=/usr/sap/D21/SYS/profile/D21_D00_wsstadt325 -D -u d21adm` 

Chnaged my script to log the "ps -ef | grep sapstartsrv" output when the system gets rebooted / powered off.

ps -ef | grep sapstartsrv
sapadm 1683 1 0 13:52 ? 00:00:01 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D
root 5706 5522 0 14:00 ? 00:00:00 sh -c ps -ef | grep sapstartsrv
root 5708 5706 0 14:00 ? 00:00:00 grep sapstartsrv

The sapstartsrv Service is started by a default SAP Service (sapinit) wich gets startet before my own Systemd Service (So on a reboot it is the reversed order [Stop my Systemd Service -> stop the Sapinit Service]) The problem seems to be that systemctl starts to kill the user session (In my case for the user: d21adm and d61adm) where the sapstartsrv process is running before my actual Systemd service is stopped. (Hope that makes at least a bit sense)

Here's an image of the whole systemd chain (my services are at the very end):
The services involved:
- sapinit.service (The default one)
- saphana.service (My custom one)

Figured out the cause of my problem as described in the following KB
https://www.suse.com/de-de/support/kb/doc/?id=7022671

Systemd kills every user.slice after 90 seconds (This timeout can't be changed) It looks like systemd just isn't made to automatically Stop SAP HANA Instances without modifying pam.d. The Solution described there seems to be a bit "hackish" but it works.

cp /etc/pam.d/system-auth /etc/pam.d/custom-su-session
vim /etc/pam.d/custom-su-session

Insert the following line before "session optional pam_systemd.so"

session [success=1 new_authtok_reqd=ok default=ignore] pam_listfile.so item=user sense=allow file=/etc/custom-su-session

This line skips the user.slice creation when the su command is executed an the user is listed in the file /etc/custom-su-session

vim /etc/pam.d/su

Replace session include system-auth
with session include custom-su-session