mjhjmtu

I'm trying to set up a simple web load balancing using CentOS 7.5 (64bit, kernel 3.10), HAProxy 1.8.13 and systemd.
It seems that the HAProxy configuration is OK, but starting the application gives me a headache. I tried it with init.d once but was then pointed into the systemd direction, where I'm currently stuck. Been looking for the cause for almost 2 days now and I have reached my limits when it comes to linux, I also couldn't find a cause for this particular behavior anywhere. Most cases like this have some issue in the haproxy config, but that's not the case here apparently.

HAProxy config file validation output

haproxy -f /etc/haproxy/haproxy.conf -c

Configuration file is valid

When I try to start it using systemd, I get the following output in the journalctl:

journalctl -u haproxy.service

Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:50 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:50 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:50 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:51 localhost.localdomain systemd[1]: Starting HAProxy Load Balancer...
Aug 13 17:07:51 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:07:51 localhost.localdomain systemd[1]: start request repeated too quickly for haproxy.service
Aug 13 17:07:51 localhost.localdomain systemd[1]: Failed to start HAProxy Load Balancer.
Aug 13 17:07:51 localhost.localdomain systemd[1]: Unit haproxy.service entered failed state.
Aug 13 17:07:51 localhost.localdomain systemd[1]: haproxy.service failed.

Systemctl shows the following status:

systemctl status haproxy.service

[root@localhost sbin]# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
 Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
 Active: failed (Result: start-limit) since Mo 2018-08-13 17:33:46 CEST; 24min ago
 Process: 1557 ExecStart=/usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid (code=exited, status=0/SUCCESS)
 Process: 1556 ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q (code=exited, status=0/SUCCESS)
 Main PID: 1557 (code=exited, status=0/SUCCESS)

Aug 13 17:33:46 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
Aug 13 17:33:46 localhost.localdomain systemd[1]: haproxy.service holdoff time over, scheduling restart.
Aug 13 17:33:46 localhost.localdomain systemd[1]: start request repeated too quickly for haproxy.service
Aug 13 17:33:46 localhost.localdomain systemd[1]: Failed to start HAProxy Load Balancer.
Aug 13 17:33:46 localhost.localdomain systemd[1]: Unit haproxy.service entered failed state.
Aug 13 17:33:46 localhost.localdomain systemd[1]: haproxy.service failed.

The systemd file for this service was mostly taken over from the sample file haproxy 1.8.13 provided. I just replaced some placeholders/variables with fixed values, since some component couldn't handle relative paths.

nano /lib/systemd/system/haproxy.service

[Unit]
Description=HAProxy Load Balancer
After=network.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/var/run/haproxy.pid"
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
ExecReload=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify

# The following lines leverage SystemD's sandboxing options to provide
# defense in depth protection at the expense of restricting some flexibility
# in your setup (e.g. placement of your configuration files) or possibly
# reduced performance. See systemd.service(5) and systemd.exec(5) for further
# information.

# NoNewPrivileges=true
# ProtectHome=true
# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,
# any state files and any other files written using 'ReadWritePaths' or
# 'RuntimeDirectory'.
# ProtectSystem=true
# ProtectKernelTunables=true
# ProtectKernelModules=true
# ProtectControlGroups=true
# If your SystemD version supports them, you can add: @reboot, @swap, @sync
# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io

[Install]
WantedBy=multi-user.target

It appears that it starts the application, stops it again and then tries to start it again, until it gives up at some point.

When I try to start the application manually from the bash, it looks like nothing is happening:

[root@localhost sbin]# /usr/sbin/haproxy -W -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
[root@localhost sbin]#

No process visible via ps or top. I tried getting some logs, but I couldn't get any log from rsyslog since it doesn't want to write into the file I specified.

Adding the haproxy config file as an appendix, but as per haproxy itself, it seems to be fine:

# Rev1
# HAProxy Global and Default definitions
global
 log 127.0.0.1 local2
 chroot /var/lib/haproxy
 pidfile /var/run/haproxy.pid
 maxconn 4000
 user haproxy
 group haproxy
 daemon
 stats socket /var/run/haproxy.sock
 stats timeout 2m

defaults
 log global
 option dontlognull
 retries 3
 timeout http-request 10s
 timeout queue 1m
 timeout connect 10s
 timeout client 300s
 timeout server 300s
 timeout http-keep-alive 10s
 timeout check 10s
 maxconn 3000

frontend LB_https_vs
 bind *:443
 mode tcp
 option tcplog
 default_backend SP_https_group

backend LB_https_group
 mode tcp
 balance source
 option httpchk GET /MyPage/Logon.aspx
 hash-type consistent
 stick-table type ip size 200k expire 2000s
 stick on src table SP_https_group
 default-server inter 15000
 #server srv1 192.168.1.1:443 id 1 port 443 check
 server srv1 192.168.1.1:443 id 1 check
 server srv2 192.168.1.2:443 id 2 check
 server srv3 192.168.1.3:443 id 3 check
 server srv4 192.168.1.4:443 id 4 check
 server srv5 192.168.1.5:443 id 5 check

listen stats
 mode http
 bind :9000
 stats enable
 stats hide-version
 stats realm HAproxy-Statistics
 stats uri /haproxy_stats
 stats auth admin:default

Any ideas what's going on here?

Thanks in advance

You need to start haproxy in a mode that runs in foreground. Furthermore, if you want to use Type=notify, you need haproxy to implement the systemd notify daemon.

It looks like you should start it using the -Ws option rather than just -W. From the --help output in the source code:

#if defined(USE_SYSTEMD)
 " -Ws master-worker mode with systemd notify support.n"
#endif

Please also note that this only works if haproxy was built with systemd support. Not sure if that's the case for you, but you can check it by looking at the --help output (it doesn't seem to support --help directly, but it prints usage info on unknown args, so --help should work for you.)

You might want to take a look at the suggested haproxy.service template in version 1.8 and the latest one in GitHub, looks like what you're using is close to the latest one, but with some differences (such as -W instead of -Ws...)