mjhjmtu

I don't know how nor why but my Debian systemd stopped starting any service, particularly sshd service so I can't access to my machine. It's a headless machine without monitor and moreover I can't connect one.

I know systemd is working because when I boot with a USB with puppy, and I chroot the debian partition, I can read logs with journalclt. Everything seems working OK, but the system suddenly gets stucked receiving watchdog events ad infinitum.

I paste the tail of the output:

[...]
mar 06 19:36:26 DEBIAN-LXDE systemd[1]: ifup@eth0.service: Child 560 belongs to ifup@eth0.service
mar 06 19:36:26 DEBIAN-LXDE systemd[1]: ifup@eth0.service: Main process exited, code=exited, status=0/SUCCESS
mar 06 19:36:26 DEBIAN-LXDE systemd[1]: ifup@eth0.service: Failed to destroy cgroup /system.slice/ifup@eth0.service, ignoring: Device or resource busy
mar 06 19:36:26 DEBIAN-LXDE systemd[1]: ifup@eth0.service: Changed running -> exited
mar 06 19:36:32 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:36:44 DEBIAN-LXDE ntpdate[729]: step time server 158.227.98.15 offset 9.571738 sec
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: Time has been changed
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: Set up TFD_TIMER_CANCEL_ON_SET timerfd.
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: Received SIGCHLD from PID 722 (ntpdate).
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: Child 722 (ntpdate) died (code=exited, status=0/SUCCESS)
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: ifup@eth0.service: Child 722 belongs to ifup@eth0.service
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: ifup@eth0.service: cgroup is empty
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 19 (stored), closing.
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: Got cgroup empty notification for: /system.slice/ifup@eth0.service
mar 06 19:36:44 DEBIAN-LXDE systemd[1]: ifup@eth0.service: cgroup is empty
mar 06 19:37:02 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:37:22 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:37:42 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:38:02 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:38:02 DEBIAN-LXDE systemd[1]: systemd-journald.service: Got notification message from PID 237 (WATCHDOG=1)
mar 06 19:38:22 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:38:42 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:39:02 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:39:22 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:39:42 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:39:42 DEBIAN-LXDE systemd[1]: systemd-journald.service: Got notification message from PID 237 (WATCHDOG=1)
mar 06 19:40:02 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:40:22 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:40:42 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:41:02 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:41:02 DEBIAN-LXDE systemd[1]: systemd-journald.service: Got notification message from PID 237 (WATCHDOG=1)
mar 06 19:41:22 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:41:42 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:42:02 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:42:22 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
mar 06 19:42:42 DEBIAN-LXDE systemd[1]: systemd-udevd.service: Got notification message from PID 293 (WATCHDOG=1)
[...]

Nevertheless the logs in /var/log/ were written few days ago. I thought the problem could be because some usb drive attached but I disconnected all of them and the result is the same. I read a lot in the web but I could't find anyting similar. Only about specific services.

I tried to write a check service as:

[Unit]
Description=Avisa cuando se arranca el sistema
Requires=network.target
After=network.target

[Service]
Type=simple
RemainAfterExit=no
ExecStart=/usr/bin/mail -s "AVISOOOO" xxxxxxxx@gmail.com

[Install]
WantedBy=default.target

But the mail is never sent.

How could I find out what the problem is?

UPDATE: I've finally managed to start the system plugin all the usb devices again. I think it was that point because I tried several thigs: reinstall grub, check the main partition, etc. Thank you for all of you that took a moment to give a hand. Thanks.

my Debian systemd stopped starting any service

There is no indication from the log supplied that this is in fact the case.

the system suddenly gets stucked receiving watchdog events ad infinitum.

These events are normal notifications, and far from the system being stuck they indicate that the system is not stuck. These are one part of the system telling another part of the system that it is not stuck, every 20 seconds.

How could I find out what the problem is?

Diagnose it properly.

If the problem is with the SSH service, look at the logs of the SSH service. Don't leap to conclusions that your system is not starting any services without logs of failures to start services. Indeed, do not leap to conclusions that the problem is with systemd without determining how you are unable to log-on. A timeout, a refused connection, and a rejected authentication can all have very different root causes, some of which might not even be anything on your machine at all.

Proceed systematically and methodically, with logs in hand at every step.

the mail is never sent.

You've magnified one problem into two. Now you have the additional task of determining how your mail system is malfunctioning, in order to then determine how something else is malfunctioning. This is not the best approach. Logs are your friend.

I've finally managed to start the system plugin all the usb devices again. I think it was that point because I tried several thigs: reinstall grub, check the main partition, etc. Thank you for all of you that took a moment to give a hand. Thanks.