systemd Auto-restart a crashed service in systemd

Systemd allows you to configure a service so that it automatically restarts in case it’s crashed.

Take a typical unit file that looks like this.

$ cat /etc/systemd/system/yourdaemon.service
[Unit]
Description=Your Daemon
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
ExecStart=/path/to/daemon

[Install]
WantedBy=multi-user.target

Most unit files are longer, but this gives you the gist of it. In the above example, if your daemon would crash or be killed, systemd would leave it alone.

You can however let systemd auto-restart it in case it fails or is accidentally killed. To do so, you can add the Restart option to the [Service] stanza.

$ cat /etc/systemd/system/yourdaemon.service
[Unit]
Description=Your Daemon
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=5s

ExecStart=/path/to/daemon

[Install]
WantedBy=multi-user.target

The above will react to anything that stops your daemon: a code exception, someone that does kill -9 <pid>, … as soon as your daemon stops, systemd will restart it in 5 seconds.

In this example, there are also StartLimitIntervalSec and StartLimitBurst directives in the [Unit] section. This prevents a failing service from being restarted every 5 seconds. This will give it 5 attempts, if it still fails, systemd will stop trying to start the service.

(Note: if you change your systemd unit file, make sure to run systemctl daemon-reload to reload the changes.)

If you ask for the status of your daemon after it’s been killed, systemd will show activating (auto-restart).

$ systemctl status yourdaemon
● yourdaemon.service - Your Daemon
   Loaded: loaded (/etc/systemd/system/yourdaemon.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: signal) since Mon 2020-01-13 09:07:41 UTC; 4s ago
  Process: 27165 ExecStart=/path/to/daemon (code=killed)
  Main PID: 27165 (code=killed, signal=KILL)

Give it a few seconds, and you’ll see the daemon was automatically restarted by systemd.

$ systemctl status yourdaemon
● yourdaemon.service - Your Daemon
   Loaded: loaded (/etc/systemd/system/yourdaemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-01-13 09:07:46 UTC; 6min ago

Pretty useful if you have a buggy service that’s safe to just restart on failure!