Alert when a systemd service fails

systemd can start another unit when a service fails. Point it at a small unit that sends the alert, and a failed service or timer job arrives in your Telegram, Discord or Pushover.

You need a BugsRadar project with a channel and the project's API key: see the quick start.

How it works

OnFailure= in a service names the units to start when the service enters the failed state. One template unit, bugsradar-alert@.service, serves every service: the name of the failed service comes in as the instance name.

The alert unit

[Unit]
Description=BugsRadar alert for %i

[Service]
Type=oneshot
EnvironmentFile=/etc/bugsradar.env
ExecStart=/usr/bin/curl -fsS --max-time 15 -H "X-Api-Key: ${BUGSRADAR_KEY}" --data-binary "%i failed on %H" "https://api.bugsradar.com/api/v3/notify?category=systemd"

%i becomes the name of the failed service and %H the host name. The key lives in a file that only root can read:

echo 'BUGSRADAR_KEY=<your project API key>' | sudo tee /etc/bugsradar.env > /dev/null
sudo chmod 600 /etc/bugsradar.env
sudo systemctl daemon-reload

Watch a service

Add OnFailure= to the service with a drop-in, so the unit file itself stays as the package installed it:

sudo systemctl edit backup.service

[Unit]
OnFailure=bugsradar-alert@%n.service

%n is the full name of the service, such as backup.service. systemctl edit reloads systemd when you save, so nothing else is needed.

Timers and restarts

• A timer starts a service, and it is the service that fails. Put OnFailure= in the service, not in the .timer unit.
• A service with Restart= enters the failed state only when systemd gives up restarting it. Short crashes that a restart fixes send no alert.

Test it

Start the alert unit by hand with any instance name:

sudo systemctl start bugsradar-alert@test.service

The message “test failed on your host” arrives within seconds. If it doesn't, journalctl -u bugsradar-alert@test.service shows what curl answered.