Alert when a backup fails

A backup that stops working unnoticed is found out on the day you need it. Let the backup script report its own failure: one HTTP request, and the message is in your Telegram, Discord or Pushover.

You need a BugsRadar project with a channel and the project's API key: see the quick start. The scripts read the key from the BUGSRADAR_KEY environment variable.

PostgreSQL: pg_dump

set -e stops the script at the first failed command, pipefail catches a failure inside a pipe, and the trap sends the alert with the line and the exit code:

#!/usr/bin/env bash
set -euo pipefail

notify() {
  curl -fsS --max-time 15 -H "X-Api-Key: $BUGSRADAR_KEY" --data-binary "$1" \
    "https://api.bugsradar.com/api/v3/notify?category=Backup&host=$(hostname)" > /dev/null || true
}
trap 'notify "Backup of shop-db failed at line $LINENO (exit code $?)"' ERR

file="/backups/shop-$(date +%F).dump"
pg_dump --format=custom --file="$file" shop
rsync -a /backups/ backup@nas:/volume1/backups/

The copy to another machine is part of the same script, so a broken rsync is reported too. Run the script from cron or a systemd timer.

MySQL and MariaDB: mysqldump

The same pattern. Without pipefail a failed mysqldump would go unnoticed, because gzip at the end of the pipe succeeds:

#!/usr/bin/env bash
set -euo pipefail

notify() {
  curl -fsS --max-time 15 -H "X-Api-Key: $BUGSRADAR_KEY" --data-binary "$1" \
    "https://api.bugsradar.com/api/v3/notify?category=Backup&host=$(hostname)" > /dev/null || true
}
trap 'notify "MySQL backup of shop failed at line $LINENO (exit code $?)"' ERR

mysqldump --single-transaction --routines shop | gzip > "/backups/shop-$(date +%F).sql.gz"

Windows: robocopy

A file backup with robocopy. Its exit codes from 8 up mean that files were not copied:

robocopy D:\Data \\nas\backup\data /MIR /R:2 /W:5 /NP /LOG:C:\Logs\backup.log

if ($LASTEXITCODE -ge 8) {
    Invoke-RestMethod -Method Post -Uri 'https://api.bugsradar.com/api/v3/notify?category=Backup' `
        -Headers @{ 'X-Api-Key' = $env:BUGSRADAR_KEY } -ContentType 'text/plain; charset=utf-8' `
        -Body "File backup to NAS failed on $env:COMPUTERNAME (robocopy exit code $LASTEXITCODE)"
}

Run it from Task Scheduler: see Windows Task Scheduler.

SQL Server

SQL Server backups usually run as SQL Server Agent jobs, maintenance plans included. Add an alert step to the job: see SQL Server Agent. The message then names the step that failed and the error of BACKUP DATABASE.

Know that the backup ran

An alert comes when a backup fails, but not when it never started — the server was off, or the job was disabled. To see every night that it ran, send an information message at the end of a successful run:

curl -fsS --max-time 15 -H "X-Api-Key: $BUGSRADAR_KEY" \
  --data-binary "Backup of shop-db finished: $(du -h "$file" | cut -f1)" \
  "https://api.bugsradar.com/api/v3/notify?category=Backup&level=information" > /dev/null || true

|| true keeps a network hiccup here from setting off the failure alert. The message arrives with the level Information. A morning without it means the backup didn't run. Keep such messages in a channel of their own if you don't want them next to errors: one project per purpose, each with its own channel.