Field report · · The 408
The backup that failed for two weeks while its monitor stayed green
A server backup failed two weeks in a row, and its monitor stayed green the whole time. A green light on a failed backup is worse than no light at all. If anything you run reports its health with a heartbeat, here’s how to make the heartbeat mean something.
What happened
The tool caught its own error and exited cleanly
The backup tool printed that it had terminated with an error, then exited 0. The heartbeat ran only if the job exited cleanly, so it fired.
- job output
- terminated with an error
- exit code
- 0
- heartbeat
- sent: up
- backup file
- not written
Nothing alerted. The monitor was doing exactly what it was told.
Why
An exit code says the program finished, not that it worked
Plenty of tools rescue their own failures and exit 0. The backups failed in the first place because they wrote to the network drive while another, larger backup was already saturating it.
Green while every run failed, so nothing asked anyone to look.
The first failed run sends a failure beat with the reason.
The rule
Exit 0 is not success
Wrap scheduled jobs so that success means “the result is there.”
- 1
The job runs
Whatever it prints and however it exits, nothing is reported yet.
- 2
Check the evidence
The job’s own success marker, or the output file’s date and size.
- 3
Report what’s true
A heartbeat only on evidence. Otherwise a failure beat with the reason.
- ⛨
Count who should be reporting
A monitor that has never received its first beat can be invisible. Alert when the number of reporting jobs drops. And don’t schedule writes inside another job’s busy window.
Where this stops: if your tool exits non-zero on every failure, its exit code is honest. Check which kind yours is before you trust it, and keep the count of reporting jobs either way.
Do this Monday
Three steps before lunch
- List your scheduled jobs and what “success” actually means for each one.
- Wrap one job so its heartbeat fires only on that evidence.
- Add an alert for when the number of reporting jobs drops.
Which of your jobs would stay green tonight if it failed?