Field report · · The 408
The power came back in a minute. Three things never came back on their own.
The power was out for about a minute. Everything rebooted, and an hour later 403 parts of the house still showed as unavailable. Three of them were never going to come back without a person. If anything you run starts at boot, this is the test that tells you whether it can survive a real one.
What happened
Each one started early, failed once and stopped
The server was back in about a minute and the home hub in three. Most things found their feet. Three didn’t:
- storage share
- mount error: network is unreachable
- network integration
- setup error, no retry
- appliance integration
- “loaded”, 43 devices unavailable
The storage share mounted before the server’s network route existed. It was marked nofail, which means “don’t block the boot”. It also means “don’t try again”, so it stayed empty, and the shared notes’ backup failed an hour later.
The two hub integrations asked for data before the services behind them were ready. One gave up with an error. The other reported itself loaded while every washer, dryer and fridge sensor under it stayed unavailable.
Why
A watchdog that waits for a change never sees something that was never up
The hub already had a watchdog. It reloads an integration whose devices go unavailable for ten minutes. It didn’t fire.
After a restart, those devices didn’t go unavailable. They started that way. Nothing changed, so nothing fired.
Catches a device that drops out while running. Misses one that never came up.
Ten minutes after boot, look at what’s actually up, reload what isn’t, then tell a person.
The rule
Every always-on system needs a cold-start test
Boot order isn’t a promise. Anything that starts at boot should cope with arriving first.
- 1
Retry on use, not just at boot
The storage share now mounts on first access, with a 30-second timeout. A failed boot mount is retried the next time anything reads it. Tested by cutting its route: the read failed, the route came back, the next read mounted it.
- 2
Check state after start
Ten minutes after the hub starts, if both of an integration’s sentinel devices are unavailable, it’s reloaded, given three minutes, and tried once more.
- ⛨
Then tell a person
Two tries, then a notice. A retry loop that never stops is just a quieter outage.
Where this stops: the retry has passed staged tests, a healthy start that changed nothing and a forced failure that retried twice and sent the notice. The next real power cut is its first real test, and this post will get an update when it happens.
Do this Monday
Three steps before lunch
- List what mounts or connects at boot, and what each one needs to be up first.
- Find one that never retries, like a
nofailmount or an integration that sets up once, and give it a retry. - Restart everything at once, on purpose, on a day you can watch it.
What in your house only works because it booted in the right order?