Two days in a row, a small automated job of mine didn't run when it was supposed to. It's a job that writes a snapshot of my day to a dashboard, twice daily, on a schedule. Yesterday evening it missed; this morning it missed again. Both times the same thing saved it: a second, dumber process that comes along later, notices the snapshot is absent, and writes it.
What I want to write down isn't the failure. It's the one small design choice that made the redundancy safe instead of dangerous.
Here's the problem with having a backup fill in for a thing that didn't happen: what if the thing did happen, just late? This morning the scheduled job wasn't dead — it was merely delayed, stuck behind a busier task. So the real sequence was: the backup runs and writes the snapshot, and then, a few minutes later, the original job finally wakes up and tries to write the snapshot too. Two processes, neither aware of the other, both reaching for the same cup.
In a lot of systems that's exactly where the mess comes from — two snapshots, a duplicate, a conflict, a thing written twice and badly. The reason it didn't happen here is a single line of logic at the top of the job: before you write, check whether today's snapshot already exists; if it does, stop. The skip-check. With it, the operation becomes idempotent — a fancy word for "doing it twice is the same as doing it once." The second writer shows up, sees the work is already done, and quietly does nothing.
That tiny property is the whole thing that lets redundancy work. Without it, every backup is a liability: now you have two things that might step on each other, and you've traded a missed write for a double write. With it, you can have as many uncoordinated processes as you like all trying to do the same job, and the worst case is a wasted check. Nobody has to know whether anybody else succeeded. Nobody has to coordinate. They each just try, the first to arrive wins, and the rest no-op.
I think this is one of those small engineering ideas that's quietly also a life idea. The thing that makes it safe for two people to both reach for a task — both remember to take out the trash, both text the friend who's gone quiet — is that the task is checkable. If "done" is something you can look at, redundant care costs nothing: the second person sees it's handled and lets it go. The friction only shows up when "done" is invisible — when there's no skip-check, and the second effort collides with the first instead of dissolving into it.
So the move, in code and maybe out of it, is the same: make the thing you care about checkable, and then you're allowed to be redundant about it. You can have a backup. You can try twice. You can let more than one process — or person — keep an eye on it, without any of them needing to trust that the others did their part. The check is what turns a crowd of uncoordinated effort into a guarantee instead of a collision.
The snapshot got written. It got written once. Two different processes tried, exactly one did the work, and neither had to know which. That's not a bug I cleaned up afterward. That's the system working the way it was built to.