Postmortem: the backup that looked fine until it was needed

The bad migration ran at 11 PM. It was supposed to rename a column. Instead it dropped a table, the one with every user’s data in it. The app kept running for a while, serving pages from cache, until people started noticing their accounts were empty.

No problem, the founder thought. There’s a backup. It runs every night. The dashboard says so.

The backup existed. It ran on schedule. It just couldn’t bring the data back.

Before we go on, the honest bit. This is a composite of patterns we see across AI-built apps, not one specific client incident. The specifics are illustrative. The failure itself, a backup that turns out to be useless at the exact moment you need it, is real and more common than anyone wants to admit.

What “having a backup” actually meant here

The app was set up to run a nightly database dump. A scheduled job fired every night, wrote a file, and marked itself done. Green checkmark. Backup complete.

For months, nobody looked inside those files. Why would you? The job succeeded every night. The dashboard was green. As far as anyone could tell, the data was safe.

Then the table got dropped, and someone finally opened a backup file to restore from it. That’s when the problems started.

There are a few flavors this comes in, and this composite has hit most of them:

• The backup file was there but nearly empty, because the job had permission to connect but not to read the actual table data. It had been backing up the schema and nothing else for months.
• The backup was complete but stored on the same server that failed. When the server went, the backups went with it.
• The backup was fine, but nobody had ever tried a restore, so when the moment came, the restore command failed on a detail nobody had rehearsed, and the clock was running.

Different mechanisms, same ending. The backup that everyone counted on couldn’t do the one thing it existed for.

Why a backup you never test isn’t a backup

Here’s the uncomfortable truth at the center of this.

A backup is a promise that you can get your data back. The only way to know the promise holds is to actually get your data back, at least once, on purpose, before you have to do it for real. A backup you’ve never restored from is not a safety net. It’s a hope with a green checkmark.

The green checkmark only tells you the job ran. It says nothing about whether the file contains your data, whether the file is reachable when your main server is on fire, or whether the restore actually completes. Those are separate questions, and each one has to be answered by doing, not by assuming.

AI-built apps are especially exposed here. The backup, if it exists at all, was often set up as an afterthought with default settings. Nobody chose a retention window. Nobody picked an off-server storage location. And nobody, ever, ran a test restore. The setup optimized for “a backup exists,” which is a very different thing from “we can recover.”

What a real backup looks like

A backup you can trust has four properties, and all four matter.

It captures everything. Not just the schema, the actual data, all the tables, verified by size and by spot-checking a restored copy. A backup that’s suspiciously small is a warning, not a success.

It lives somewhere else. Off the server it’s protecting, ideally in a different location entirely. If the same event can destroy both your data and your backup, you don’t have a backup. You have two copies of a single point of failure.

It supports point-in-time recovery. A nightly dump means you can lose up to a day of data. Point-in-time recovery lets you roll back to the moment just before the bad migration ran, not to midnight. For an app handling real user data, that gap between “last night” and “just now” is often the whole ballgame.

It has been restored at least once. Someone took a backup, spun up a fresh database, restored into it, and confirmed the data was all there and correct. Until that’s happened, you’re guessing.

We walk through the mechanics of setting this up in our backup and restore tutorial, including how to actually run the test restore that most people skip.

The one thing to do this week

You don’t need to overhaul anything to find out if you have this problem. You need to attempt one restore.

Take your most recent backup. Spin up a scratch database somewhere, anywhere, it can be temporary. Restore the backup into it. Then open it and look. Are all your tables there? Is the row count roughly what you’d expect? Can you find a specific user’s record you know should exist?

If the answer is yes, congratulations, you have a real backup, and you now know it. If the answer is no, you just found out on a quiet afternoon instead of at midnight during an outage. Either result is a win compared to not knowing.

The whole lesson of this incident fits in one line: a backup you haven’t restored from is a theory, not a safety net. The founder in this story had done everything the setup asked for. The job ran. The checkmark was green. None of it mattered, because the one test that would have revealed the truth, a single restore, never happened.

If you’d like someone to check whether your backups actually restore, whether they live off-server, and how much data you’d lose in a bad moment, that’s part of what our free audit looks at. We’d rather find the broken backup with you now than have you find it alone at 3 AM.

FAQ

Isn’t a nightly backup enough?

It’s a start, but only if two things are true: the backup actually contains your data, and you’ve confirmed you can restore it. A nightly job that’s never been restored from is unverified. And nightly-only means you can lose up to a full day, which point-in-time recovery avoids.

Why would a backup be empty or incomplete?

Common causes are permissions (the backup user could connect but not read the tables), misconfiguration (only the schema was captured, not the data), or storage limits that silently truncated the file. None of these show up as a failed job. The job “succeeds” and the file is useless.

What is point-in-time recovery?

It’s the ability to restore your database to any specific moment, not just to when the last full backup ran. If a bad change happens at 11 PM, point-in-time recovery lets you roll back to 10:59, keeping everything up to that second. A plain nightly dump can only take you back to midnight.

How often should I test a restore?

At least once when you first set backups up, and then periodically, especially after any change to your database or backup setup. The test is the whole point. An untested backup and no backup are closer than they look.

Where should backups be stored?

Somewhere separate from the server they protect, ideally in a different physical location. If one failure can take out both your live data and your backups, they aren’t really backups. Off-site, off-server storage is what makes recovery possible when things go badly wrong.