The Backup You Have Is Not the Backup You Think

Almost every organization I talk to says it has backups. Very few have ever restored one. The distance between those two sentences is where data loss actually lives.

I found every one of these problems in my own infrastructure, not in a client's. I run a small estate of servers for my own projects, I wrote the backup jobs myself, and I still spent several weeks recently discovering that the thing I had been calling "backed up" was doing considerably less than I believed. That is the useful part. If it can happen to someone who built the system on purpose and reads the logs, it is certainly happening in businesses where backups are something a vendor set up once and nobody has opened since.

TL;DR: A backup job reporting success every night proves only that the job ran, not that it captured everything, wrote anything usable, or could ever be restored. In my own infrastructure I found databases silently outside the backup set, failed runs leaving zero-byte files that retention counted as real copies, verification that passed on empty archives, and an off-site upload that would happily ship yesterday's data and call it a good night. All of it reported green. If you cannot say when you last restored something, you do not have a backup - you have a job that runs.

What Makes a Backup a Real Backup?

A real backup has four properties, and most setups have one or two.

It is complete - it covers everything that matters, including things added after the backup was designed. It is separate - it lives somewhere the original disaster cannot reach, meaning a different machine and ideally a different building. It is verified - something has looked inside the file and confirmed there is data in it, rather than trusting that the job exited without complaining. And it is restorable - someone has taken a copy, put it back, and confirmed the result matches what was there.

A copy of a file on the same server is not a backup. A backup job that has never been restored is a hypothesis. The gap between "we have backups" and "we can recover" is where the cost sits, and it almost never becomes visible until the day you need it.

A Checklist Item Is Not a Check

The first thing I found was the plainest. My nightly database backup was missing databases. Not once - twice, months apart.

The job worked from a list of database servers it had been told about. Every night it walked that list, dumped what it found, shipped the results to another machine, and exited cleanly. The report was green because, from the job's point of view, everything it had been asked to do had succeeded. A list only knows what is in it.

Both times the pattern was identical: a new project went into production, the database came up with it, and the entry in the backup list was never added. The first time it was two databases running unprotected for weeks. The second time, months later, it was two more - and one of them held analytics history for everything I run, which cannot be re-fetched from anywhere. Every other database in the estate can be rebuilt from source data if it burns down. That one is only ever a record of things that already happened, and it had never been backed up at all.

Here is what makes this worth writing about rather than filing under "I should be more careful." There was already a checklist item covering exactly this. My own written procedure for standing up a new project says, in plain language, to declare backup coverage. I wrote that item. And the databases were missed anyway - twice, under a procedure that explicitly said not to miss them.

At that point it stops being an oversight and becomes a design problem. A tick box is not a check - it relies on a human being attentive at exactly the right moment, months apart, while doing something else. The fix was to stop maintaining a list altogether. The job now asks the server what is actually running before it backs up anything, works out which of those are databases, and fails loudly if it finds one that is not covered. A database can no longer be missed by being forgotten, because nobody is being asked to remember.

One detail from that rebuild decides whether a "smarter" fix actually works. My first instinct was to match on names - anything called something database-shaped gets picked up. That would have failed. One of the databases missed the first time around was running an image with no useful name attached to it at all, so a name-matching rule would have skipped exactly the system that had already been skipped once. The detection had to key off something structural - what the thing is configured as, not what it happens to be called. If you are building a coverage check, test it against your worst-named system, not your best one.

Success Is Not the Same as a Backup

The second failure is nastier, because it produces a folder that looks correct.

The dump tool creates its destination file before it does the work. If the database is unreachable the tool fails - but the file already exists, and it is zero bytes. My retention policy counted files. It did not weigh them.

Follow that forward. Retention kept fourteen nights. Fourteen consecutive failures would have rotated out every genuinely good copy and left me with fourteen empty files in a folder that looked completely healthy from the outside - right number of dated entries, right names, nothing obviously wrong until somebody tried to open one. A silent failure with a self-cleaning cover story.

That is not hypothetical. Two nights in a row failed exactly that way, on a database whose server had been stopped, and neither night raised anything. I caught it while looking at something else.

The correction is small and worth copying: the job now writes to a temporary file, verifies the contents, and only then moves it into place. If anything fails, nothing is left behind. An absent backup is a problem you can detect. A zero-byte backup with today's date on it is a problem that hides.

This is the same shape as something I found after a power outage exposed three unrelated risks in a Proxmox cluster - a backup target on a single non-redundant drive, with the same failure profile as the data it protected. In both cases the thing was named "backup," counted as a backup, and was not one.

Three Ways I Got Verification Wrong

Once I decided the job should look inside the file rather than trust an exit code, I got it wrong three times before I got it right. All three are worth stating because they are ordinary mistakes anyone would make.

The tool wasn't where I thought it was. The verification reads the stored archive back and confirms it is a valid, readable backup. I first ran that check on the machine holding the backups - which did not have the database tooling installed. Everything failed, and every archive was rejected and deleted. That sounds like a disaster, but it was the best possible outcome, because I had written the check to fail closed: when it could not prove a backup was good, it refused to keep it. A verification step that fails open would have passed everything silently and I would never have learned it was not running at all. Make your checks fail loudly, then make sure you have seen them fail at least once.

An empty database produces a perfectly valid backup. My second attempt required each archive to contain some minimum number of objects, on the theory that a real backup is not empty. That immediately failed several databases that were legitimately empty, and revealed the deeper problem: a "the file is structurally valid" check passes happily on a file containing nothing. Validity and content are different questions, and if you only ask the first, an empty result looks like a healthy one.

The check read the beginning and stopped. The third version worked on every small database and failed on the largest and most important one. The reason was mundane plumbing - the verification only needed the front of the archive to reach its verdict, so it stopped reading there, and the process feeding it the rest of the file died mid-sentence and reported an error. Every trivial database passed. The one that mattered most failed, for a reason that had nothing to do with the backup itself.

Verification is code, code has bugs, and a broken verifier is worse than an honest one - it will either reject good backups or, in a slightly different arrangement, bless bad ones.

Nobody Was Checking the Job Before It

Then there is the handoff. The nightly dumps land on a second machine, and a separate job on that machine ships everything off-site. Two jobs, two hosts, one chain.

Nothing connected them. The off-site job snapshotted whatever was sitting in the folder and uploaded it. On a night when the first job failed, the second would have found yesterday's copies still sitting there, uploaded those, and exited green - so the off-site record would show a successful backup every single night while quietly freezing in place.

Two jobs chained together are fine. A second runner who starts sprinting without checking whether anyone handed him the baton is not.

The off-site job now refuses to run unless three things are true: the folder exists and has contents, there is a set of copies carrying today's date, and today's count is not lower than yesterday's. That last one is the cheap version of the coverage problem from earlier - yesterday's count is a free baseline, and it turns a database silently dropping out of the rotation into a red alert the next night instead of a year of silence.

A Schedule That Meant the Wrong Time

A small one, but the kind that survives for years unnoticed.

Two machines, two different time zone settings, one job scheduled for 4am. On the second machine, "4am" meant 11pm the previous day - four hours before the job whose output it exists to ship. It would have run on time by its own reckoning and shipped stale data forever while reporting perfect health.

I caught it the same day, by luck rather than process. The rule applies anywhere: any schedule that spans two machines needs its time zone written down explicitly, not assumed. The default is whatever the operating system was installed with, and that is not a decision anybody made.

Restoring Is Where You Learn What You Actually Have

Two restore rehearsals taught me more than all the verification work combined.

The first one looked like a corrupted backup. I restored a copy into a scratch database, compared it to the live system, and found pieces missing - structures the running database clearly had and the restored copy did not.

It was not damaged. It was old. The backup had been taken before a schema change deployed, so it faithfully represented the database as it existed the night it was captured. Everything was working as designed.

That is a genuinely dangerous trap, because the natural reaction to "the restore is missing things" is to conclude the backup system is broken and rebuild it. Compare a restore against what the system looked like when the backup was taken, not against what it looks like now. Otherwise every backup older than your most recent change looks corrupt, and you will either waste days chasing a phantom or lose confidence in a backup set that is perfectly good.

The second rehearsal was the more valuable one. I restored from the off-site copy specifically, rather than from a fresh local dump, because the off-site copy is the one that matters on the day the building is gone. And I checked one thing in particular: whether the rules controlling who is allowed to see whose data came back intact.

Those rules live inside the database itself rather than in the application code, which makes them exactly the sort of thing a restore can drop quietly - the tables come back, the row counts match, everything looks right, and the boundary between one customer's data and another's has silently evaporated. They came back. But I only know that because I looked, and I only looked because I asked what a restore could plausibly lose without making any noise about it. That question is worth asking about your own systems.

The Gap I Still Have

My backups are now checked for coverage, verified for content, chained safely between hosts, shipped off-site, and alerted on. Nothing has ever been restored on a schedule.

Two restores exist. Both were one-off, both were done by hand, both because I decided to do them. There is no recurring test that takes the newest backup, puts it into a throwaway database, compares it to the source, and tells me when that stops working. Until there is, what I have is a system that proves my archives are readable - further than most setups go, and still not the thing I need.

Readable is not restorable.

What a Business Should Take From This

You do not need to audit your backup scripts to find out whether you have this problem. Three questions will tell you almost everything:

When did we last restore something? Not "when did the backup last run" - when did a human take a backup, put it back, and confirm the result was correct? If the answer is "at setup" or "never," the backups are unproven, however green the dashboard is.

What would tell us a backup silently stopped working? Every failure in this post reported success. If the only signal you watch is the absence of an error message, you are blind to the category of failure that matters most - the one where the job runs happily and captures nothing. You need something that alerts on a backup being smaller than it was, missing a system it covered last week, or simply not appearing.

Does anything verify the backup ran before the copy of it gets shipped somewhere else? If the off-site step just uploads a folder, it will faithfully upload a stale folder for as long as the folder sits there.

None of this is exotic or expensive. Most of what I fixed was small logic changes to jobs that already existed - the same pattern as most of the infrastructure behind this site, where the cost is attention rather than licensing. What it requires is deciding that "the backup job is green" is the start of the conversation rather than the end of it.

If you are not certain what your backups actually contain, or nobody has restored one since the day they were configured, that is a good thing to find out on a quiet afternoon rather than a bad one.