Your Release Process Should Not Be Able to Ship Untested Code
The pipeline ran the full test suite against a specific version of the code. Every test passed. Then it logged into the server and told it to pull down the latest code and build that. Those are two different things, and I had been running it that way for months without noticing.
TL;DR: A release pipeline that tests one version of your code and then builds "whatever is newest" can ship code that was never tested, and nothing about it looks broken. The fix is to make the deploy check out the exact version that passed, add an automated health check against the live site that rolls back on its own when a release is bad, and then deliberately break every safety check once to confirm it can actually fail. On one of my own projects, a bad release rolled itself back in 17 seconds and no visitor ever saw it.
What Is a Deploy Gate?
A deploy gate is an automatic check that runs immediately after new software is released, against the real running site, before that release is declared good. It asks a simple question: is the thing customers are about to use actually working? If the answer is no, the gate refuses the new version and puts the previous known-good version back, without waiting for a human to notice.
That is different from testing. Tests run on a build machine, in isolation, before anything is released. A gate runs after, against production, on the real thing. Most small and mid-sized companies have some form of the first one and nothing at all resembling the second. When a release goes bad, the alerting system is a customer phone call.
The Bug I Had Been Shipping With for Months
Here is the shape of the problem, in plain terms.
When I merge a change, an automated pipeline picks up that exact version of the code, runs the tests, the security scan and the quality checks against it, and if everything is green it moves on to the release step. The release step connected to the server and, in effect, said: fetch the newest code from the repository and build it.
Nothing enforced that "the newest code" and "the code we just tested" were the same thing. Nine times out of ten they were, because I had merged one change and nothing else landed in the two minutes that followed. But if a second change merged while the first pipeline was still running its tests, the build that reached production was the second one, and the second one had been tested by nobody. The pipeline still reported a clean, green, successful release. Every log line said what it was supposed to say.
That is the part worth sitting with. This was not a bug that surfaced as a failure. It was a gap that could only ever surface as a silent wrong answer, on a random day, under timing you cannot reproduce on demand. It had been that way by construction from the day the pipeline was written. I found it by auditing the pipeline against a written standard, not by hitting it.
The fix is small and boring. The release step now receives the precise version identifier that the tests ran against, fetches that one version by its full commit ID, and builds that. If the repository has moved on, the deploy does not care. The image it produces is named after that same version, so the thing running in production can be traced back to the exact set of tests that approved it. Same pipeline, same steps, one less way to be wrong.
I had this same class of gap on more than one project, because I had copied the pipeline. That is the other lesson: a defect in a template is a defect in everything built from the template, and the way you find it is by reviewing the template, not by watching for symptoms.
A Test Suite Is Not a Release Process
Passing tests tell you the code behaved correctly on a build machine, with test data, in a controlled environment. That is genuinely valuable and it is not the same claim as "the thing customers use came up and works."
Plenty of failures live in the space between those two claims. A configuration value that exists on the build machine and not the server. A database that did not come back up in the right order. A dependency that was fine in the test environment and is a different version in the image. An application that starts, listens, and refuses every request. Tests cannot see any of that, because tests never touch it.
So the release step now ends with an automated check against the live site. It requests the real pages, in a real browser, and confirms they actually paint the elements they are supposed to paint. It watches for security policy violations and blocked resources while those pages load. If any of it fails, the release is refused and the previous known-good version is put back automatically, without me in the loop.
I have measured this on a real failure rather than a hypothetical one. On one of my own projects I deliberately pushed a broken release, and the gate caught it, rolled back, and had the site serving the previous good version again in 17 seconds. Total public disruption was under a minute. Nobody was paged. Nothing was decided under pressure at an awkward hour. It is the same principle as any other redundancy: the value is not that it exists, it is that it works when it is needed, and the only way to know that is to have watched it work.
One detail that matters more than it sounds like it should: the gate has to reach the new version before the public path does, and the rollback has to live in the same automated step as the check. If your health check runs somewhere else and files a ticket, you have monitoring, not a gate. Monitoring tells you about the outage. A gate ends it.
The Part Everybody Skips: Prove the Check Can Fail
This is the habit I would most like to hand to other people, because it costs almost nothing and it is almost never done.
Every new check I added got deliberately broken once, on purpose, to confirm it went red. Plant the defect the check is supposed to find, watch it fail, remove the defect, watch it pass. Only then is the check allowed to be trusted.
Two examples from my own systems explain why.
The first was a readiness check that had only ever been observed passing. It reported, on failure, that the site "did not respond within 60 seconds." That message was false. It made sixty attempts and nothing bounded how long any single attempt could wait, so when I actually stopped the database to see what would happen, the requests did not fail fast, they hung. Sixty attempts times an unbounded wait is not a minute. It is however long anyone is willing to stare at it. The check had been correct on every green run and its failure path had never once been exercised, which is exactly the condition under which a comforting message goes unquestioned.
The second was a check that compares a live database against what the code says the database should look like. I tested it by deliberately introducing a difference, ran the check, and it reported everything clean. Good news, except it was not: the change I planted referenced something that did not exist, the operation failed, the error was swallowed, and the check carried on and reported a clean result. "I looked and found no problems" and "I never actually looked" produced identical output.
That is the whole argument. A green check has two possible meanings, and you cannot tell them apart by looking at the green. Breaking it once tells you which one you have. It took a few minutes per check and it changed my confidence in the entire set from assumed to demonstrated.
The same reasoning applies to counts. A scan that reports zero problems is meaningless if it also inspected zero things. Whenever a check produces a "clean" result, I want it to also tell me how much it examined, so a broken probe cannot masquerade as a clean bill of health.
The Release That Broke, and Why That Was the System Working
Shortly after all of this went in, a change passed every test, merged, and the deploy failed.
The application would not start. The health check made its requests, got nothing back, spent its entire budget waiting, refused the new version and put the previous one back. The site served the old build the whole way through. No visitor saw a thing. I found out because the pipeline told me, not because anyone complained.
The cause is almost funny. Databases change shape over time, and those changes are applied as a numbered series of small scripts. Because a change that has already run against a live database must never be quietly rewritten afterwards, the system takes a fingerprint of each script's contents when it applies it, and refuses to start if the bytes of an already-applied script ever change. That guard is correct and I want it there.
What I had done, as part of tidying things up, was edit a comment inside a script that had been applied to production months earlier. Just a comment. No functional change whatsoever. The fingerprint does not know or care what a comment is. It saw different bytes, concluded that history had been rewritten underneath it, and refused to boot. Comments count.
Two things about that are worth saying plainly, because the tidy version of this story would leave them out.
First, my initial theory was wrong. I decided the problem was a startup timing issue between the database and the application, and I was not imagining it: that was a real defect, it could genuinely have caused a failed deploy, and I fixed it. It was also not what was happening. I re-ran the deploy with the fix in place and it failed in exactly the same way, which is the moment the guessing had to stop.
Second, the reason the guessing lasted as long as it did is that the application's fatal log line said only that it had failed to start, with an empty space where the cause should have been. The error object was being logged under a name the logging library did not recognize as an error, so it serialized to nothing. The actual message, which named the file and said the fingerprint did not match, appeared the instant I ran the failing version by hand and looked at its output directly. That should have been step one. When something will not start, read what it says on the way down before theorizing about why.
The failure itself is the best evidence I have that the work was worth doing. A change that passed every automated test still broke production, and production never noticed, because something downstream of the tests was watching the real site and had the authority to undo the release on its own.
What This Means for a Business
If your company ships software, or pays someone to ship it for you, there are three questions worth asking. None of them require you to be technical, and the answers tell you a great deal.
Does the deploy build the exact thing that passed the tests? Not "the latest code," not "the main branch," not "the current version." The specific one. If the answer involves the word "latest," you have the gap I described at the top of this post, and it will produce a bad day at a time you cannot predict.
What happens automatically when a release is bad? If the answer is that someone notices and then decides what to do, your recovery time is however long it takes a human to be told, be available, and be confident. If the answer is that the release is refused and the previous version comes back on its own, your recovery time is measured in seconds and does not depend on who is awake.
When did we last watch a safety check fail on purpose? If nobody can name a date, the honest status of those checks is unknown rather than passing. A check that has only ever been seen green has not been tested; it has been observed.
None of this requires an expensive platform or a dedicated team. My own automation and delivery work runs on ordinary open-source tooling and modest hardware, most of it the same self-hosted stack this site runs on. What it requires is treating the release process as a system with its own failure modes, rather than as plumbing that either works or does not.
If you are shipping software and you are not sure what your pipeline actually promises, that is a good thing to find out on a calm afternoon rather than during an outage. Get in touch and we can walk through it.