A Green Dashboard Is Not a Working System

A background service on a server I run quietly stopped doing its job for nine days. It was still running, and its health check stayed green the entire time. Nothing alerted, because as far as every dashboard I owned was concerned, nothing was wrong.

TL;DR: A health check only tells you the answer to the question it was written to ask, and most of them ask an easier question than the one that matters. Monitoring is really three separate jobs - noticing a problem, deciding who to tell, and getting the message to a human - and each one fails in its own way. If you have never watched an alert actually arrive on someone's phone or in someone's inbox, you do not have alerting. You have a decoration.

What Is a Health Check?

A health check is a small, repeated test that asks a service "are you okay?" If the answer is yes, a light stays green. If it is no, something is supposed to happen - a restart, an alert, a page.

The important part is that the check is a question somebody wrote by hand. It has no idea what your service is for. A check that asks "is this program still running?" will stay green while the program sits there doing nothing useful, because the program is, in fact, still running.

The Check Was Asking an Easier Question

Here is what happened on my own system. The service runs inside a container whose network traffic is routed through a separate connection layer. That layer moved to a different endpoint, which is routine and happens on its own schedule. The service's internal connection state did not survive the move. It kept running, kept accepting work, and completed none of it.

The health check tested whether the container could reach the internet. It could. The connection layer was healthy, general traffic flowed fine, and the light stayed green. What the check never tested was whether the service was doing the one thing it existed to do. Nine days went by. I found it by noticing that work I had queued had not finished, not because anything told me.

Restarting it fixed the problem in under a minute, which is the irritating part. No hard failure to diagnose, no corrupted data, no difficult repair. Nine days of nothing, undone by a restart, because the check asked whether the lights were on instead of whether anybody was home.

The fix was to rewrite the check so it asserts the thing that actually indicates work is happening - an internal counter that drops to zero when the service is dead in exactly that way - and to wire the failing check to an automatic restart. Same service, same dashboard, different question.

Ask what your check would have to see to go red, then ask whether the failure you are actually afraid of would produce that.

Monitoring Is Three Jobs, Not One

The usual mistake, mine included, is treating "monitoring" as one thing you either have or do not have. It is three, and they fail independently.

Detect. Something notices a problem. This is where the wrong-question failure lives. Detection can be perfectly reliable and still be pointed at the wrong layer.

Route. Something decides who cares. A disk filling up on a test machine at 2am is not the same event as a customer-facing site going down, and if both produce the same alarm, people learn to ignore both.

Deliver. Something puts the message in front of an actual human. Almost nobody tests this step, and it is the one that silently breaks most often.

Large organizations run three separate tools for these. A small business can perfectly well have one service doing all three - but you should still be able to say which part handles which job, because when it fails you need to know which of the three broke.

The Alarm Cannot Live Inside the Building

There is one rule that governs all of this: the alarm path must not depend on the thing it is watching.

If your monitoring tool runs on the same server, in the same rack, on the same internet connection as the systems it watches, it cannot tell you the building is dark. It is inside the building. When the power goes out, the thing that was supposed to report the outage went out with it. I have written before about what a power outage exposes in a server room, and this is the same shape: the safeguard and the thing it protects share a single point of failure.

So the signal answering "did this job run at all?" lives outside my network entirely, on a hosted service with no dependency on my hardware, power, or internet. Everything else can be local. That one cannot.

Two Failures, Two Different Alarms

There are two completely different ways a scheduled job fails, and they need two completely different mechanisms.

The job ran and broke. It started, hit an error, and stopped. This one is easy: the job is present at the moment of failure and can report its own bad outcome. Most people cover this.

The job never started at all. The schedule got disabled during maintenance. The machine was off. A power cut ate the window. There is no failure to report, because there was no run. There is only silence.

The second is the dangerous one, and almost nothing covers it by default. Silence looks exactly like success. This is how a business discovers, at the worst possible moment, that its nightly backup stopped running a year ago and nobody was told, because "no error message" felt like good news.

That failure needs an inverted mechanism: instead of waiting to hear about a problem, something outside expects a check-in on a schedule and raises the alarm when it does not arrive. Absence becomes the signal.

The Alerting Path I Built, Measured, and Deleted

I wired up a push-notification service so failures would reach my phone. It was configured correctly. Every test message reached the server - I confirmed that directly.

Not one ever reached my phone. Messages appeared only when I opened the app and manually refreshed. Notification permissions were on the whole time. The documented fix from the project's own known-issues page changed nothing. It turned out to be a long-standing upstream problem on that platform, reported by other users over several years, not a mistake in my setup.

So I removed it completely. Not paused, not left in place with a note - removed, including every hook that fed it. A dead alerting path that still looks wired up is worse than no alerting at all, because it buys you the confidence of coverage you do not have. Alerts moved to email, which is unglamorous and actually delivers.

The measurement was the valuable part. Had I never tested delivery end to end, I would have gone on believing I was covered.

Three Quiet Ways Alerting Turns Itself Off

None of these produced an error. All three of them happened to me.

A cleanup routine ate the reporting step. A later piece of code that ran when a job finished silently replaced the piece responsible for reporting the outcome. Jobs announced they had started and never reported finishing. The status sat permanently at "waiting for first result," which reads as configured-and-fine rather than broken.

A permissions change made a settings file unreadable. Tightening file permissions left a small configuration file unreadable by the account the jobs ran under. The code treated "cannot read this file" and "this file does not exist" as the same case - deliberate, so jobs would still work before alerting was set up. Every alert on that machine stopped instantly while every job kept reporting success.

A monitor that could never go green. I added a check for something that, by design, could not be reached from where the monitor sat. It was permanently red. Within days I was ignoring that dashboard's colors entirely. A monitor that can never go green trains you to ignore red, which costs you the monitors that were working.

What Monitoring From Inside Can Never See

One limitation worth admitting plainly, because it applies to almost every small-business setup I have seen: monitoring that runs inside your own network proves your systems are up. It says nothing about the path your customer takes to reach you.

An expired certificate at the edge, a DNS record pointed at the wrong place, a misconfigured front door - any of these leaves every internal light green while your site is unreachable from the outside. My own site is self-hosted on hardware I own, and everything between my front door and a visitor's browser sits outside what internal monitoring can see. Covering that needs a check that runs from somewhere else and reaches you the way a customer does.

Questions Worth Asking Your Own Team

You do not need a full audit to find out whether your monitoring works. Three questions will do it.

What would tell us a scheduled job stopped running entirely? Not failed - stopped. If the honest answer is "we'd notice eventually," you have a silence problem.

Has anyone actually watched an alert arrive? Not "it's configured." Watched it land, on a real device, recently.

Does anything check our site the way a customer reaches it? From the outside, through the same certificate and DNS a visitor uses.

Uncomfortable answers here are normal, and usually cheap to fix. If you want a second set of eyes on what your dashboards are actually proving, let's talk.