When Your Own Security Setting Takes You Offline

A rate limit is supposed to stop one badly behaved visitor from hammering your website. On one of my own sites, it was stopping the entire internet at once. Not because it failed. Because it worked exactly as configured, and the configuration was wrong in three separate ways that each hid the others.

TL;DR: A rate limit set to twenty requests a minute was counting every visitor on earth as a single visitor, which meant one normal page load could lock everyone out for a minute. Three stacked misconfigurations caused it, and on a second site the limit could be reset at will by the person being limited. Security settings do not announce when they are wrong. You have to go look.

What Is a Rate Limit?

A rate limit is a cap on how many requests one visitor can make to your website in a given amount of time. Ask for ten pages a second and the server starts refusing you until you slow down. It is one of the cheapest protections you can put on a public site: it blunts brute-force login attempts, slows down scrapers, and stops a single machine from soaking up all your capacity.

The whole idea rests on one assumption. The server has to be able to tell one visitor apart from another. If it cannot, the limit stops being a shield and becomes a tripwire strung across your own front door.

Twenty Requests a Minute, For Everybody

The setting on my site said twenty requests per minute. That sounds reasonable if you picture it as twenty requests from one person. It was not. It was twenty requests per minute for every visitor on the planet, combined, pooled into one shared allowance.

Here is why that number is so much smaller than it looks. When you load a web page, your browser does not make one request. It makes one for the page, then more for the stylesheet, the fonts, the logo, each photo, and the small script that runs the contact form. A single cold page load on my site pulled roughly a dozen files. One visitor arriving for the first time was enough to eat most of the shared budget by themselves.

I confirmed it live rather than reasoning about it. I fired a burst of ordinary requests at the homepage. The first thirteen came back fine. Every one after that was refused. Not refused for me specifically. Refused for anyone who happened to show up in the next sixty seconds. The site was taking itself offline, quietly, most likely already doing so in production, and nothing anywhere reported it as a problem. From the outside it just looked like the site was occasionally slow or broken.

Three Bugs Stacked On Top Of Each Other

Once I started pulling the thread, the single wrong number turned out to be three unrelated mistakes that happened to line up.

The first was an option that did not exist. The configuration included a setting meant to exempt static files like images and stylesheets from the limit, so only the parts that actually needed protecting would be counted. That option was not a real option in the version of the library I was using. It was not rejected, it was not flagged, it was not logged. It was silently ignored. The limit applied to every image and every stylesheet, which is precisely how one page load turns into thirteen counted requests.

The second was the identity problem. The site sits behind a proxy that terminates encryption and routes traffic, which is a normal and good arrangement, and something I have written about in how this site is self-hosted. But the application was never told to look past that proxy to find the real visitor. So from the application's point of view, every request in the world arrived from the same place: the proxy. One visitor, forever, all of us. That is what collapsed the per-visitor limit into a single global pool.

The third only surfaced while I was fixing the other two, and it is the one I would not have predicted. Because of the way the limiter attached itself to the application, it only covered routes that were registered after it loaded. The contact form endpoint, the one part of the site that genuinely needed a limit, was registered before it. It had no limit at all. The static images, which needed nothing, were fully covered. The protection was applied exactly backwards from the intent, and the configuration file read perfectly correctly the whole time.

A Rate Limit an Attacker Can Reset

The second site I run had a worse version of the same problem, and it is worse in a way that is easy to miss.

On that one, the application had been told to trust the proxy. Good so far. But it had been told to trust the whole chain of hops in front of it, and to take the first entry as the visitor's identity. The network layer in front of my site appends to that chain rather than replacing it. Which means the first entry in the chain is not written by my infrastructure. It is written by whoever is making the request.

The identity used to count requests against a visitor was being partly supplied by the visitor.

I proved it in about thirty seconds. I sent three requests with three made-up identities attached. Each one came back with a fresh, full allowance. No decrement, no accumulation, no limit. A control request with nothing attached counted down normally, so the limiter itself was working fine. It was just being told a new lie each time about who was calling.

A rate limit that the person being limited can reset is not a rate limit. It is a status indicator. Anyone determined enough to attack the site would have sailed straight through it, while every ordinary visitor stayed subject to it.

The Thing I Found by Accident

While I was at it, I swept every public site I run to check whether any of them had the same identity problem. That sweep turned up something bigger than what I went looking for.

One of the sites was answering requests directly at its origin. The protective network layer in front of it, the one providing filtering and abuse protection, could be skipped entirely by anyone who knew where to knock. It was not a subtle misconfiguration in a security setting. It was a security layer that was simply not in the path. I would not have found it if I had not been auditing something else, which is the same pattern I ran into after a power outage, where the recovery went fine and the audit found the real problems.

Then came the part I keep thinking about. A later check showed that the sweep itself had been incomplete. Four live sites had been missing from the list since the day they launched. Not failing their checks. Absent from the checks. Every audit I had run had cheerfully reported clean results while never looking at them at all.

That is the durable lesson here, more than any of the individual bugs. Generate your check list from what is actually running, not from what you remember running. A host that is missing from the list is not flagged, not failed, and not fixed. It is skipped, silently, by every future check you run, and the report comes back green.

The Unglamorous Finding

There is one more, and it is not a security finding so much as a business one.

While reviewing the contact form on one of these sites, I opened its stored submissions. One hundred and twenty-five messages over five months. Every single one was spam. Link spam, scams, and templated bot noise. Zero were from real people.

The form worked. It had always worked. It rendered, it accepted input, it saved what it received. But nobody had ever looked in the box, so a channel that was returning nothing of value looked, from the outside, exactly like a channel that was working. That is a very easy assumption to make about any system that exists but is not checked: the phone line nobody calls, the shared inbox nobody reads, the alert address that stopped forwarding two years ago.

Three Questions Worth Asking

If your business runs a public website or a customer-facing application, these are worth putting to whoever maintains it. None of them require you to be technical.

Is our rate limit counting individual visitors, or is it counting everybody at once? If the answer is the second one, the limit is a self-inflicted outage waiting for a busy day.

Can a visitor influence how they are identified? If the identity used to count requests comes from something the visitor sends, the limit can be reset by anyone who cares to.

Is our list of things to check generated from what is actually running, or from memory? A green report that skipped four systems is worse than no report, because it tells you to stop looking.

None of these problems announced themselves. There was no alert, no error, no red dashboard. The settings looked right in the file. They only turned out to be wrong when somebody sat down and tested what they actually did.

If you would like a second set of eyes on what your public-facing systems are really doing, let's talk.