Interesting technique, credit where it's due. Giving the whole swarm one shared bucket instead of letting each IP have its own private allowance is the correct fix for that shape of problem, per-IP limits are basically toothless against anything spread across enough addresses.sporniket wrote: Tue Sep 15, 2026 6:56 am IIRC troed had posted his own insight for how throttling **some** swarms when the various IPs and whatnot are linked. https://blog.troed.se/posts/nginx-meta-rate-limiting/
Worth being clear about what it's actually doing though. At the volume he's describing, hundreds of thousands of hits a day, a 1 request/second bucket can only pass around 86,000 in a full day even with zero requests ever rejected. So most of that traffic isn't being throttled and served a bit later, it's just being dropped. Different status code to an outright block, same practical outcome for the bulk of it.
It also only works because Meta happens to be a neat, tidy case today, one company with a published, stable IP range you can write a geo block for. That's not even permanent for Meta itself. Google actually moved its own published crawler ranges to a new location a few months back, no redirect, old URLs quietly returning empty or stale data instead of erroring, so anyone whose rule pulled from the old address was running on a dead list without knowing it. Ranges are a feed, not a fact you encode once.
And most of what actually turns up here isn't even that tidy to begin with. We've had crawlers running off rotating rented cloud IPs with nothing stable to tag at all, one that renamed itself mid-week and could do it again any time it fancies, and a flood that avoided a signature-based blocker entirely just by not carrying the one parameter it was watching for. There's no bucket you can build for something that doesn't sit still long enough to be one, doesn't publish a range in the first place, or doesn't match the shape you built the rule around.
The sharper version of that problem: we've also had a flood spread across 163,000-odd distinct addresses, roughly one request per IP. No per-IP rate limit catches that, at any threshold, because the thing a rate limit actually depends on, repetition from one source, isn't there to detect. It only shows up at all once you stop looking IP by IP and look at the aggregate.
Even where bucketing does work, it's a bucket per bot, and the server only has so much room to hand out. Every bucket draws from the same shared ceiling, so the more of them you're maintaining, the smaller each one has to get to keep the total honest. Eventually you're maintaining fifty small buckets each dropping nearly everything from their own bot anyway, which is roughly what one block would have done with far less upkeep.
None of that is an argument against blocking abusive crawlers, to be clear, that part isn't in question, KeenableBot earned its block outright: read robots.txt once, ignored the one line in it, ran at ten times the asked rate. The trouble isn't the principle, it's leaning on correctly identifying a moving target as the method, and that's the part that keeps failing here specifically.
We went a fair way down the rate limiting road ourselves too, and it took a fair few different shapes of failure before we gave up on it. HTTP/2 was the first wrinkle, a browser opens one real connection but multiplexes dozens of streams over it, and each stream counts as its own "connection" as far as nginx is concerned, so limits tuned for the old one-connection-per-request world started catching ordinary visitors for no reason. From there the numbers just crept everywhere, different limits in different parts of the config, each one bumped up individually whenever it broke something, until there were several different values scattered around that nobody could really reason about as a whole any more.
Tidying that into one sensible global number didn't hold either, store pages with a lot of images needed it raised again and again, and it would still occasionally fail even after that. Meta's own crawler made the deeper point clearest: switching it from an outright ban to nginx 444, the lightest response nginx can give, still wasn't enough at the volume it was sending, because the TCP handshake and TLS negotiation happen before nginx ever gets to look at the request and decide to reject it. There's nothing lighter available at that layer. The only thing lighter than 444 sits below nginx entirely, at the network level, before a connection is even accepted. Add in that measuring whether any of it was actually doing anything turned out to be its own trap, connection limits and request limits fail with different status codes, so an audit that only checks one of them can quietly miss the other working, or not working, the whole time. By the end it was more effort to keep correctly tuned than it was worth.
TL;DR: clever idea, but it depends on reliably identifying the bot, and that's the part that keeps failing, making the traffic cheap did more for us than any rate limit ever managed.