Server updates

[exxos]

Thought the real-time threat map by spamhaus was interesting https://www.spamhaus.com/threat-map/

Pretty much tallies with all the botnets I have posted about recently as well..

[exxos]

A quick note on the meta-externalagent blocks that went in today.

This is Meta's own AI/indexing crawler, separate from the one that generates Facebook link previews (facebookexternalhit), which is untouched and still works fine, so sharing links to the forum on Facebook is unaffected.

What's blocked: that specific crawler, identified by its user agent, hitting our download links. Each individual IP address was sticking close to a respectable 2 requests a second, roughly what a robots.txt-abiding crawler should do. The problem was the number of IP addresses doing it simultaneously, dozens of them all at once, adding up to combined traffic in the region of 1,000 requests a second, enough to start slowing the server down. Some of these are also caught and banned outright once they repeat.

What's not blocked: real visitors, including anyone arriving via a Facebook link, and any other legitimate search or AI crawler (Google, Bing, etc.) are untouched.

Some history on this one. Meta's crawler was banned outright a few months ago because the sheer volume was taking the server down. I lifted that ban to see if things had improved this time round, they had not, so clearly they are still sending far more traffic than any server should reasonably have to absorb. A couple of weeks ago I switched from an outright ban to just serving them 403s instead, but that only filled up the logs while still leaving several thousand connections a minute landing on the server, which is just noise for no benefit.

So meta AI bot is permanently banned again now, because it simply will not behave. For what it's worth I'm generally against banning AI crawlers on principle, I think they should have as much right to read public content as anyone else. But that only holds while they behave themselves, and Meta clearly is not going to.

[exxos]

Some of you will have noticed intermittent connection trouble to the forum over the past few days rather than just general slowness.

A couple of weeks back I retired a few of the fail2ban jails watching for certain floods, they'd gone quiet for a while and didn't seem to be catching anything worth keeping running. Typical timing, the same threats have started back up again since, so today involved putting a few of those jails back to work and adapting nginx directly to drop that traffic at the door rather than let it through.

Two separate floods were making things worse together. One was random probes hammering files and folders that don't exist on this server, hunting for exposed config and credential files. Every one of those still mints its own short-lived cache entry, so thousands of them in a row just fills the cache up with junk that nobody will ever ask for twice.

The other was the cache's own two hour refresh only running once a day rather than little and often, meaning a big pile of pages would go stale before the next top up. When a flood then lands on a stack of pages that have all quietly expired around the same time, every single one of those hits the backend directly to be rebuilt instead of being served instantly from cache, which is exactly the sort of pile up that causes the connection trouble some of you saw.

Both are sorted now, the junk probes get dropped immediately rather than reaching the cache at all, and the refresh runs every five minutes instead of once a day so far fewer pages are ever sat there stale waiting to be hit all at once.