exxos's Atari, Amiga & retro forum

Thought the real-time threat map by spamhaus was interesting https://www.spamhaus.com/threat-map/

Pretty much tallies with all the botnets I have posted about recently as well..

A quick note on the meta-externalagent blocks that went in today.

This is Meta's own AI/indexing crawler, separate from the one that generates Facebook link previews (facebookexternalhit), which is untouched and still works fine, so sharing links to the forum on Facebook is unaffected.

What's blocked: that specific crawler, identified by its user agent, hitting our download links. Each individual IP address was sticking close to a respectable 2 requests a second, roughly what a robots.txt-abiding crawler should do. The problem was the number of IP addresses doing it simultaneously, dozens of them all at once, adding up to combined traffic in the region of 1,000 requests a second, enough to start slowing the server down. Some of these are also caught and banned outright once they repeat.

What's not blocked: real visitors, including anyone arriving via a Facebook link, and any other legitimate search or AI crawler (Google, Bing, etc.) are untouched.

Some history on this one. Meta's crawler was banned outright a few months ago because the sheer volume was taking the server down. I lifted that ban to see if things had improved this time round, they had not, so clearly they are still sending far more traffic than any server should reasonably have to absorb. A couple of weeks ago I switched from an outright ban to just serving them 403s instead, but that only filled up the logs while still leaving several thousand connections a minute landing on the server, which is just noise for no benefit.

So meta AI bot is permanently banned again now, because it simply will not behave. For what it's worth I'm generally against banning AI crawlers on principle, I think they should have as much right to read public content as anyone else. But that only holds while they behave themselves, and Meta clearly is not going to.

Some of you will have noticed intermittent connection trouble to the forum over the past few days rather than just general slowness.

A couple of weeks back I retired a few of the fail2ban jails watching for certain floods, they'd gone quiet for a while and didn't seem to be catching anything worth keeping running. Typical timing, the same threats have started back up again since, so today involved putting a few of those jails back to work and adapting nginx directly to drop that traffic at the door rather than let it through.

Two separate floods were making things worse together. One was random probes hammering files and folders that don't exist on this server, hunting for exposed config and credential files. Every one of those still mints its own short-lived cache entry, so thousands of them in a row just fills the cache up with junk that nobody will ever ask for twice.

The other was the cache's own two hour refresh only running once a day rather than little and often, meaning a big pile of pages would go stale before the next top up. When a flood then lands on a stack of pages that have all quietly expired around the same time, every single one of those hits the backend directly to be rebuilt instead of being served instantly from cache, which is exactly the sort of pile up that causes the connection trouble some of you saw.

Both are sorted now, the junk probes get dropped immediately rather than reaching the cache at all, and the refresh runs every five minutes instead of once a day so far fewer pages are ever sat there stale waiting to be hit all at once.

Spent today hunting down a caching gremlin that let a few pages serve stale content well past when they should have refreshed. Turned out a caching header wasn't being handled quite right, which meant duplicate copies of the same page were quietly stacking up under different names, and the system got confused about which one was actually live. All fixed now (hopefully), and while I was in there I also rebuilt the background refresher so it only bothers pages that genuinely need it instead of hammering thousands every few minutes regardless. Should mean snappier pages and one less mystery for me to trip over at 3am.

Just found some more issues which I had not taken into account :roll: The basic floor is that the cache only helps on second hits. In the worst case when the cache is empty, a bot comes along and requests all pages all at once on the server goes down half an hour. Once the cache is built, everything after that is easy to keep up with. But then eventually the cached pages have to expire, in the 100's, and the flood comes back and takes us down again due to all the backend hits.

To prevent that, I rebuild the cache at a controlled rate instead of letting it expire naturally. The challenge is that you can’t refresh a cached page until it actually expires, so all expired pages still need to be rebuilt in one go. Doing that too fast creates an “internal flood” that overloads the server, and doing it too slowly causes timing drift where refreshes fall behind. It's a balancing act cannot be won ! :roll:

The new system which is WIP currently, will fix this by continuously cycling the cache at a gentle, rate‑limited pace of only a few pages per second,so everything stays fresh without ever overwhelming the backend. It’s basically a way of keeping the cache updated on our own terms rather than letting it explode all at once.

Basically annoying as it looks like will have to pay for another month of AI's subscription to work through all these problems :(

EDIT:

I had a "bug" with cleantalks extension which was causing huge cache problems as well. I did a "hack" to get around the problem a couple weeks ago as a workaround. I used AI to write a MD file on the problem and sent it to Cleantalk. They got back to me today and fixed the problem ! Not often I give praise to a company, but they deserve it considering I was likely their only customer who will run into that particular problem ! Their support has been pretty good overall, its why I stick with them. I've reported all sorts of weird bugs and even gave my own fixes to problems in the past. So I guess they feel more inclined to take my bug reports seriously :)