Server updates

[exxos]

Server RAM fix — RAM disks that weren't actually in RAM

Bit of a facepalm one this. The forum uses RAM disks (tmpfs) to hold the page cache, so bot floods get served straight from memory instead of hammering PHP and the database. That's the whole point of them — memory is fast, disk is slow.

Except the kernel had quietly paged a large chunk of those RAM disks back out to the swap file... on disk. So my "RAM cache" was partly a disk cache. Worst of both worlds: I'd spent the memory budget on it AND still took the disk hit, plus swap latency on top.

Digging into why, it turned out MySQL was hoarding around 270MB for an internal diagnostics feature (performance_schema) that I've literally never used. That wasted memory created the pressure that pushed my RAM disks into swap in the first place. Turned it off and MySQL dropped from 1.3GB to 284MB.

Then I pinned nginx, PHP and MySQL so the kernel is now physically forbidden from swapping them out, ever. Swap usage went from 1.2GB to zero, and the RAM disks are genuinely resident in RAM again.

Net result: about 1GB of memory reclaimed, more PHP workers available for when the bots do get through, and no VPS upgrade needed after all. Which is nice, as I was about to reluctantly pay for one.

[exxos]

Server maintenance roundup

Spent a long day going through the server from top to bottom. Bit of a marathon, but the results were worth it.

Performance

Nginx CPU usage dropped from around 12% to 3%. PHP dropped from 12% to around 3.5% after finding the wiki cache wasn't being kept in memory properly. So the server is now doing roughly a third of the work it was doing this morning for exactly the same traffic.

Also reclaimed about a gigabyte of RAM from the database. It turned out a diagnostics feature was quietly reserving hundreds of megabytes it never actually used, and that wasted memory was creating enough pressure that the kernel was pushing my RAM disks out to the swap file. So my "RAM cache" was partly sitting on disk, which is the exact opposite of the point of having one. Turned the diagnostics off, pinned everything that matters so it can physically never be swapped, and swap usage went from over a gigabyte to zero.

The upshot is I was about to reluctantly pay for a bigger VPS, and now I don't need to.

Security certificate change

The site was using an RSA certificate, which is fine, but every time a bot connects it forces the server to do a fresh cryptographic handshake, and RSA handshakes are expensive. Given the bots open a new connection for almost every single request and never reuse them, the server was spending a big chunk of its CPU just doing handshakes.

So I added a second, modern ECDSA certificate alongside it. Browsers and bots that support it get the cheaper one automatically, and anything old still falls back to RSA, so nothing breaks. That single change is where most of the nginx CPU saving came from. Bigger than I expected, honestly.

Email deliberately still uses the old RSA certificate, because mail servers across the internet are a much more mixed bag than browsers, and I'd rather not discover the hard way that somebody's ancient mail server can't talk to mine.

An SSL Labs oddity

While testing the certificates, SSL Labs kept marking the site down a grade because it "couldn't connect" over IPv6. Spent far too long chasing that one, assuming it was my firewall blocking them.

It wasn't. IPv6 works fine here, other IPv6 test sites connect without any trouble, and the server happily serves thousands of IPv6 visitors an hour. Turns out their scanner sends the request from an address range that isn't actually announced to the internet, so my server's reply has nowhere to go and their end just times out.

So it appears to be a bug on their side rather than mine. Gave up chasing it in the end. The grade is fine and nothing is actually broken.

Automatic flood defence

The script that was supposed to warn me about high CPU turned out to be pretty poor. It spent over ten minutes deciding whether there was even a problem, ran every half hour, and was therefore actively grinding away about a third of the time. It also reported itself as the biggest CPU user in its own alert emails, which gave me a good laugh once I spotted it.

Rewrote it properly. It now reads from the new monitoring instead of measuring from scratch, so it makes its mind up instantly rather than taking ten minutes. That means it can check every five minutes instead of every thirty, so it reacts to a flood far sooner.

The bigger change is what it does when it fires. If the CPU is genuinely pinned for a sustained period, it now assumes we're under attack and automatically runs every flood detection and blocking script I have, one after another, in order of how quickly each one is likely to help. It keeps doing that on every check until the load comes back down.

It's also properly locked, so if a blocking run takes an hour (and some of them can) it won't stupidly start a second copy on top of the first while it's still working.

So in theory, if a big flood lands while I'm asleep or away, the server should now start fighting back on its own rather than sitting there waiting for me to notice.

Unban request bug

Also fixed a daft bug in the banned IP checker. If you requested an unban, it looked your username up in a way that was case sensitive, so "Badwolf" would come back as "user not found" even though the account obviously exists. Worse, the page was pre-filling the box with the all lowercase version of your own name, so it was effectively guaranteeing its own failure.

The IP was always actually unbanned, so nobody was left stuck. It was purely the notification email lying to me about who it was for. Fixed now, and it handles any capitalisation you throw at it.

Housekeeping

Found several leftover debugging scripts still running from previous late-night troubleshooting sessions. One had been running for nearly four days, taking a snapshot of the system every single second, and had quietly written a 334MB log file that was never cleaned up. Ironically, a script written to watch for high CPU had itself become one of the biggest CPU users on the box.

Another had been trying to connect to the database every second for days, and had never once worked because a password field was still left as a placeholder. It just logged the same error a few hundred thousand times.

All of them stopped, and around half a gigabyte of pointless log files removed.

Also rewrote the monitoring properly. It now samples everything once a second, costs a fraction of a percent of one CPU core, and can tell me at a glance what the server is actually doing, including automatically labelling any CPU spike with whatever caused it. So a scheduled overnight job now shows up as a scheduled overnight job, rather than a mystery spike I have to go digging for at midnight.

Plus a handful of smaller bug fixes along the way.

Nothing here changes anything for members. The site should just be a bit quicker.

[mrbombermillzy]

Hey @exxos just noticed clicking on links from site search results sometimes seem to go to the wrong page..(?)

e.g. If I search and get this page:

Screenshot 2026-07-14 at 07-45-06 exxos's Atari Amiga & retro forum - Search.png

Clicking on either the title link or the 'jump to post' link on the far right goes to:

Screenshot 2026-07-14 at 07-45-37 C103175 REV.1.1 - exxos's Atari Amiga & retro forum.png

Ive displayed the exact search terms entered for your information, as bizarrely I couldnt get it to fail on the 2nd try and different searches with other keywords/results all seems to link fine and I cant seem to replicate it. (Luckily I took the pictures as proof as soon as it happened).


:dizzy:


Sorry to cause you any extra work. :) :hide:

[exxos]

Thanks for the report @mrbombermillzy . Should be fixed now. It's the "logged out saved pages" which still being a bit of a problem at times.

[exxos]

My new stats seem cool. Just trying to figure out why the bot net suddenly stopped again. Same as before, just seems to turn off instantly , for no apparent reason.

Capture.PNG

[stephen_usher]

My new stats seem cool. Just trying to figure out why the bot net suddenly stopped again. Same as before, just seems to turn off instantly , for no apparent reason.

The crims probably ran out of funds to pay for the botnet service.

[exxos]

The crims probably ran out of funds to pay for the botnet service.

Could well be !

[mrbombermillzy]

Thanks for the report @mrbombermillzy . Should be fixed now. It's the "logged out saved pages" which still being a bit of a problem at times.

Good job! Thanks. :)

And hooray about the drop off in botnet news! :cheer:

[exxos]

Server work: a proper clear-out of the blocking system

I have spent today going right through the server's anti-bot defences, and it turned into a much bigger job than I expected. A few things worth explaining.

1. Some blocking rules were catching genuine members

The forum has been under fairly constant botnet attack for a long time now, and over the months I have built up a lot of automatic blocking to cope with it. Going back through it properly today, I found a couple of the rules were occasionally flagging real people rather than bots.

The causes were daft in hindsight. One rule treated a slightly malformed request as an attack, when really that is just what happens on a mobile signal that keeps dropping or an older connection. Another matched a text pattern that turns up in the forum's own stylesheet addresses, so you could be flagged simply for loading a page properly.

The good news is that nobody actually lost access. Regular logged-in members are automatically whitelisted, and the whitelist overrides the blocking, so the five members these rules flagged carried on browsing without noticing a thing. I can see their traffic continuing straight through. The safeguard did exactly what it was built to do.

The reason it still matters is that the whitelist only knows the address you were last seen on. If your connection handed you a new address, which happens constantly on mobile, the whitelist would not have caught up and you could have been locked out with no idea why. To you it would just look like the site was down, and you would have no way of telling me, because you could not reach the site to tell me.

2. So the whole approach has changed

Home broadband and mobile connections in the UK, Europe, the USA and so on can no longer be blocked by address at all. Anything suspicious from those networks now gets dealt with request by request instead.

The reasoning is simple once you see it. A mobile address is not one person, it is thousands of people sharing one address at the same time, so blocking it does not stop a bot, it removes a chunk of the country. And a home broadband address changes hands regularly, so a permanent block stops being about the machine that misbehaved and starts being about whoever inherits that address next month.

The other half of it is that the page caching I put in earlier this year has changed the maths. Bot traffic used to genuinely hurt, which is why the blocking got as aggressive as it did. Now most of it is absorbed by the cache at almost no cost, so there is simply no need to be that heavy handed any more.

I went back through everything that had been blocked over the months and released anything that looked like a real person or a real ISP. It came to a lot of addresses. Actual botnet traffic, the machines that have never once loaded a page successfully, stays blocked.

3. A lot of the old defences have been retired

This is the part that surprised me most. Over the years I have built up a big pile of individual rules, each one written in response to some specific attack at the time. Going through them today, most had quietly become dead weight.

Some were solving a problem that simply stopped happening months ago and had not fired since. Some were duplicating work that newer, smarter scripts now do far better. And one had not actually done anything at all for a very long time, because it was looking in a file that never contained what it was looking for. Nobody had noticed, because a rule that catches nothing looks exactly like a rule that is doing its job perfectly.

The bigger picture is that the nature of the attacks has changed. The old style of defence is built around spotting one badly behaved machine and shutting it out, and it is still good at that. But hardly anything attacks that way any more. It is nearly all botnets now, thousands of home computers around the world each making a handful of requests, none of them individually doing anything that looks suspicious. You cannot catch that by watching one address at a time, and the newer scripts I have written look at the wider patterns instead.

The trouble is that an old rule that no longer catches anything is not free. It sits there, still armed, still able to make a mistake, and the only thing it can realistically catch these days is a real person doing something slightly unusual. Which is exactly what happened. So a good number of them have now been retired, and the ones that remain are the ones that still earn their keep.

The whole setup is considerably simpler than it was this morning, which is usually a sign you are heading the right way.

4. Guests were occasionally served a broken page

The spam filter briefly checks new visitors before letting them through. Every so often the page it showed during that check was being saved by the forum's page cache and then handed out to everyone else, so instead of the forum you would get a mostly blank page.

It was maddeningly intermittent. I had a script watching for it around the clock and it finally caught one red handed at 2am, with all the evidence saved, which made the cause obvious.

It is properly fixed now, and fixed in the right way. Previously I was trying to spot and reject bad pages one type at a time, which only works until something new turns up. Now the cache will only store a page if the forum itself confirms it genuinely produced that page. Anything else, the spam check, an error page, or something that does not even exist yet, simply cannot be cached.

5. An SEO crawler was eating nearly half the server's traffic

While going through the logs I found that one commercial SEO crawler was responsible for around 40% of everything the server did in a day. Roughly 29,000 requests, from three machines, and not one of them achieved anything, because every request was being redirected and it never followed the redirect.

To be clear, this is not a search engine. It sends nobody here. It crawls the forum, puts the contents into a private database, and sells access to that database to its own customers. It takes and gives nothing back.

Blocked. Real search engines are of course still welcome, and so is anything that actually sends people here.

6. Odds and ends

There was also a long standing quirk where certain badly formed web addresses were being served as though they were normal, which meant the same page could exist twice as far as the cache was concerned. That is now tidied up on both the forum and the wiki.

In short

Nobody lost access, but they could have done, and that is now much harder. The caching is trustworthy again. The blocking is a lot better at telling a person from a bot, and a lot less willing to punish an entire network for the behaviour of one machine on it.

If anything looks wrong, or you have had odd trouble reaching the site recently, please say so.

[exxos]

Botnets never seem to appear when you want them to ! :lol: :roll:

We been pretty mellow the past few days now.

Capture.PNG

The CPU spikes are "background tasks" running mostly. Generally just max out a CPU core for a few seconds. The connections is the interesting one... now its just a waiting game I guess again...
:ball: