Server updates

[exxos]

Sorting out an email problem that had been quietly broken for a while

Regular readers will know I've had an on/off battle with mail deliverability for a few years now, mostly aimed at Microsoft/Hotmail addresses (the main thread on that is here, and a related one here). That one's still unresolved on their end, and after this long getting nowhere with their support I've more or less accepted it's a "some day" problem rather than a "today" one.

While going through the mail server logs properly (something I don't do often enough, in fairness) I found something separate and rather more concerning: DKIM signing had been switched off on the server entirely.

For anyone not familiar, DKIM is one of three checks (along with SPF and DMARC) that receiving mail servers use to work out whether an email claiming to be from a domain actually came from that domain, or from someone pretending. DKIM specifically adds a digital signature to outgoing mail, signed with a private key that only the sending server holds, which the receiving server checks against a public key published in DNS. If it doesn't sign correctly, or isn't there at all, your mail looks a lot more like something a spammer would send, which does nobody any favours.

I was fairly sure I'd set this up properly years ago using a different DKIM system entirely, but going back through everything there's no evidence any other system was ever installed, it's always been the same one. Best guess is it started causing mail to hold or fail at some point, got switched off to make the problem go away, and then just got forgotten about, the way these things do when there's no obvious symptom pointing back at it. It stayed that way until Claude spotted it a few days ago while going through the logs for something else entirely.

Digging further turned up a second, older problem sitting underneath the first. This server sends mail for several domains, including the two exxos ones (exxosforum and exxoshost) and others, and all of them had been signing under a single DKIM key registered to exxosforum's own domain. That's fine for exxosforum itself, but for the rest it breaks DMARC alignment, since the signature doesn't match the domain the mail claims to be from. So even with DKIM nominally working, most of the mail leaving this box would still have looked suspicious to anywhere strict enough to check properly.

As for why it likely got switched off, the probable culprit was a missing setting (milter_default_action) that controls what the mail server does if the DKIM signing service doesn't answer in time. Left unset, it defaults to holding the mail rather than sending it, which is exactly the kind of thing that ends with someone switching the whole lot off in frustration rather than tracking down why mail was getting stuck.

Separately, the store side of things sends its own mail through PHPMailer rather than the forum's own mail path, and that turned up its own small pile of history while I was in there. There were two duplicate copies of it kicking about, both carrying the same manual patch I'd made a while back to stop it falling over when an email address or name came through empty. Both copies got consolidated down to one, and then that one was updated to the current official release, which has its own, slightly more thorough version of the same fix built in, so the manual patch isn't needed any more. While in there it also turned up that the store's own attempt at DKIM signing had been pointing at the wrong file the whole time, the DNS text record instead of an actual private key, so it had been silently failing on every single email without ever throwing an obvious error. Removed, since real signing now happens properly at the server level for everything regardless.

Fixed now, properly:

• A separate DKIM key generated for each domain, with new records published in DNS for each one
• The missing setting corrected so a slow signing check no longer holds mail hostage
• DKIM re-enabled and verified with real sent mail, headers checked across every domain, all passing
• PHPMailer tidied up to one current copy instead of two old ones, and the broken app-level DKIM attempt removed

One last, smaller thing found while going through everything: the wiki's www address (www.atari-wiki.com) has no IPv4 record at all, only IPv6, so anyone on an IPv4-only connection can't reach it if they land on the www version. Since that domain isn't one I control the DNS for, adding the missing record isn't something I can do myself, so the fix for now is just to never link to the www version, the plain atari-wiki.com works fine for everyone regardless of connection type.

To be clear, none of this explains the Microsoft situation, which remains its own mystery. But it should have been fixed a long time ago regardless, it's just that running a server these days comes with so many things to fix, update and keep on top of that there simply aren't enough hours in the day to keep trawling the internet hoping for an answer on every single one of them.

It's only because of using AI properly lately that I've been able to go back and actually revisit problems like this rather than leave them filed under "one day". DKIM is now confirmed working, which alone is a fair chunk of time saved, what would probably have been weeks of poking around trying to work it out by hand took a couple of hours of Claude session tokens instead.

[exxos]

Furthermore to the previous post's work today above...

While going through the same batch of log work, a separate thing turned up worth documenting properly rather than just quietly dealing with it: a slow, ongoing trickle of automated traffic hitting old-style forum permalinks. First spotted a few days back, covered here with a screenshot at the time of it being recognised.

2.PNG (162.06 KiB) Viewed 58 times

Rather than just guess at how big a deal it actually was, I had it checked properly against a full week of real log data, day by day, so here's the honest picture rather than a vague "it's been bad":

Code: Select all

Date                          Matches   Reached server   % of that day's load
14 Sep                          5,404          5,403               4.9%
15 Sep                          12,441        12,440              15.5%
16 Sep                          16,207        16,207              18.8%
17 Sep                           1,144           558               3.0%
18 Sep (the peak)                34,323        27,127              29.2%
19 Sep                           4,429          2,653              15.0%
20 Sep                             382           125                1.8%
21 Sep (today, partial to 15:47)   630           198                3.6%

At its worst, on the 18th, this traffic was responsible for nearly 30% of everything the server had to actually process that day. Two days later it had dropped to under 2%, a 90 times drop, and it's sat at a low background trickle since, roughly one request a minute at the time of writing.

It hasn't stopped entirely, and it may well pick back up again at some point, so there's now a proper monitor keeping an eye on it. Nothing's being acted on yet, but if it does come back at anything like the scale of the 18th, there's a quick way to deal with it ready to go.

Out of curiosity I also had a look at what it's actually requesting, rather than just how much of it there is. It isn't browsing the site live: every link it's working through is old, the newest one seen today was posted back in February, and even at the busiest point on the 18th nothing newer than about three months old showed up at all. It's heavily weighted towards the sort of thing that's been linked on every page for years, the forum rules, the upload guide, that sort of post, and it works through the same handful of links over and over from different addresses before moving on. Looks like it's just replaying an old list it picked up at some point rather than discovering anything current.

And on the subject of why those particular posts keep turning up...

Had a look at why certain posts, the forum rules, the "how to register" one, the image upload guide and so on, keep showing up over and over in the traffic. Turns out it has nothing to do with anyone taking a special interest in what those posts actually say. They are all pinned as global announcements, which means the forum prints them at the top of every single section on the board, hundreds of sections deep. So anything working its way through the forum picks up those same eight posts again and again, simply because they are stapled to the top of everywhere, not because of what is written in them. Checked it properly too, comparing how visible each pinned post actually is around the site against how often it gets hit, and the two match up almost exactly whatever the post is about. A rules post sitting in a members-only area that guests cannot even see got zero hits, which would not make sense if the content itself was the draw rather than just being plastered on every page like a stuck record.

[exxos]

Forum updates will be happening very shortly and the forum may go down during updates...

EDIT: Done Nothing Burned to the ground by the looks of it

[exxos]

phpBB updated to 3.3.19, and they have fixed a problem we already hit

The forum has been updated to phpBB 3.3.19 tonight. The update was rehearsed on a private copy of the forum first, and the actual changeover took 26 seconds, so most of you will not have noticed a thing. It is mainly a security release, so it was not one to sit on.

What caught my eye was one of the new features. phpBB say they have improved guest session handling to "help combat potential issues when a large number of guest sessions are created", for example during excessive crawling by bots. That is exactly the problem we ran into here.

What we found
Every visitor who is not logged in gets a guest session, which is a row in the sessions table. During the bot floods the board was showing tens of thousands of "users online", nearly all of them bots. phpBB looks through that table on ordinary page loads, so once it gets that big it stops being a symptom and becomes the cause. The database slows down, pages slow down, requests queue up, and the table stays full. Round and round it goes.

What we did about it

• Most guest pages are now served from a cache before phpBB even starts, so most bots never create a session in the first place.
• Session IDs were stripped out of the links on guest pages, so bots were no longer handed an endless supply of "new" URLs to crawl.
• An extra database index was added on the sessions table to make those lookups much cheaper.
• The blocking scripts deal with the worst of the floods before they reach the forum at all.

What phpBB have now done

• Guest sessions are now cleared after 15 minutes of inactivity, instead of hanging about for an hour, so the table empties about four times faster.
• They have added an index on the sessions table on the same two columns we indexed by hand. Great minds and all that. We will drop our own copy later as it is now a duplicate.
• There is a new "AI crawlers" group, so AI bots can be handled separately from normal search engine bots.

It is nice to see phpBB arrive at the same answer, a bit like finally getting the manual after you have already worked out the game. It will not stop a flood creating sessions, but it means the leftovers get swept up much sooner. I will keep an eye on the session numbers over the next few weeks to see how much difference it makes in practice.

As always, if anything looks odd after the update, shout.

[exxos]

I said a few weeks ago I could see this coming, and here we are. Detecting bad traffic gets harder every month, not easier, and I want to be straight with everyone about where I think this is heading.

I've still got a couple more tricks up my sleeve to keep blocking the worst of it, but my honest prediction is that over the next few months the detection side of things is going to start losing ground. Even with AI helping build the detection logic, I think it's a losing battle in the long run, because I strongly suspect AI is now being used on the other side too, to help evade exactly the detection it's helping me build. Some botnets are still fairly dumb and easy to spot, whole networks of them in fact, but the landscape on the clever end is shifting faster than I've seen it shift before, and it feels like it's basically becoming a game of diminishing returns.

That's the whole reason behind the guest versus member resource split I mentioned a while back. Guests and members each get their own chunk of server resources. If the guest side gets overwhelmed, guests will see the forum struggle or go down. Members will not, because the member pool is separate and protected. The honest version of the endgame is that I'm prepared to let the guest and bot traffic fight it out amongst themselves on their own pool of resources, rather than keep pouring endless effort into a fight I don't think stays winnable, and put my energy into keeping the member side solid instead. I think it's a case of when, not if, the guest pool starts taking real hits over the coming months, so if you're reading this as a guest, please sign up for a proper account. I can't promise guest access stays reliable indefinitely, and I'd rather you heard that from me now than find the forum unavailable later with no warning.

The network-level blocking is still doing its job, picking up new bad networks on ordinary traffic every day, so it's not like nothing is working. But the trend over the past year has been unmistakable: the bots keep getting better at evading detection, and I'm running out of road on that front.

I'll also say this is a battle on two fronts, not one. The bot traffic side is already effectively a full-time job on top of my actual job, watching it, tweaking it, working out the shape of each new wave. But there's a second front opening up: accounts signing up in bulk that get through every defence and every signup check we have, and then it becomes a daily grind working out whether a given post is a genuine person or not. Our moderator team are very on the ball with this, but I can see that side alone turning into its own full-time job within the next year, running in parallel with the traffic side rather than instead of it.

Worth mentioning too: I've been keeping an eye on the EEVblog forum lately, which has put literally every page behind Cloudflare, which is frankly a bit much, and they are still going down. Plenty of people there seem to treat Cloudflare as the Holy Grail. It clearly isn't one, or they wouldn't still have a problem. I did ask, reasonably I thought, how they know Cloudflare isn't also quietly blocking some of their genuine visitors along with the bad traffic. The answer was silence, followed by more praise for Cloudflare "fixing" things. So even setting aside any specific tool, the point stands: our defences here work, and they work extremely well, but I can't guarantee that's still true in a year's time, and blind faith in any one product clearly isn't the answer either.

There's also a newer problem brewing on the signup front specifically. A couple of days ago we had what looked like a genuine member signup, apparently advertising some product, with terminology mixed together in a strange way, and the post itself read as clearly AI-written. Whether that got through signup and CAPTCHA as a real determined person or something more automated, I honestly don't know yet. I suspect this is the shape of what's coming: even AI systems that apparently want their daily Atari fix badly enough will start signing up in bulk specifically to get around the guest-side limitations, which just moves the "is this actually a human" problem up a level, from guests to members.

I've wondered, and it's been suggested to me too, whether in a year's time we might need to move to a paid signup to become a member. I want to be clear that's not a money grab. It's that I don't think there's going to be any reliable way left to tell bot, AI and human traffic apart, and charging even a small amount is one of the few remaining filters. Throwing more money at server capacity isn't something I'm willing to do either, because that just means paying to give the bots exactly what they want while they give absolutely nothing back to the forum. The trouble is a paid signup isn't a clean fix either. We already saw the donation link get hammered by people testing stolen card numbers to see which ones worked, so a signup fee would likely just become the next target for the same kind of abuse.

So consider this your advance warning. If you're lurking, sign up now while it's free and easy. I can't guarantee that stays true forever. Sorry for the doom and gloom, but this is genuinely where things stand, and I don't expect next year to be any easier.