Server updates

[exxos]

Sorting out an email problem that had been quietly broken for a while

Regular readers will know I've had an on/off battle with mail deliverability for a few years now, mostly aimed at Microsoft/Hotmail addresses (the main thread on that is here, and a related one here). That one's still unresolved on their end, and after this long getting nowhere with their support I've more or less accepted it's a "some day" problem rather than a "today" one.

While going through the mail server logs properly (something I don't do often enough, in fairness) I found something separate and rather more concerning: DKIM signing had been switched off on the server entirely.

For anyone not familiar, DKIM is one of three checks (along with SPF and DMARC) that receiving mail servers use to work out whether an email claiming to be from a domain actually came from that domain, or from someone pretending. DKIM specifically adds a digital signature to outgoing mail, signed with a private key that only the sending server holds, which the receiving server checks against a public key published in DNS. If it doesn't sign correctly, or isn't there at all, your mail looks a lot more like something a spammer would send, which does nobody any favours.

I was fairly sure I'd set this up properly years ago using a different DKIM system entirely, but going back through everything there's no evidence any other system was ever installed, it's always been the same one. Best guess is it started causing mail to hold or fail at some point, got switched off to make the problem go away, and then just got forgotten about, the way these things do when there's no obvious symptom pointing back at it. It stayed that way until Claude spotted it a few days ago while going through the logs for something else entirely.

Digging further turned up a second, older problem sitting underneath the first. This server sends mail for several domains, including the two exxos ones (exxosforum and exxoshost) and others, and all of them had been signing under a single DKIM key registered to exxosforum's own domain. That's fine for exxosforum itself, but for the rest it breaks DMARC alignment, since the signature doesn't match the domain the mail claims to be from. So even with DKIM nominally working, most of the mail leaving this box would still have looked suspicious to anywhere strict enough to check properly.

As for why it likely got switched off, the probable culprit was a missing setting (milter_default_action) that controls what the mail server does if the DKIM signing service doesn't answer in time. Left unset, it defaults to holding the mail rather than sending it, which is exactly the kind of thing that ends with someone switching the whole lot off in frustration rather than tracking down why mail was getting stuck.

Separately, the store side of things sends its own mail through PHPMailer rather than the forum's own mail path, and that turned up its own small pile of history while I was in there. There were two duplicate copies of it kicking about, both carrying the same manual patch I'd made a while back to stop it falling over when an email address or name came through empty. Both copies got consolidated down to one, and then that one was updated to the current official release, which has its own, slightly more thorough version of the same fix built in, so the manual patch isn't needed any more. While in there it also turned up that the store's own attempt at DKIM signing had been pointing at the wrong file the whole time, the DNS text record instead of an actual private key, so it had been silently failing on every single email without ever throwing an obvious error. Removed, since real signing now happens properly at the server level for everything regardless.

Fixed now, properly:

• A separate DKIM key generated for each domain, with new records published in DNS for each one
• The missing setting corrected so a slow signing check no longer holds mail hostage
• DKIM re-enabled and verified with real sent mail, headers checked across every domain, all passing
• PHPMailer tidied up to one current copy instead of two old ones, and the broken app-level DKIM attempt removed

One last, smaller thing found while going through everything: the wiki's www address (www.atari-wiki.com) has no IPv4 record at all, only IPv6, so anyone on an IPv4-only connection can't reach it if they land on the www version. Since that domain isn't one I control the DNS for, adding the missing record isn't something I can do myself, so the fix for now is just to never link to the www version, the plain atari-wiki.com works fine for everyone regardless of connection type.

To be clear, none of this explains the Microsoft situation, which remains its own mystery. But it should have been fixed a long time ago regardless, it's just that running a server these days comes with so many things to fix, update and keep on top of that there simply aren't enough hours in the day to keep trawling the internet hoping for an answer on every single one of them.

It's only because of using AI properly lately that I've been able to go back and actually revisit problems like this rather than leave them filed under "one day". DKIM is now confirmed working, which alone is a fair chunk of time saved, what would probably have been weeks of poking around trying to work it out by hand took a couple of hours of Claude session tokens instead.

[exxos]

Furthermore to the previous post's work today above...

While going through the same batch of log work, a separate thing turned up worth documenting properly rather than just quietly dealing with it: a slow, ongoing trickle of automated traffic hitting old-style forum permalinks. First spotted a few days back, covered here with a screenshot at the time of it being recognised.

2.PNG (162.06 KiB) Viewed 15 times

Rather than just guess at how big a deal it actually was, I had it checked properly against a full week of real log data, day by day, so here's the honest picture rather than a vague "it's been bad":

Code: Select all

Date                          Matches   Reached server   % of that day's load
14 Sep                          5,404          5,403               4.9%
15 Sep                          12,441        12,440              15.5%
16 Sep                          16,207        16,207              18.8%
17 Sep                           1,144           558               3.0%
18 Sep (the peak)                34,323        27,127              29.2%
19 Sep                           4,429          2,653              15.0%
20 Sep                             382           125                1.8%
21 Sep (today, partial to 15:47)   630           198                3.6%

At its worst, on the 18th, this traffic was responsible for nearly 30% of everything the server had to actually process that day. Two days later it had dropped to under 2%, a 90 times drop, and it's sat at a low background trickle since, roughly one request a minute at the time of writing.

It hasn't stopped entirely, and it may well pick back up again at some point, so there's now a proper monitor keeping an eye on it. Nothing's being acted on yet, but if it does come back at anything like the scale of the 18th, there's a quick way to deal with it ready to go.

Out of curiosity I also had a look at what it's actually requesting, rather than just how much of it there is. It isn't browsing the site live: every link it's working through is old, the newest one seen today was posted back in February, and even at the busiest point on the 18th nothing newer than about three months old showed up at all. It's heavily weighted towards the sort of thing that's been linked on every page for years, the forum rules, the upload guide, that sort of post, and it works through the same handful of links over and over from different addresses before moving on. Looks like it's just replaying an old list it picked up at some point rather than discovering anything current.

And on the subject of why those particular posts keep turning up...

Had a look at why certain posts, the forum rules, the "how to register" one, the image upload guide and so on, keep showing up over and over in the traffic. Turns out it has nothing to do with anyone taking a special interest in what those posts actually say. They are all pinned as global announcements, which means the forum prints them at the top of every single section on the board, hundreds of sections deep. So anything working its way through the forum picks up those same eight posts again and again, simply because they are stapled to the top of everywhere, not because of what is written in them. Checked it properly too, comparing how visible each pinned post actually is around the site against how often it gets hit, and the two match up almost exactly whatever the post is about. A rules post sitting in a members-only area that guests cannot even see got zero hits, which would not make sense if the content itself was the draw rather than just being plastered on every page like a stuck record.