Server updates

[exxos]

Quick update on the server work

Been carrying on with the tidying up over the last couple of days, mostly double checking rather than fixing anything new. This is the final polishing stage now, going back through everything with fresh eyes to make sure nothing genuine is still getting caught by mistake.

Found a couple of small loose ends along the way. One redirect that was looping forever for no good reason, quietly wasting resources on both ends. A couple of old leftover files from a past update that had no reason to still be sitting there publicly. Nothing serious, just tidying up.

I have also improved the monitoring side of things so it is easier to see trends over a longer stretch rather than just the last day or two. Should make it much easier to spot anything unusual going forward, and to keep an eye on things behaving properly over time rather than just in the moment.

On the bigger picture, it has been a fair few days now since anything resembling a real flood. Honestly, I cannot tell you for certain whether that is because it is all being blocked properly now, whether whatever was hitting us has simply moved on to easier targets elsewhere, or a bit of both. Time will tell either way. The site has been sitting quiet and comfortable regardless, which is the actual goal here.

As always, if the site ever seems slow or unreachable for you, let me know.

[exxos]

NOTICE:

I'm going to "retire" my largest fail2ban jail later tonight. I won't go into the reasons why at this point.. But basically I'm moving the rules higher up in the chain to nginx level. This means connections will get dropped way faster and consume way less resources than they do now.

But this means writing new nginx regex which may cause issues. I will test as much as I can before going live. But users won't get banned, you will get a 403 page..

Also some other rules in nginx level will get dropped also. So we may see a surge in guest hits. I will monitor the situation. If hits grow to unsustainable levels then those rules will have to go back. This is part of trying to resolve mobile ips being a problem as they mix good and bad traffic. Hence part of why rules are moving to nginx level for faster actions.

So let me know if something looks or acts wrong later on.

[exxos]

The changeover is now live. Let me know if anyone gets unexpected 403 errors. I will need your IP and time of the event to trace it properly.

Also several forum extensions have been updated just to recent versions.

[exxos]

Bit of a behind the scenes update

Been doing some housekeeping on the server the last couple of days, nothing that changes how the forum looks or works day to day, but worth a quick post since some of it fixes things a few of you may have noticed.

Fixed some old broken images and links
If you've ever opened an older post and seen a missing smiley or a broken image icon instead of the actual smiley, that should be sorted now. Turned out to be a leftover from when the site moved under /forum/ years back, some old links never got updated to match and were just quietly 404ing this whole time. Same fix caught a few old forum and extension links too.

Moved some of the automated blocking into the web server itself
Been meaning to sort this one for a while. The old system permanently blocked an address the moment it did something suspicious, and never let it go afterwards. Sounds sensible on paper, but a lot of you are on mobile data or a home broadband connection where the address you're using today isn't really "yours", it can get handed to someone else in as little as a minute, sometimes even less. So a permanent block on an address could end up hitting a completely innocent person shortly after, through no fault of their own, just because someone else misbehaved from that same address before they got it.

Switched that over so the bad request just gets blocked on the spot, instantly, with nothing held against the address afterwards. Same protection, just without anyone risking getting caught by someone else's bad behaviour down the line.

This has actually been a genuine issue for a long time, which is why I have been telling people over the past months to avoid using mobile data to get on here if they could help it. Hopefully with this sorted that advice shouldn't really be needed any more.

General cleanup on the bot and security side
Went through a fair bit of the rest of the automated blocking too and tightened up a few things, mainly making sure genuine visitors, including mobile users and people outside the UK, don't get caught by mistake the way some of the older rules could occasionally do. Also found and fixed a couple of small performance things that were making some of the background maintenance scripts run slower than they needed to.

The bingbot mystery
Bit of a fun one. While going through the logs I kept seeing hundreds of odd looking 404s, page titles full of garbled text with random gambling and spam links buried in them. First thought was someone was actively trying it on again after the wiki got hacked a good few years back now.

Turned out not to be the case. Checked properly rather than just assuming, and every single one traced back to Bing's own crawler, still dutifully revisiting a big list of spam page URLs it indexed years ago back when the wiki really was compromised. No referrer, no active attempt, just an old search engine to do list that never got cleared off. Confirmed it really was Bing and not something spoofing it, so nothing to worry about there, just a strange bit of internet archaeology.

Anyway, forum's running the same as always, just a bit tidier under the hood. Shout if you spot anything odd.

[prog99]

I've had that with bing too on a site I manage. We moved to a new platform (to WordPress from a custom CMS) and setup proper redirects(301 return code) but bing still insists on checking the old URLs. Google figured it out within hours.

[exxos]

I've had that with bing too on a site I manage. We moved to a new platform (to WordPress from a custom CMS) and setup proper redirects(301 return code) but bing still insists on checking the old URLs. Google figured it out within hours.

Yeah Bing is a pain. When the wiki first went up the traffic for the first year was horrendous. Just Bing keeps checking the spam links over and over again :roll: .

[exxos]

Firewall whitelist changes

For a long time now there has been a script running on the server every 30 minutes which looks at who has been logged in recently, and makes sure their IP range is whitelisted and not sat in one of the blacklists. It works, but it is a bit of a blunt tool.

It scans the last 7 days of sessions every single time it runs, whether anything has changed or not, and most of the time it finds nothing to do.
The other problem is the 30 minute gap. If you logged in, did what you needed and logged out again inside that window, the script could miss you completely.

So I have written a small forum extension to do the job properly. When you load a page while logged in, it works out your IP range and checks if it has seen it before. If it has, that is the end of it, nothing else happens. If it is a new range, it hands it over to the firewall side which whitelists it and pulls it out of the blacklists if it happens to be in there.

The main thing this fixes is speed. It now happens the moment you appear rather than up to half an hour later. It also picks up IP changes while you are still logged in, which the old script had no real way of doing.

The extension itself cannot touch the firewall, and that is deliberate. All it does is write your IP to a file in memory, and a separate process picks it up from there. That way nothing on the forum side has any firewall access at all, which is how I want to keep it.

While doing this I also found the old script was whitelisting some of the AI crawlers by mistake, because it only skipped bots with names in brackets and some of them do not have them. That has gone. No bot gets whitelisted by this now, good or otherwise. Bot names are just a text string that anyone can claim to be, so whitelisting anything based on it was never a great idea. I will sort proper handling for the legitimate crawlers separately.

Nothing changes for anyone using the forum, it should just mean fewer people getting caught out by the firewall in the first place. If you do ever find yourself blocked, the unban request page still works the same as always, just log in from mobile data or another connection and fill it in.

[exxos]

Today's surge is brought to you by META AI training...

meta.PNG