Been doing a fair bit of tidying under the bonnet over the last few days, so here is a summary of what changed.
Database accounts have been locked down properly. Every application on the server had far more access than it needed, in some cases enough to read or wipe databases belonging to completely unrelated things. Each one is now scoped to its own database and nothing else. This is the sort of change nobody notices when it works, which is rather the point.
The firewall gained some proper safety plumbing. There was a small window where two things writing to the block list at once could have left the saved copy in a mess. That is now serialised, and the save itself is atomic, so a save interrupted halfway leaves the previous good copy in place rather than a half-written file. Also fixed a permissions problem that had been quietly stopping part of the nightly backup from doing its job, and cleared out a duplicate copy that was being sent to the same place twice over.
The bigger job was writing down how the whole thing actually works. Two years of changes had left scripts on disk that stopped being used months ago, comments describing behaviour that no longer happens, and at least one job cheerfully reporting success every day while doing absolutely nothing. There is now a proper map of what runs, what it touches and what is genuinely dead, which should stop me fixing things that were switched off in August.
That exercise turned up a handful of real bugs too, including one that would have started failing that same night.
The other bit of work was making everything board-aware rather than assuming there is only one forum on the server. Various parts had my forum's name baked into them, and those now work it out from the forum's own settings instead. So as a side effect, this server could now host a second forum alongside this one without the two treading on each other. Whether that ever happens is another matter entirely, but the groundwork is done.
Cleantalk released the fixes I requested a couple days ago. Great news. So it's now updated and not conflicting with my firewall and extension logic. A CAPTCHA bug was also found and fixed which may have caused login issues under rare conditions. Some ACP fixes done relating to cosmetic labeling errors. Lots of other minor tweaks which took some days. Plus ongoing server updates which seem to come up 50 a week.
Nothing here changes anything you will see day to day, which is generally the mark of a good week's work.
Can all registered uses please login, even just for a few minutes..
It helps build a picture where our "good traffic" is coming from..
Thanks :)
It helps build a picture where our "good traffic" is coming from..
Thanks :)
Server updates
Re: Server updates
Upcoming change: post links
A quick heads up on a small change coming to the forum. Nothing you need to do, but you might notice links looking slightly different.
Every post on the forum has its own link, the one you get from the little icon next to the post time. Handy for pointing people at a specific post, and I use it all the time myself. The trouble is how those links work behind the scenes. A link to post number 12345 does not say which page it lives on, so every time someone clicks one the server has to stop, look it up in the database, work out the page, and then send them there. The page itself is already sitting in the cache ready to go, but the link has to be resolved first, and that bit cannot be cached.
Now think about what a page actually contains. Twenty posts per page means twenty of those links on every single page, plus another one on the "last post" column of every forum listing. A human might click one of them now and again. A search engine or an AI crawler clicks every single one. So a topic page that was already served from the cache once gets requested another twenty times through the post links, and each of those twenty is a fresh trip through PHP and the database to land on exactly the same page. It is a bit like posting twenty copies of the same letter to yourself to check the postman is still working.
This came to light while going through the logs and watching GPTBot (OpenAI's crawler) at work. Over a couple of hours it accounted for about a quarter of everything the cache could not serve, and nearly all of it was post links. It was not doing anything wrong, just politely following every link it found, which is exactly what crawlers do. The server coped fine, but it was a lot of effort for nothing.
Some of you may remember an earlier version of this problem. Back in July every one of those post links was also being stored in the cache as a separate copy of the page, so one topic page could end up sitting in memory twenty times over and pushing out pages people actually wanted. That was fixed at the time, and post links now redirect to the proper page instead. That cured the wasted memory, but each link still needs the database lookup before it can redirect, and that is the bit this change deals with.
The fix is simple. The forum already knows which page it is on when it draws those links, so from now on it will write the page number straight into the link. Clicking a post link then just loads the page from the cache and jumps to the post, no database lookup needed. Any old style post links you have already copied or bookmarked will carry on working exactly as before, and the "copy link" on posts still points at the post, it just goes there by a more direct route.
The only edge case is if a post earlier in a topic gets deleted after you copied a link, which can shift later posts onto a different page. If that ever happens the page will notice the post is not where it expected and quietly hop to the right place, so you should not see any difference.
Short version: fewer wasted requests for the server, no difference for you, and a slightly quieter life for the poor old database.
A quick heads up on a small change coming to the forum. Nothing you need to do, but you might notice links looking slightly different.
Every post on the forum has its own link, the one you get from the little icon next to the post time. Handy for pointing people at a specific post, and I use it all the time myself. The trouble is how those links work behind the scenes. A link to post number 12345 does not say which page it lives on, so every time someone clicks one the server has to stop, look it up in the database, work out the page, and then send them there. The page itself is already sitting in the cache ready to go, but the link has to be resolved first, and that bit cannot be cached.
Now think about what a page actually contains. Twenty posts per page means twenty of those links on every single page, plus another one on the "last post" column of every forum listing. A human might click one of them now and again. A search engine or an AI crawler clicks every single one. So a topic page that was already served from the cache once gets requested another twenty times through the post links, and each of those twenty is a fresh trip through PHP and the database to land on exactly the same page. It is a bit like posting twenty copies of the same letter to yourself to check the postman is still working.
This came to light while going through the logs and watching GPTBot (OpenAI's crawler) at work. Over a couple of hours it accounted for about a quarter of everything the cache could not serve, and nearly all of it was post links. It was not doing anything wrong, just politely following every link it found, which is exactly what crawlers do. The server coped fine, but it was a lot of effort for nothing.
Some of you may remember an earlier version of this problem. Back in July every one of those post links was also being stored in the cache as a separate copy of the page, so one topic page could end up sitting in memory twenty times over and pushing out pages people actually wanted. That was fixed at the time, and post links now redirect to the proper page instead. That cured the wasted memory, but each link still needs the database lookup before it can redirect, and that is the bit this change deals with.
The fix is simple. The forum already knows which page it is on when it draws those links, so from now on it will write the page number straight into the link. Clicking a post link then just loads the page from the cache and jumps to the post, no database lookup needed. Any old style post links you have already copied or bookmarked will carry on working exactly as before, and the "copy link" on posts still points at the post, it just goes there by a more direct route.
The only edge case is if a post earlier in a topic gets deleted after you copied a link, which can shift later posts onto a different page. If that ever happens the page will notice the post is not where it expected and quietly hop to the right place, so you should not see any difference.
Short version: fewer wasted requests for the server, no difference for you, and a slightly quieter life for the poor old database.
Re: Server updates
Post links, part two
Following on from the last update about post links, there is a second half to the story which turned out to be more interesting than expected.
A quick recap for anyone who missed it. Every post on the forum has its own link, and the old style looked like this:
viewtopic.php?p=145386
That tells the server which post you want but not which page it lives on, so the server has to stop and look it up in the database every single time before it can send you anywhere. The new style looks like this:
viewtopic.php?t=8519#p145386
That says which topic and which page, with the post number on the end so your browser scrolls straight to it. No lookup needed, and the page itself comes from the cache. Same destination, considerably less work.
The extension went live and the post links changed over as expected. Then, poking around at the logs to see how much difference it had made, I noticed the old style links were still turning up rather more than they should have been. Following that back led to the quote button.
When you quote someone, the little arrow that jumps you back to the original post is also a link, and that one was still using the old format. It turns out phpBB builds that link in a completely different place to the rest, in a part of the software that has no way of knowing which page the quoted post is on. It just writes the old style link every time and hopes for the best.
Worse, it does not store that link anywhere. It rebuilds it fresh every time the page is drawn. So there was no way to go through and tidy up old posts, because there was nothing in them to tidy up. There are roughly 49,000 posts on here containing a quote, and every single one was generating an old style link every time anyone looked at it, forever.
The fix was to replace that part of phpBB with a version that does the lookup once for the whole page rather than once per quote, so the cost is a single extra database query when a page is built, and then nothing at all for everyone who reads it afterwards.
There are a few cases where it deliberately leaves the old style link alone rather than risk getting it wrong, mostly around moderators and deleted posts, since those people see a different set of posts to everyone else and the page numbers would not line up. Those links carry on working exactly as before.
TL;DR, and why any of this matters
A cached page costs the server almost nothing to hand out. Thousands of people can read the same page and it is served straight from memory. That is what keeps the forum quick and what keeps it running on hardware I can actually afford.
An old style post link cannot be cached. It has to be worked out from scratch every single time, and that means the database gets involved on every click. One of those is nothing. The problem is the sheer number of them: twenty per page from the post links, plus another one for every quote, and search engines and AI crawlers follow every last one of them across the whole forum, every time they come round. That was hundreds of thousands of requests that each had to be worked out individually, all to arrive at pages that were already sitting there cached and ready to go.
The change means those links now land on the cached page directly instead. Same destination, no lookup, no database. For you nothing changes at all, other than the forum staying quick under load instead of grinding when something decides to crawl the lot at three in the morning.
The savings are hard to overstate. Old style post links accounted for around 96% of every request the cache could not serve, and each one of those had to be built from scratch. The new format lands on a page that is already cached, so the same click costs a fraction of what it used to. It will take a while to show fully, since search engines are still working through lists of old links they collected months ago, but the direction is not in any doubt.
Nothing you need to do, and the old links still work. Just a bit less digging about for the server every time someone quotes someone else.
Following on from the last update about post links, there is a second half to the story which turned out to be more interesting than expected.
A quick recap for anyone who missed it. Every post on the forum has its own link, and the old style looked like this:
viewtopic.php?p=145386
That tells the server which post you want but not which page it lives on, so the server has to stop and look it up in the database every single time before it can send you anywhere. The new style looks like this:
viewtopic.php?t=8519#p145386
That says which topic and which page, with the post number on the end so your browser scrolls straight to it. No lookup needed, and the page itself comes from the cache. Same destination, considerably less work.
The extension went live and the post links changed over as expected. Then, poking around at the logs to see how much difference it had made, I noticed the old style links were still turning up rather more than they should have been. Following that back led to the quote button.
When you quote someone, the little arrow that jumps you back to the original post is also a link, and that one was still using the old format. It turns out phpBB builds that link in a completely different place to the rest, in a part of the software that has no way of knowing which page the quoted post is on. It just writes the old style link every time and hopes for the best.
Worse, it does not store that link anywhere. It rebuilds it fresh every time the page is drawn. So there was no way to go through and tidy up old posts, because there was nothing in them to tidy up. There are roughly 49,000 posts on here containing a quote, and every single one was generating an old style link every time anyone looked at it, forever.
The fix was to replace that part of phpBB with a version that does the lookup once for the whole page rather than once per quote, so the cost is a single extra database query when a page is built, and then nothing at all for everyone who reads it afterwards.
There are a few cases where it deliberately leaves the old style link alone rather than risk getting it wrong, mostly around moderators and deleted posts, since those people see a different set of posts to everyone else and the page numbers would not line up. Those links carry on working exactly as before.
TL;DR, and why any of this matters
A cached page costs the server almost nothing to hand out. Thousands of people can read the same page and it is served straight from memory. That is what keeps the forum quick and what keeps it running on hardware I can actually afford.
An old style post link cannot be cached. It has to be worked out from scratch every single time, and that means the database gets involved on every click. One of those is nothing. The problem is the sheer number of them: twenty per page from the post links, plus another one for every quote, and search engines and AI crawlers follow every last one of them across the whole forum, every time they come round. That was hundreds of thousands of requests that each had to be worked out individually, all to arrive at pages that were already sitting there cached and ready to go.
The change means those links now land on the cached page directly instead. Same destination, no lookup, no database. For you nothing changes at all, other than the forum staying quick under load instead of grinding when something decides to crawl the lot at three in the morning.
The savings are hard to overstate. Old style post links accounted for around 96% of every request the cache could not serve, and each one of those had to be built from scratch. The new format lands on a page that is already cached, so the same click costs a fraction of what it used to. It will take a while to show fully, since search engines are still working through lists of old links they collected months ago, but the direction is not in any doubt.
Nothing you need to do, and the old links still work. Just a bit less digging about for the server every time someone quotes someone else.
Re: Server updates
The one about the new bot
While digging through the logs after the post link changes, I noticed the access log had gone from around 10MB a day to over 300MB. That is a thirty-fold jump, so it was worth finding out what was generating it.
The answer turned out to be a single crawler. Up until the 10th it called itself SleepBot, then from the 11th it switched to calling itself KeenableBot and carried on doing exactly the same thing. Yesterday it was 51% of every request hitting the server. Today it is 61%. More than half the forum's entire traffic is one bot that nobody asked for and that gives nothing back.
What it does is walk every forum on the board, over and over, all day. It requests each forum twice, once in a form that gets redirected and once properly, so it doubles its own workload for no reason. That comes to around 520,000 requests a day, at a steady 500 a minute.
The robots.txt file on here asks every crawler to wait one second between requests. That is not a big ask. This one has read robots.txt exactly once, yesterday evening, and has not looked at it since. Meanwhile individual addresses belonging to it have been recorded doing ten requests in a single second, which is ten times the rate it was asked to observe. So adding a rule to robots.txt for it specifically would most likely be ignored, the same as the existing one is.
Which brings us to why it is not simply banned. It runs from about 23 different addresses a day, all of them Google Cloud, and it rotates them. Blocking today's list does nothing about tomorrow's. Worse, Google Cloud addresses sit inside a range this server deliberately allows through, because that same range is where Googlebot lives, and I have been down the road of accidentally blocking Googlebot before. That was not a fun week. So the cloud-range approach is off the table.
Blocking it by name is possible and is what I will be doing, but it is a temporary measure at best. This thing has already changed its name once in three days. Crawlers that get blocked by name generally come back under a different one, and the whole thing becomes a game of whack-a-mole that I have been playing for about two years now.
The more robust approach is to block on behaviour rather than identity, since a crawler can rename itself but cannot easily stop behaving like a crawler. That takes more care to get right, because the one thing worse than a bot eating your bandwidth is a rule that accidentally locks out real people. I spent an hour this evening chasing exactly that and had to abandon one promising-looking rule because the traffic it would have caught turned out to include genuine members.
So, for now: named block goes in, logs get watched, and we see what it calls itself next week.
The forum itself has been fine throughout, in case anyone is wondering. The server has absorbed the lot without breaking a sweat, which is largely the point of all the caching work that has gone in over the past couple of months. It is more that there is something faintly ridiculous about a retro computing forum spending more than half its resources feeding a machine that will never buy a floppy drive.
Worth remembering what is actually being taken here. Everything on this forum was written by people, for free, to help other people fix their machines. It gets scraped in bulk to train something commercial, and the only part of that arrangement that costs anyone money is my hosting bill.
EDIT
To be clear about where I stand, because it would be easy to read all this as me being anti-AI, and I am not. I use it daily, including for a fair chunk of the work that has gone into this forum's own software. I have no objection to any of it existing. What I object to is the shape of the arrangement.
People here write up their repairs and their findings for free, to help the next person with the same broken machine. That gets hoovered up in bulk to build something commercial, somebody takes a funding round on the back of it, and the only participant whose costs go up is me. I would think rather better of the whole business if any of them ever asked first, or offered anything back, or even just crawled politely instead of at ten times the rate they were asked to.
While digging through the logs after the post link changes, I noticed the access log had gone from around 10MB a day to over 300MB. That is a thirty-fold jump, so it was worth finding out what was generating it.
The answer turned out to be a single crawler. Up until the 10th it called itself SleepBot, then from the 11th it switched to calling itself KeenableBot and carried on doing exactly the same thing. Yesterday it was 51% of every request hitting the server. Today it is 61%. More than half the forum's entire traffic is one bot that nobody asked for and that gives nothing back.
What it does is walk every forum on the board, over and over, all day. It requests each forum twice, once in a form that gets redirected and once properly, so it doubles its own workload for no reason. That comes to around 520,000 requests a day, at a steady 500 a minute.
The robots.txt file on here asks every crawler to wait one second between requests. That is not a big ask. This one has read robots.txt exactly once, yesterday evening, and has not looked at it since. Meanwhile individual addresses belonging to it have been recorded doing ten requests in a single second, which is ten times the rate it was asked to observe. So adding a rule to robots.txt for it specifically would most likely be ignored, the same as the existing one is.
Which brings us to why it is not simply banned. It runs from about 23 different addresses a day, all of them Google Cloud, and it rotates them. Blocking today's list does nothing about tomorrow's. Worse, Google Cloud addresses sit inside a range this server deliberately allows through, because that same range is where Googlebot lives, and I have been down the road of accidentally blocking Googlebot before. That was not a fun week. So the cloud-range approach is off the table.
Blocking it by name is possible and is what I will be doing, but it is a temporary measure at best. This thing has already changed its name once in three days. Crawlers that get blocked by name generally come back under a different one, and the whole thing becomes a game of whack-a-mole that I have been playing for about two years now.
The more robust approach is to block on behaviour rather than identity, since a crawler can rename itself but cannot easily stop behaving like a crawler. That takes more care to get right, because the one thing worse than a bot eating your bandwidth is a rule that accidentally locks out real people. I spent an hour this evening chasing exactly that and had to abandon one promising-looking rule because the traffic it would have caught turned out to include genuine members.
So, for now: named block goes in, logs get watched, and we see what it calls itself next week.
The forum itself has been fine throughout, in case anyone is wondering. The server has absorbed the lot without breaking a sweat, which is largely the point of all the caching work that has gone in over the past couple of months. It is more that there is something faintly ridiculous about a retro computing forum spending more than half its resources feeding a machine that will never buy a floppy drive.
Worth remembering what is actually being taken here. Everything on this forum was written by people, for free, to help other people fix their machines. It gets scraped in bulk to train something commercial, and the only part of that arrangement that costs anyone money is my hosting bill.
EDIT
To be clear about where I stand, because it would be easy to read all this as me being anti-AI, and I am not. I use it daily, including for a fair chunk of the work that has gone into this forum's own software. I have no objection to any of it existing. What I object to is the shape of the arrangement.
People here write up their repairs and their findings for free, to help the next person with the same broken machine. That gets hoovered up in bulk to build something commercial, somebody takes a funding round on the back of it, and the only participant whose costs go up is me. I would think rather better of the whole business if any of them ever asked first, or offered anything back, or even just crawled politely instead of at ten times the rate they were asked to.