The previous article on this site, Server Security in 2026: What We Learned The Hard Way, covered the network layer. It documented how blocklists, fail2ban and per-IP thinking stopped being sufficient against distributed botnets, and it ended with the conclusion that anything reaching the application layer is a battle already lost at sufficient scale.
That conclusion was correct, and it was also incomplete. Because there is a large category of traffic that is not an attack, cannot reasonably be blocked, and still arrives in volumes that will flatten a small server. Search engines, AI crawlers, scrapers, archivers, SEO tools, and an enormous quantity of automated traffic that will not identify itself at all. You cannot block your way out of that. This article is about what happened next.
Some real examples from this server's logs over recent months, so nobody thinks this is theoretical.
This one deserves its own section, because it affects every phpBB board on the internet and most administrators do not know it is happening to them.
phpBB gives every visitor a session. Browsers hold their session in a cookie. Anything that arrives without one gets the fallback instead, which is to put the session ID into every link on the page.
It is worth killing a common assumption here, because I relied on it myself for a long time. Not accepting cookies used to be a decent sign you were dealing with a bot. That is no longer true. Plenty of the automated traffic hitting this server accepts cookies perfectly well and holds them across requests. The same goes for the other old favourite: bots used to give themselves away with ancient browser versions, and now they report current ones, and on more than one occasion here they have reported version numbers that do not exist yet. None of the easy signals are reliable any more.
The mechanism itself works like this. The crawler then sees a page full of links it has never seen before, because the session ID in each one is new, and queues them all for crawling. When those queued links are crawled later, often by a different machine hours or days afterwards, that machine is also a first-time visitor, also gets a new session, and also receives a page full of apparently new links.
The result is an infinitely large crawl space generated from a finite forum. A board with fifteen thousand topics can generate an unlimited number of unique URLs, each one costing a database write, a session row and a full page render. This is why administrators see millions of hits and millions of "unique visitors" on boards with a few thousand posts. It is not that the bots are malicious. It is that the software and the crawlers are interacting in a way that produces an unbounded loop.
Two distinct things are going on and they are worth separating. Some of the session IDs arriving are simply old, harvested from links that search engines indexed years ago with a session ID baked into them and which they dutifully recrawl forever. Those are harmless in intent. The others are outright fabrications, invented by the thousand and sprayed across a botnet. The difference shows in the pattern rather than the value: a genuine phpBB session belongs to one address, whereas the same fabricated identifier turns up from hundreds of addresses at once, in this case as many as 186 for a single made-up ID, mostly on residential ranges in the same few networks.
The same discussion is running on phpBB's own community forum, where administrators are reporting 700GB of traffic in a single day, and one read-only archive board recorded five million unique visitors in a month. Nobody there has a clean answer either, and the most common recommendation is to put a large third party in front of the whole site, which the previous article on this site explains why I am not prepared to do.
The crawl space is only half the damage. The other half is what a random session ID does to caching, and this is the part that turns an annoyance into a genuine emergency.
A cache works by storing a finished page and recognising the same request next time. Recognition is done on the URL. Put a random session ID into that URL and every single request becomes a URL the cache has never seen before, so every single request is a miss. The cache never gets a chance to help. Worse, each miss stores another copy, so the cache fills with thousands of identical pages that differ only by a session ID nobody will ever ask for again, and it evicts the pages real visitors actually want in order to make room for them. At the volumes described earlier in this article, a caching layer in that state is not merely useless. It is actively making things worse: doing extra work, consuming all its memory, and still passing every request through to the forum software and the database anyway.
The obvious fix is to ignore the session ID when working out which stored page to serve, so that every variation of the URL collapses back onto the one stored copy. That is what happens on this server now, and it is the single change that made everything else possible. Every fabricated session ID in the world resolves to the same stored page, served straight from memory, and the forum software is never involved at all.
What is not obvious is how dangerous that change is if you make it naively. The moment you decide that two URLs are the same page, you are also deciding that whatever was stored for the first visitor is safe to hand to the second. If that stored copy carries anything belonging to the first visitor, you have just given it to everybody. This is not hypothetical. It happened here, early on, and the symptom is guests arriving at the site already logged in as somebody else. That is about as serious as a forum bug gets, and it is why so many administrators who try this give up and conclude that phpBB simply cannot be cached.
So stripping the session ID at the web server is not a fix on its own. It is a fix that only works alongside a guarantee: that any page about to be stored has been positively confirmed to contain nothing personal, and that the web server can distinguish an actual logged-in member from anything merely claiming to be one. phpBB provides neither. That is what forced the writing of a custom extension here, and it is why the extension had to come first and the caching second, rather than the other way round.
My view, having spent months on the consequences, is that the whole session ID in the URL system should be scrapped outright. Not improved, not made smarter, removed. Look at what it costs against what it buys. It buys the ability to serve a working session to the very small number of visitors who refuse to accept cookies. It costs an unbounded crawl space, a database filling with sessions for visitors that do not exist, and a permanent invitation to every crawler and scraper on the internet to generate infinite unique URLs from a finite site. That is an enormous amount of trouble taken on for the benefit of a handful of people, and in 2026 anybody genuinely browsing with cookies disabled is going to have a poor time on most of the web regardless. Require a cookie, tell the visitor plainly if they have not got one, and the entire problem disappears.
There is a second half to this that matters just as much. phpBB's cookies are ordinary cookies, and ordinary cookies can be forged by anybody who can be bothered. During testing here I set a "logged-in member" cookie by hand, repeatedly, and the server believed me every time. That is fine as long as the forum itself is doing the checking, because the forum verifies the session behind the cookie. It is not fine the moment you want the web server in front to make decisions based on it, which is exactly what caching requires. A front-end cache that trusts a forgeable cookie is a front-end cache that can be told what to do by any bot that reads a tutorial. That is precisely why a custom cookie extension had to be written here, and it is covered further below.
On this server the session ID problem is solved. Fabricated session IDs no longer generate unique work, and the database is no longer accumulating rows for visitors that do not exist. How that was done is not published here.
I want to be careful here, because phpBB has served this community for a very long time and none of what follows is a complaint about the people who maintain it. But the honest conclusion from the last few months is that phpBB's core design does not survive contact with 2026 traffic, and no amount of administrator effort at the edges changes that.
The design assumption running through the whole application is that a page request comes from a person. Every page is assembled fresh, from the database, on every single request. That was a completely reasonable assumption when it was made. It is now false for the overwhelming majority of requests any public board receives.
Several specific things follow from it.
There is no page cache in core. phpBB does have a caching system, but it caches database results, compiled templates and permission sets. All of that reduces the cost of building a page. None of it avoids building the page. So the best case in core is a cheaper render, when what is actually needed is no render at all. This matters enormously, because a guest viewing a topic sees very nearly the same page as every other guest viewing that topic. That is the ideal candidate for full page caching, and core does nothing with it.
The session ID fallback is a structural flaw, not a bug. Putting the session into the URL when there is no cookie made sense in an era when cookie-less browsers were a real consideration. In 2026 its only remaining effect is to hand every crawler on the internet an infinite supply of unique URLs pointing at the same content. As above, it should not be the default, and in my view it should not exist at all.
Guests should never reach the backend. This is the principle everything else follows from, and it is worth stating on its own. A guest is asking for a public page that looks the same for every other guest asking for it. There is no reason on earth for that request to invoke the forum software, open a database connection, check permissions, assemble a template and build a session. It should be answered from a cache, in a fraction of a millisecond, and the forum itself should never know the request happened. Get that right and the flood becomes irrelevant, because the floods are almost entirely guest traffic. Every problem described in this article is downstream of the fact that phpBB, out of the box, does the opposite.
Bot handling is a hand-maintained list. Core bot detection is a list of user agent strings in the admin panel. It has to be manually updated forever, it is trivially spoofed by anything that wants to spoof it, and any bot not on the list gets the full session treatment described above. There is a widely shared trick of adding catch-all entries matching the strings "bot" and "spider", with people reporting large load reductions from doing so, and the fact that this trick is necessary rather than being the default behaviour says most of what needs saying.
The link surface multiplies the crawl space. Every topic page carries links for sort orders, view modes, print views, unread markers and post permalinks. Each is a legitimate feature. Collectively they mean one page of content presents a crawler with many distinct URLs, and a crawler has no way to know that most of them lead back to the same thing. Canonical handling exists but is not sufficient on its own, and on this board it turned out to be emitting two conflicting canonical tags on every topic page, one of which contained a session ID.
Fixing it from outside is genuinely hard. Everything above can, with effort, be worked around by an administrator writing custom extensions. I know, because I have done it. But the hook points available to an extension are not always in the right place, anything touching session handling is core rather than extension territory, and the amount of verification required to be certain you have not accidentally made one visitor's session visible to everyone else is considerable. This is not work a typical volunteer board administrator can reasonably be expected to take on, and it certainly is not work that should need doing thousands of times over by thousands of separate administrators.
Cookieless browsing has to go, and that is a decision rather than a technicality. I do not want to lose members, and I am aware some people feel strongly about cookies. But the arithmetic is not close. Accommodating one visitor who refuses cookies is what generates the unbounded crawl space, and that in turn is what brings millions of requests from hundreds of thousands of addresses. One person's preference against the survival of the board is not a trade any small forum can make in 2026. Enable cookies or browse somewhere else, and I say that with regret rather than annoyance.
What a 2026-appropriate design would look like, in outline:
That last one deserves a word, because it is the one that would help the most people. Everything else on that list is invisible plumbing. Geographic blocking is something a board owner can reason about directly: they know where their members are, and they know that the retro hardware, or the local hobby, or the regional club their forum exists for has no audience on the other side of the world. Here it is done outside phpBB, with custom scripts and web server rules that need constant maintenance, and I can do that because I run the server. Most phpBB administrators are on shared hosting and cannot touch any of it. For them the forum software is the only layer they have access to, so a checkbox in the admin panel is not a lesser version of a firewall rule. It is the difference between having a defence and having none.
Put the pieces together and the picture is fairly stark. Twenty permalinks per page multiplying every topic twenty-fold, an unbounded session-generated crawl space on top of that, and a session table large enough to slow the database that feeds all of it. None of those are attacks. They are the software working exactly as designed, against a kind of traffic that did not exist when it was designed. That is why I have ended up writing extensions to change how phpBB behaves rather than just tuning a web server in front of it.
That has to come from upstream. Individual administrators solving it one board at a time does not scale, and the boards that cannot solve it are the ones quietly going offline.
To be clear about what this section does and does not say: blocking is a real and useful part of the defence here, and a lot of traffic never gets past it. What it cannot be is the whole answer.
The reason is rotation. A large share of this traffic comes from pools of addresses that constantly cycle, and the same address that sends a flood today may carry an ordinary visitor tomorrow, from anywhere in the world. A threshold needs a repeat offender to catch, and there isn't one. Blocking by user agent fails for a similar reason, because they are trivially forged and, as covered earlier, no longer distinguish automation from a browser. And blocking by address range runs into mobile carriers and residential broadband, where a single address is shared by hundreds of real people and reassigned to somebody else within hours. Ban it today and you have banned a stranger tomorrow, permanently, with no way for them to find out why.
What does work is blocking by region, and this board does it. There are large parts of the world that have never produced a single member login here, and are never going to, because the retro hardware this site exists for was never sold there and there is no market for it there now. Most of the bad traffic arrives from exactly those places. Blocking them outright removes something like a fifth of it at the door before anything else has to think about it, and costs nothing that this community was ever going to use. The audience this forum actually has, the UK, Europe, North America, Australia and New Zealand, is never blocked by range, precisely because that is where the real people are.
That is a trade made with open eyes. It will occasionally catch a genuine visitor travelling or on a VPN, and I would rather that than hand the board back to the floods. But it is a blunt instrument that thins the traffic. It does not solve anything, and the remaining traffic is the difficult kind.
There is also a more basic point that took a while to sink in. Search engines and AI crawlers are not attackers. Being visible in search is how a niche community gets found by the handful of people each year who are looking for it. Blocking crawlers to protect the server is a form of winning by switching the lights off.
So the target changed. The question stopped being "how do we stop this traffic" and became "how do we make this traffic cost us nothing".
The short version is that a guest page view on this forum now costs a read from memory and never touches the forum software or the database at all. Getting there took months, and it needed four things that did not previously exist.
A way to know who is really logged in. This is the foundation, and it was the hardest part. Caching a page is easy. Caching a page without ever accidentally serving one member's session to somebody else is not. As covered above, a cookie claiming to be a logged-in member proves nothing, because phpBB's own cookies are ordinary cookies and anyone can set one. A custom phpBB extension was written that issues a cryptographically signed cookie at login, which the web server can verify but nobody can fabricate. Everything else in the system hangs off that single signal: who gets a cached page, who gets a fresh one, and whose page is allowed to be stored in the first place.
That extension does several other jobs along the same lines, all of them concerned with making absolutely certain that a page which is about to be stored and shown to thousands of strangers contains nothing personal. There is one thing worth stating plainly for anyone thinking of building this themselves: get this wrong and you will serve one visitor's login session to every subsequent guest. That happened here once, early on, and it is the reason the design is now built to refuse to cache anything it cannot positively confirm is safe, rather than caching everything and trying to spot the exceptions.
A way to stop bots minting infinite unique pages. Covered above. Closely related, a second small extension was written to deal with post permalinks, which were quietly consuming a large share of the server's memory for pages that were almost never read twice.
A way to keep the stored pages fresh. A cache only helps if the page somebody asks for is actually in it. On a board with fifteen thousand pages and a limited amount of memory, keeping the useful ones warm without hammering the server to do it turned out to be a genuinely awkward engineering problem, and the first two designs did not work at all. They appeared to work. Every measurement said they were working. They were not doing anything whatsoever, which cost a couple of evenings to discover.
A way to guarantee members are never affected. Since floods cannot be stopped, the final piece was to make sure they cannot consume the capacity real members need. Logged-in members now have server capacity reserved for them that guest traffic cannot touch, keyed to that same signed cookie so it cannot be claimed by a bot pretending to be a member. Under a severe flood, an already logged-in member should not notice anything at all.
Fixing phpBB's behaviour is only half the battle, and arguably the easier half, because at least it stays fixed. The other half is everything happening in the web server in front of it, and that part is a permanent job.
The custom filters running here drop bad traffic before it reaches the application at all: exploit probes, credential and secret file scanners, forged user agents, scrapers, request patterns that no real browser produces. They work. They also rot, continuously, because every one of them was written in response to what was observed last month.
The landscape genuinely does not sit still. Within the past year on this one server: bots moved off HTTP/1.1 and onto HTTP/2, removing what had been a reliable signal for telling automation from browsers. Scrapers moved from obvious tool user agents to perfectly plausible fabricated browser strings. Attack traffic moved off cheap datacentre ranges and onto residential and mobile networks where blocking has real human cost. And crawlers began appearing in categories that existing rules did not cover, so administrators who had carefully blocked "AI crawlers" discovered they had left "AI search" wide open and were being crawled through the gap.
Every filter also carries a false positive risk, and finding those is not optional. One pattern here, aimed at scanners looking for developer tool configuration files, was quietly matching a completely legitimate wiki resource because of a substring collision in its query string. Another rule intended to catch very old browsers was catching a well-behaved commercial crawler because its user agent happened to contain an old operating system string. Neither showed up in testing. Both were found by reading real logs afterwards, which is the only method that reliably works.
What none of that conveys is the rhythm of it, which is the part that grinds. Every change buys a few days, sometimes a couple of weeks, of calm. Then it starts again with something fresh. Block a signature and a different one appears within days. This has happened, conservatively, a hundred times over the past two years. The adaptation is far too fast and far too consistent to be somebody sitting there watching my server and adjusting by hand.
I cannot prove what is doing the adapting. It would be easy and lazy to say AI, and I have no evidence beyond the pattern. But the pattern is very consistent: a defence goes up, it works, and then it stops working in a way that looks like something worked out what the defence was keying on and stepped around exactly that. Whatever the mechanism, the practical consequence is the same. There is no such thing as a fix here, only a lead that lasts a fortnight.
The honest cost of that, since this article is trying to be honest about costs: this has been running for over two years and it became a full-time job. An unpaid one, on top of an actual job, to keep the lights on for a hobby server that makes no money. That is not a complaint so much as the answer to why small forums are closing. Most people would have shut it down eighteen months ago, and I would not blame them.
So the honest position is that this is not a system that gets finished. It is a system that gets maintained, and the maintenance is the actual product. Anybody selling a fixed set of rules as a permanent solution to this is selling last year's answer.
A few lessons that generalise, without the specifics.
Underneath all of this sits one number that explains why the problem keeps getting worse.
A scraper makes a request. That costs it effectively nothing: a fraction of a second on rented infrastructure, or on somebody else's compromised router, at a price so close to zero it is not worth measuring. The same request, on an unprotected forum, costs the operator a database round trip, a full page assembly, memory, CPU and bandwidth. The cost ratio between the two sides is enormous, and it runs entirely in the wrong direction.
That asymmetry is the whole game. It is why intent does not matter and why the polite crawlers cause as much damage as the rude ones. It is why robots.txt has become largely advisory: it is a request to please cost me less, addressed to a party with no reason to agree. And it is why the only defences that hold are the ones that change the ratio rather than the ones that argue about who deserves access.
The wider version of this is harder to be cheerful about. Content that took years to accumulate, written by unpaid volunteers, is being harvested at industrial scale by well-funded organisations, and the cost of that harvesting lands on the volunteers. Nobody is going to fix that arrangement on behalf of small forums, so the practical response is the one taken here: make it cheap to serve, keep the lights on, and get on with it.
The forum currently serves guest page views from memory in a few thousandths of a second, without invoking the forum software or the database. Traffic that would have flattened the server a year ago now arrives, gets served, and shows up as a line in a log file. The floods have not stopped. They have simply stopped mattering.
That is the honest summary. Nothing here made the traffic go away. It made the traffic cheap.
The reason I am documenting this rather than just quietly running it is that small communities are closing over exactly these problems. Not for lack of interest, and not for lack of members. They close because the technical bar for keeping a small forum online has risen enormously in about two years, and the person running it has a job and a life and no particular desire to become a full-time systems administrator by accident.
Twenty years ago you could put a forum on cheap shared hosting and forget about it. Today the same forum, with the same two hundred genuine visitors a day, will be found within days by scrapers and crawlers that neither know nor care how small it is. If nobody on the team can rebuild the caching layer and the session handling from the ground up, the board goes read-only, then it goes offline, and years of accumulated knowledge goes with it. The people move to a chat platform where nobody has to administer anything, and where everything said today will be effectively unfindable in three years.
That is the actual cost of all this, and it is a good deal higher than a server bill.
If you run a phpBB board and you recognise the symptoms in this article, I would be interested to hear from you. Not as a sales pitch, because there is nothing to sell at this point and no promises being made. But I have spent a long time on this particular problem and I would like to know how many other people are fighting the same thing, and whether what has been built here would be of any use to anybody else.
You can reach me through the forum. If enough people are in the same position, it is worth thinking about what could be done with it. If not, at least this article is on the record for the next administrator who goes looking at three in the morning and finds only the same twelve unhelpful search results everybody else has already found.