Can all registered uses please login, even just for a few minutes..
It helps build a picture where our "good traffic" is coming from..
Thanks :)
It helps build a picture where our "good traffic" is coming from..
Thanks :)
Server updates
Re: Server updates
Server will be having some DNS updates and fixes now. Some people may see DNS errors until its all tried and tested.
Re: Server updates
IPv6 was down for three days, and here is why
If you connect over IPv6, the site has been unreachable since Tuesday morning. Everything on IPv4 carried on as normal, which is exactly why it went unnoticed for so long.
The cause turned out to be a routine package upgrade on the 11th. systemd-networkd restarted as part of it, the server lost its global IPv6 address, and it never got it back. The interface stayed up throughout, so nothing looked broken from the inside.
A packet capture showed what was really going on. The server was asking the network for its address, the network was answering correctly within a couple of milliseconds every single time, and the server was then quietly losing its nerve and asking again. Two minutes later, same question, same answer, same silence. It kept this up for three days without ever once completing the conversation.
The fix was to stop asking. The address is derived from the network card and never changes, so it is now set statically instead of being requested at boot. That removes the broken exchange from the picture entirely.
One unexpected bonus: a couple of days of intermittent email delays turned out to be the same fault, and those have gone too.
If you connect over IPv6, the site has been unreachable since Tuesday morning. Everything on IPv4 carried on as normal, which is exactly why it went unnoticed for so long.
The cause turned out to be a routine package upgrade on the 11th. systemd-networkd restarted as part of it, the server lost its global IPv6 address, and it never got it back. The interface stayed up throughout, so nothing looked broken from the inside.
A packet capture showed what was really going on. The server was asking the network for its address, the network was answering correctly within a couple of milliseconds every single time, and the server was then quietly losing its nerve and asking again. Two minutes later, same question, same answer, same silence. It kept this up for three days without ever once completing the conversation.
The fix was to stop asking. The address is derived from the network card and never changes, so it is now set statically instead of being requested at boot. That removes the broken exchange from the picture entirely.
One unexpected bonus: a couple of days of intermittent email delays turned out to be the same fault, and those have gone too.
Re: Server updates
I'm going to be busy building new extensions and fixes for the next couple weeks.. Its just clear that phpbb simply wasn't designed for our modern internet traffic. Considering how old the core is, it's of no surprise. I'm basically refactoring how phpbb handles traffic. This is not easy. I didn't want to start making core changes as every forum update would undo all the work. But thankfully it looks like a lot of extensions can do the job.
The extensions have many functions for traffic monitoring, and making sure nothing blocks memebers by mistake. There is a whole arsonal of tools under development. Registered users will get access to a extension where if their ip gets blocked, they log in with a different ip and can unblock themselves. That is feed back into other extensions which monitor reputation of networks. I'll not go into how it all works as it would probably take 20 pages to explain it all.
Problem is phpbb handles various pages differently which all need a extension to address. A bit of a mess is a understatement. It's not just extensions. They are basically just a GUI for what goes on in the background and tied in deeply which how nginx handles traffic. It's sort of like automatic traffic shaping. It's clear that the traffic changes every few days and keep tweaking stuff to compensate is a loosing battle. So rather than trying to deal with the traffic, phpbb gets to decide what traffic gets at resources and basically talks to the firewall automatically when something doesn't look right. All these changes are a huge undertaking but I believe changing how phpbb deals with traffic in the first place is the direction to go in.
Block lists have all gone. Anything they blocked has been removed. They are to slow to keep up and end up blocking memebrs or good bots I want to allow. Trying to filter out what traffic I want to keep vs what block lists were saying block was again fighting a loosing battle. Users these days report all AI scrapers like Claude, and I like Claude and don't want him blocked. But the blocklists disagree and block him. So in order to reduce all such conflicts, I'm just not using blocklists anymore. It just constantly creates more variables and moving parts that I just don't have time to deal with. So the whole system had to go.
The whitelist system is sort of gone. With ip rotation the list was just going stale every few days. So extensions monitor it all a lot better now being tied into the phpbb core itself. But users should still login as often as possible for the system to build a picture of where our good traffic is coming from. Most don't default to a hard ban now and hit 403 pages. Things are still being tweaked currently though.
Rate limits have basically been all taken out. They were doing more harm than good in the end. It caused busy users to hit 403 type pages mostly. Busy users could not be reliably seperated from bots. Like I mentioned a couple weeks or so ago, members get allocated resources as well as guests. If any guest traffic floods the server, it takes down the guest pool. Memebers will be totally unaware anything even happened. The traffic which floods the most will just hit 403 type pages and if continues they get banned.
So instead of trying to fight the traffic, a much better system is being built. The end game and likely worst case, is guest access goes down. Users logged in get priority resources and that means memebers should never effectively see the server go down. That's the aim these days. Guest traffic gets a kind of reputation based resource limit and the worst traffic just gets dropped or banned automatically.
None of this is quick to setup. But I think it will ultimately be the best system to have in the end.
The extensions have many functions for traffic monitoring, and making sure nothing blocks memebers by mistake. There is a whole arsonal of tools under development. Registered users will get access to a extension where if their ip gets blocked, they log in with a different ip and can unblock themselves. That is feed back into other extensions which monitor reputation of networks. I'll not go into how it all works as it would probably take 20 pages to explain it all.
Problem is phpbb handles various pages differently which all need a extension to address. A bit of a mess is a understatement. It's not just extensions. They are basically just a GUI for what goes on in the background and tied in deeply which how nginx handles traffic. It's sort of like automatic traffic shaping. It's clear that the traffic changes every few days and keep tweaking stuff to compensate is a loosing battle. So rather than trying to deal with the traffic, phpbb gets to decide what traffic gets at resources and basically talks to the firewall automatically when something doesn't look right. All these changes are a huge undertaking but I believe changing how phpbb deals with traffic in the first place is the direction to go in.
Block lists have all gone. Anything they blocked has been removed. They are to slow to keep up and end up blocking memebrs or good bots I want to allow. Trying to filter out what traffic I want to keep vs what block lists were saying block was again fighting a loosing battle. Users these days report all AI scrapers like Claude, and I like Claude and don't want him blocked. But the blocklists disagree and block him. So in order to reduce all such conflicts, I'm just not using blocklists anymore. It just constantly creates more variables and moving parts that I just don't have time to deal with. So the whole system had to go.
The whitelist system is sort of gone. With ip rotation the list was just going stale every few days. So extensions monitor it all a lot better now being tied into the phpbb core itself. But users should still login as often as possible for the system to build a picture of where our good traffic is coming from. Most don't default to a hard ban now and hit 403 pages. Things are still being tweaked currently though.
Rate limits have basically been all taken out. They were doing more harm than good in the end. It caused busy users to hit 403 type pages mostly. Busy users could not be reliably seperated from bots. Like I mentioned a couple weeks or so ago, members get allocated resources as well as guests. If any guest traffic floods the server, it takes down the guest pool. Memebers will be totally unaware anything even happened. The traffic which floods the most will just hit 403 type pages and if continues they get banned.
So instead of trying to fight the traffic, a much better system is being built. The end game and likely worst case, is guest access goes down. Users logged in get priority resources and that means memebers should never effectively see the server go down. That's the aim these days. Guest traffic gets a kind of reputation based resource limit and the worst traffic just gets dropped or banned automatically.
None of this is quick to setup. But I think it will ultimately be the best system to have in the end.
Re: Server updates
Just to give an idea what is being built in the background..
It basically is a list of all networks in all countries, and it lists if any members have been seen in those networks and everything is flanked accordingly. If the system detects a member could be potentially blocked it flags up a warning and a simple "fix it now" type button, then its all logged what actually triggered any potential traffic drops or bans and the system just tells me pretty much outright what happens rather than me spending hours looking everywhere. There are various options of protecting users and the things that I won't list them all because I don't want to give the game away of how our protection and detection systems are working behind-the-scenes on a public forum.
Mostly these are just a failsafe and monitoring systems but also phpbb sees a lot of the traffic which is all reported. If anything is found looking even slightly suspicious it flags up a warning to the moderators to look into.
As said previously, it helps if users can login regularly so the system can continue to monitor where all the good traffic is currently coming from.
It basically is a list of all networks in all countries, and it lists if any members have been seen in those networks and everything is flanked accordingly. If the system detects a member could be potentially blocked it flags up a warning and a simple "fix it now" type button, then its all logged what actually triggered any potential traffic drops or bans and the system just tells me pretty much outright what happens rather than me spending hours looking everywhere. There are various options of protecting users and the things that I won't list them all because I don't want to give the game away of how our protection and detection systems are working behind-the-scenes on a public forum.
Mostly these are just a failsafe and monitoring systems but also phpbb sees a lot of the traffic which is all reported. If anything is found looking even slightly suspicious it flags up a warning to the moderators to look into.
- 1.PNG (123.97 KiB) Viewed 10 times
- 2.PNG (308.43 KiB) Viewed 10 times
- 3.PNG (81.35 KiB) Viewed 10 times
As said previously, it helps if users can login regularly so the system can continue to monitor where all the good traffic is currently coming from.