Can all registered uses please login, even just for a few minutes..
It helps build a picture where our "good traffic" is coming from..
Thanks :)
It helps build a picture where our "good traffic" is coming from..
Thanks :)
Server updates
Re: Server updates
Server will be having some DNS updates and fixes now. Some people may see DNS errors until its all tried and tested.
Re: Server updates
IPv6 was down for three days, and here is why
If you connect over IPv6, the site has been unreachable since Tuesday morning. Everything on IPv4 carried on as normal, which is exactly why it went unnoticed for so long.
The cause turned out to be a routine package upgrade on the 11th. systemd-networkd restarted as part of it, the server lost its global IPv6 address, and it never got it back. The interface stayed up throughout, so nothing looked broken from the inside.
A packet capture showed what was really going on. The server was asking the network for its address, the network was answering correctly within a couple of milliseconds every single time, and the server was then quietly losing its nerve and asking again. Two minutes later, same question, same answer, same silence. It kept this up for three days without ever once completing the conversation.
The fix was to stop asking. The address is derived from the network card and never changes, so it is now set statically instead of being requested at boot. That removes the broken exchange from the picture entirely.
One unexpected bonus: a couple of days of intermittent email delays turned out to be the same fault, and those have gone too.
If you connect over IPv6, the site has been unreachable since Tuesday morning. Everything on IPv4 carried on as normal, which is exactly why it went unnoticed for so long.
The cause turned out to be a routine package upgrade on the 11th. systemd-networkd restarted as part of it, the server lost its global IPv6 address, and it never got it back. The interface stayed up throughout, so nothing looked broken from the inside.
A packet capture showed what was really going on. The server was asking the network for its address, the network was answering correctly within a couple of milliseconds every single time, and the server was then quietly losing its nerve and asking again. Two minutes later, same question, same answer, same silence. It kept this up for three days without ever once completing the conversation.
The fix was to stop asking. The address is derived from the network card and never changes, so it is now set statically instead of being requested at boot. That removes the broken exchange from the picture entirely.
One unexpected bonus: a couple of days of intermittent email delays turned out to be the same fault, and those have gone too.
Re: Server updates
I'm going to be busy building new extensions and fixes for the next couple weeks.. Its just clear that phpbb simply wasn't designed for our modern internet traffic. Considering how old the core is, it's of no surprise. I'm basically refactoring how phpbb handles traffic. This is not easy. I didn't want to start making core changes as every forum update would undo all the work. But thankfully it looks like a lot of extensions can do the job.
The extensions have many functions for traffic monitoring, and making sure nothing blocks memebers by mistake. There is a whole arsonal of tools under development. Registered users will get access to a extension where if their ip gets blocked, they log in with a different ip and can unblock themselves. That is feed back into other extensions which monitor reputation of networks. I'll not go into how it all works as it would probably take 20 pages to explain it all.
Problem is phpbb handles various pages differently which all need a extension to address. A bit of a mess is a understatement. It's not just extensions. They are basically just a GUI for what goes on in the background and tied in deeply which how nginx handles traffic. It's sort of like automatic traffic shaping. It's clear that the traffic changes every few days and keep tweaking stuff to compensate is a loosing battle. So rather than trying to deal with the traffic, phpbb gets to decide what traffic gets at resources and basically talks to the firewall automatically when something doesn't look right. All these changes are a huge undertaking but I believe changing how phpbb deals with traffic in the first place is the direction to go in.
Block lists have all gone. Anything they blocked has been removed. They are to slow to keep up and end up blocking memebrs or good bots I want to allow. Trying to filter out what traffic I want to keep vs what block lists were saying block was again fighting a loosing battle. Users these days report all AI scrapers like Claude, and I like Claude and don't want him blocked. But the blocklists disagree and block him. So in order to reduce all such conflicts, I'm just not using blocklists anymore. It just constantly creates more variables and moving parts that I just don't have time to deal with. So the whole system had to go.
The whitelist system is sort of gone. With ip rotation the list was just going stale every few days. So extensions monitor it all a lot better now being tied into the phpbb core itself. But users should still login as often as possible for the system to build a picture of where our good traffic is coming from. Most don't default to a hard ban now and hit 403 pages. Things are still being tweaked currently though.
Rate limits have basically been all taken out. They were doing more harm than good in the end. It caused busy users to hit 403 type pages mostly. Busy users could not be reliably seperated from bots. Like I mentioned a couple weeks or so ago, members get allocated resources as well as guests. If any guest traffic floods the server, it takes down the guest pool. Memebers will be totally unaware anything even happened. The traffic which floods the most will just hit 403 type pages and if continues they get banned.
So instead of trying to fight the traffic, a much better system is being built. The end game and likely worst case, is guest access goes down. Users logged in get priority resources and that means memebers should never effectively see the server go down. That's the aim these days. Guest traffic gets a kind of reputation based resource limit and the worst traffic just gets dropped or banned automatically.
None of this is quick to setup. But I think it will ultimately be the best system to have in the end.
The extensions have many functions for traffic monitoring, and making sure nothing blocks memebers by mistake. There is a whole arsonal of tools under development. Registered users will get access to a extension where if their ip gets blocked, they log in with a different ip and can unblock themselves. That is feed back into other extensions which monitor reputation of networks. I'll not go into how it all works as it would probably take 20 pages to explain it all.
Problem is phpbb handles various pages differently which all need a extension to address. A bit of a mess is a understatement. It's not just extensions. They are basically just a GUI for what goes on in the background and tied in deeply which how nginx handles traffic. It's sort of like automatic traffic shaping. It's clear that the traffic changes every few days and keep tweaking stuff to compensate is a loosing battle. So rather than trying to deal with the traffic, phpbb gets to decide what traffic gets at resources and basically talks to the firewall automatically when something doesn't look right. All these changes are a huge undertaking but I believe changing how phpbb deals with traffic in the first place is the direction to go in.
Block lists have all gone. Anything they blocked has been removed. They are to slow to keep up and end up blocking memebrs or good bots I want to allow. Trying to filter out what traffic I want to keep vs what block lists were saying block was again fighting a loosing battle. Users these days report all AI scrapers like Claude, and I like Claude and don't want him blocked. But the blocklists disagree and block him. So in order to reduce all such conflicts, I'm just not using blocklists anymore. It just constantly creates more variables and moving parts that I just don't have time to deal with. So the whole system had to go.
The whitelist system is sort of gone. With ip rotation the list was just going stale every few days. So extensions monitor it all a lot better now being tied into the phpbb core itself. But users should still login as often as possible for the system to build a picture of where our good traffic is coming from. Most don't default to a hard ban now and hit 403 pages. Things are still being tweaked currently though.
Rate limits have basically been all taken out. They were doing more harm than good in the end. It caused busy users to hit 403 type pages mostly. Busy users could not be reliably seperated from bots. Like I mentioned a couple weeks or so ago, members get allocated resources as well as guests. If any guest traffic floods the server, it takes down the guest pool. Memebers will be totally unaware anything even happened. The traffic which floods the most will just hit 403 type pages and if continues they get banned.
So instead of trying to fight the traffic, a much better system is being built. The end game and likely worst case, is guest access goes down. Users logged in get priority resources and that means memebers should never effectively see the server go down. That's the aim these days. Guest traffic gets a kind of reputation based resource limit and the worst traffic just gets dropped or banned automatically.
None of this is quick to setup. But I think it will ultimately be the best system to have in the end.
Re: Server updates
Just to give an idea what is being built in the background..
It basically is a list of all networks in all countries, and it lists if any members have been seen in those networks and everything is flanked accordingly. If the system detects a member could be potentially blocked it flags up a warning and a simple "fix it now" type button, then its all logged what actually triggered any potential traffic drops or bans and the system just tells me pretty much outright what happens rather than me spending hours looking everywhere. There are various options of protecting users and the things that I won't list them all because I don't want to give the game away of how our protection and detection systems are working behind-the-scenes on a public forum.
Mostly these are just a failsafe and monitoring systems but also phpbb sees a lot of the traffic which is all reported. If anything is found looking even slightly suspicious it flags up a warning to the moderators to look into.
As said previously, it helps if users can login regularly so the system can continue to monitor where all the good traffic is currently coming from.
It basically is a list of all networks in all countries, and it lists if any members have been seen in those networks and everything is flanked accordingly. If the system detects a member could be potentially blocked it flags up a warning and a simple "fix it now" type button, then its all logged what actually triggered any potential traffic drops or bans and the system just tells me pretty much outright what happens rather than me spending hours looking everywhere. There are various options of protecting users and the things that I won't list them all because I don't want to give the game away of how our protection and detection systems are working behind-the-scenes on a public forum.
Mostly these are just a failsafe and monitoring systems but also phpbb sees a lot of the traffic which is all reported. If anything is found looking even slightly suspicious it flags up a warning to the moderators to look into.
- 1.PNG (123.97 KiB) Viewed 36 times
- 2.PNG (308.43 KiB) Viewed 36 times
- 3.PNG (81.35 KiB) Viewed 36 times
As said previously, it helps if users can login regularly so the system can continue to monitor where all the good traffic is currently coming from.
Re: Server updates
Anti-bot system update, plus a few odds and ends
Looks like the system is holding up just fine. I have been monitoring things each day and everything has a green light basically.
I have noticed Grok seems to be getting blocked. But there doesn't seem to be a published IP list to whitelist. Problem is, the IP keeps changing and its randomly finding itself in blocked ranges. Nothing I can really do about that one.
The largest flood of traffic hit a few days ago, but didn't even make a dent in what we can server these days.
Following on from the last update, the bot and AI detection system is now properly up and running behind the scenes, and it's had a proper week of tweaking and testing to go with it.
The main thing you might actually notice: very occasionally when posting, you could be asked to click a shape that's different from the others in a small grid. It only shows up in specific situations, most of you will likely never see it at all. If you do, just click the odd one out and carry on as normal.
There's a fair bit going on under the hood to try and protect genuine members from ever getting caught up in this sort of thing. Various checks run automatically specifically to stop real users from being wrongly banned, and because of that, actual banning is now pretty rare, it's mostly just quietly keeping an eye on things instead. Anything that does look genuinely off gets flagged straight to the moderator team's notifications immediately, so it's not just sitting there unnoticed.
One thing worth repeating in case it got lost in the last post, if a post ever does get held back for a quick look rather than going straight up, it is NOT lost or deleted, it's just sitting in a short queue waiting for one of us to glance at it. No need to retype your life story a second time.
Also been chasing down a few smaller gremlins this week, including a login page hiccup and a couple of other bits that were quietly misbehaving, all sorted now.
As ever, not going into detail on how any of it actually works under the bonnet. Wouldn't want to hand out the answers.
I have not noticed any issues and certainly nobody has reported any issues, so I would say everything is completely solved at this point !
As a side note, if the forum does actually run slow actually go off-line for a few moments, that is completely my fault as I am running various AI taskes on the server which can max out the CPU sporadically. But I think it has only happened once so far and the outage was only brief.
Looks like the system is holding up just fine. I have been monitoring things each day and everything has a green light basically.
- Capture.PNG (402.37 KiB) Viewed 16 times
I have noticed Grok seems to be getting blocked. But there doesn't seem to be a published IP list to whitelist. Problem is, the IP keeps changing and its randomly finding itself in blocked ranges. Nothing I can really do about that one.
The largest flood of traffic hit a few days ago, but didn't even make a dent in what we can server these days.
- Capture.PNG (108.62 KiB) Viewed 16 times
Following on from the last update, the bot and AI detection system is now properly up and running behind the scenes, and it's had a proper week of tweaking and testing to go with it.
The main thing you might actually notice: very occasionally when posting, you could be asked to click a shape that's different from the others in a small grid. It only shows up in specific situations, most of you will likely never see it at all. If you do, just click the odd one out and carry on as normal.
There's a fair bit going on under the hood to try and protect genuine members from ever getting caught up in this sort of thing. Various checks run automatically specifically to stop real users from being wrongly banned, and because of that, actual banning is now pretty rare, it's mostly just quietly keeping an eye on things instead. Anything that does look genuinely off gets flagged straight to the moderator team's notifications immediately, so it's not just sitting there unnoticed.
One thing worth repeating in case it got lost in the last post, if a post ever does get held back for a quick look rather than going straight up, it is NOT lost or deleted, it's just sitting in a short queue waiting for one of us to glance at it. No need to retype your life story a second time.
Also been chasing down a few smaller gremlins this week, including a login page hiccup and a couple of other bits that were quietly misbehaving, all sorted now.
As ever, not going into detail on how any of it actually works under the bonnet. Wouldn't want to hand out the answers.
I have not noticed any issues and certainly nobody has reported any issues, so I would say everything is completely solved at this point !
As a side note, if the forum does actually run slow actually go off-line for a few moments, that is completely my fault as I am running various AI taskes on the server which can max out the CPU sporadically. But I think it has only happened once so far and the outage was only brief.
Re: Server updates
Some minor DNS changes are being done.. nothing should be affected but you never know..