Server updates

[exxos]

I've extended the whitelist range on logged in users' IP addresses, that's about all I can do at this point.. But that fix may have to go away if it starts becoming the problem with increased traffic again.

It does make us a little bit vulnerable to mobile attacks when the IP addresses cannot be blocked but there is no way to easily differentiate between things unless it's all done on the application layer.. And I stopped doing that a couple weeks ago as the server couldn't keep up. So banning on the raw IO layer was the only efficient way to handle all the IPs.

Unfortunately, I just don't think there's going to be any solution to anything without some sort of collateral damage somewhere along the line. The server is set up to take one hell of a beating, how everything runs now. But obviously I cannot do anything about IP addresses which are good users, then bad users, then good users, then bad users as they're just going to end up banned. When there's millions of ips attacking us from mobile networks, banning is the only way we can keep on top of it all.

EDIT:
Having a quick look around and it seems that mobile networks actually do IP sharing across hundreds if not thousands of users. That would explain why weird attacks and requests which come right in the middle of legitimate quests from the same IP address.

[JezC]

Yup, had to un-ban myself several times while using mobile data earlier today...but at least that works so far!

[exxos]

Yup, had to un-ban myself several times while using mobile data earlier today...but at least that works so far!

Forum users should be white listed on ranges now, but how effective that will be for white listing and banning I have no idea...

Let me know if its better / worse.

I opened up a phpbb forum topic on similar.
https://www.phpbb.com/community/viewtop ... #p16078168

But TL;DR

A small chunk from a mobile IP yesterday.

ext.PNG (127.05 KiB) Viewed 776 times

You can see bad requests mixed in with proper forum links. So I suspect 2 people were using the same IP address. One users who we all know well, and another which is sending bad requests. Its not the first time I have seen this happen either.

But like I say, that address is sending bad requests likely from a bad bot, and also requests from a legitimate user on the same IP..

So do you ban it or not ban it..... The bad requests come from HTTP1 again..

The attacks from mobile networks lately is huge. it is why I just have to ban them. I cleared the blacklist again and we are hitting about 80% CPU on all 4 CPU cores.. Once backups kick in, even though CPU limited, it will start slowing down a bit.

Indeed tried to make the unbanning as simple and as quick as possible for people.. Its all I can really do at this point. Is why I suggest people not use mobiles because it is just going to be somewhat irritating if people keep getting banned..

I have been brainstorming with Grok, but it all really ends up revolving around requiring more CPU power to allow application layer filters. Thats what I had before, but that was also one reason why the forum went down last time.. The CPU can't keep up with all the requests even dishing out static 404 pages. So I just had to cut all the applications out of the loop and block earlier in the firewall chain. It works really well. But control over banning and whitelisting gets more problematical.

[exxos]

I seen a couple of people trying to unban. There was a problem with the unban not working with IP6 addresses. Now fixed.

If a IP is found banned, its now added to the whitelist. But like I said said yesterday, it won't work if your IP keeps changing.

I've had to revert to a previous "no agent" ruleset as the one I updated last night seems to be malfunctioning. I am not sure why as I cannot find any log entries to the banned IPs.

I did ask GTP, GROK, COPILOT, the same question why user agents are getting banned , they seem to be getting banned on the IP_CHECK page due to problems with user agents. But each AI gave me a totally different answer to what the problem is ! But I don't have time thismorning to investigate further. Will investigate tonight when I get back home.

[exxos]

I've found a problem in a ruleset and will do the fix later tonight.There is a bug in detecting brower versions. But it wouldn't generally effect legitimate traffic anyway. There may be other things a fault. I'm still investigating.

Also people need to put their username in the unban page if you want me to investigate why the ban happened so I can investigate the issue and pm you if I need more info etc.

Some users still are not staying logged in long enough for the system to pickup your ip to whitelist. Logging in and out constantly and getting banned is prettymuch self inflicted at this point!

People need to us the IP check page so the system logs the issue for me to investigate. If nobody does that, I'll assume there are no issues.

[czietz]

I seen a couple of people trying to unban. There was a problem with the unban not working with IP6 addresses. Now fixed.

FWIW, yesterday and this morning, I had big problems with banning. Just clicking on the login link on the forum start page was apparently enough to ban my IP. I tried different IPs via VPNs, mobile network etc, but was never successful getting past the login page before the respective IP would be banned.

It did work now, though.

But like I said said yesterday, it won't work if your IP keeps changing.

A bit problematic, since I don't have any Internet connect with a fixed IP. But I'm aware that this is my problem; so just as a comment, not as criticism.

[exxos]

FWIW, yesterday and this morning, I had big problems with banning. Just clicking on the login link on the forum start page was apparently enough to ban my IP. I tried different IPs via VPNs, mobile network etc, but was never successful getting past the login page before the respective IP would be banned.

It did work now, though.

If you can give me your latest IPs or better use the ip ban check page, I can look into that.

I've been updating rules the past 3'weeks. One jail seems to have a bug somewhere. Ill be working on the issue tonight.

If you stay logged in for 2 hours your ip range now gets whitelisted to help mitigate peoples IPs changing.

[czietz]

If you can give me your latest IPs or better use the ip ban check page, I can look into that.

Thank you! As my current IP is working (otherwise I wouldn't be posting), it probably makes no sense to investigate that. But I've bookmarked your IP check page now, and will use it if it happens again.

[exxos]

I am working on a resolution to the current bans.. It wasn't detecting HTTP1 traffic properly and classifying HTTP2 as bad traffic in some cases

I am currently empting the problem jail of all banned IPs. I have created a small IP LOG with known good and bad requests, with the help of a python script, I am testing those IPs to make sure detection works as its supposed to ! Hopefully that problem jail will behave afterwards !

[exxos]

All done.

There may be slight slowdowns for the next few hours while the ban list rebuilds. There was 100k banned ips in that one jail alone. The new jail should do better in not banning people by mistake. It also has stricter matching, so it should find bad bots more effectively. It's only been running a few seconds and several thousand ips have been identified in that one jail alone.