Server updates

[exxos]

We now have some weird Chinese bot doing God knows what to the wiki. Also a massive botnet trying to login and create accounts on the wiki. :roll:

I've been trying not to ban all the IPs because of the sheer amount of them.. (*) But I've actually gone back to this approach of sorts.. Because my aggregation script will scan the IP addresses every hour and ban permanently entire subsets anyway.

(*) Annoyingly most IPs only hit the server once a month. But there are 100's of thousands of IPs each day hitting various things which causes a significant load when the forum and wiki are hit. Life was a lot simpler with basic HTML pages.. Immensely lightweight and nobody cares about them any more ! :lol:

I've also started dropping connections more in nginx. They hit the logs so failban will pick them up. But they don't get as far as hitting PHP and SQL which are the main resource hogs.. So now I think I have solved that problem, the CP was biking because of the traffic fail2ban is now having to process :lol: :roll: but I suppose eventually it will settle down when enough ranges have been blocked..

[exxos]

Today's attack is brought to you by a Vietnamese botnet it seems. They are faking session tokens.. So now all the fake tokens are banned. In fact there is huge ranges which are banned when there are a lot of hits in particular sub sets.

The wiki is still under attack from a botnet trying to login and create accounts but this is currently under control. They are all being banned but there are just so many IPs... Ranges get blocked automatically every few hours anyway.

Also getting hits to the forum rules page which is just bizarre. several hours over 40,000 hits. need to look into that next..

[exxos]

That's enough for today !

Capture.PNG

[exxos]

Was just going to log off when a huge botnet started attacking us !

Maxed out the CPU, and SQL was getting overloaded !

Capture.PNG

Capture.PNG

[rubber_jonnie]

Very weird this morning, I still couldn't get on the site and had to reboot my laptop to get back in.

[exxos]

Very weird this morning, I still couldn't get on the site and had to reboot my laptop to get back in.

IP change?

Best to check the ban page to see if IP blocked or not..

[rubber_jonnie]

Very weird this morning, I still couldn't get on the site and had to reboot my laptop to get back in.

IP change?

Best to check the ban page to see if IP blocked or not..

Unlikely as I just rebooted my laptop and not my router.

EDIT - my current address is IPV6 and not banned.

[exxos]

Maybe stale cache / connection :shrug:

AI did mention there was no timouts for connections. That's fixed now. Hopefully it will help next time we get attacked.

Also raised various connections limits, for the 3rd time this month. But we just at the CPU limits now.

[exxos]

"meta-externalagent/1.1" now banned. I didn't want to start banning "good crawlers" but its not obeying robots.txt and trying to opening up thousands of connections. It's done about 200,000 requests so far which started slowing the down the forum just :roll:

Believe it or not, we still being hit hard from the Brazil botnet attacks. Wider ranges of IPs are now being blocked. These ones are not even showing up in any blocklists yet either! But they are being banned in realtime by fail2ban thankfully anyway.

A script scans the IP ranges every hour and blocks them automatically in the firewall. The idea is , if a botnet starts attacking us, after a hour, those IPs get "grouped" and blocked by ranges. So "in theory" it should "pre-block" ranges which haven't yet started hitting us. It does work, but as attacks keep getting bigger and bigger, scripts and rules have to be refined to keep up. I've done a lot of changes to the server config these past couple weeks to help with it all again.

[exxos]

I have done overhaul today on how the firewall stuff operates. Half of the fail2ban rules running were not even catching any IPs anymore as other rules written later basically took over anyway. So it saved a bit of overhead. I've also significantly reduced the memory usage due to a "oopsie" in the fail2ban config.

SQL,PHP-FPM, Nginx have had a lot of changes made as well. We should be able to handle spikes in traffic a lot easier now. We hit 20,000 guests yesterday all hammering the forum. The wiki was getting hammered along with that facebook bot.. Some of them seem to come from cloud and data centre infrastructures are not even legitimate users ! It was all a bit of a mess !

A lot of rules have basically become redundant because the main attacks of late are all botnets. As I previously feared, they have now moved on to HTTP2 protocols :( Fail2ban isn't really helping with botnet attacks as they all look like legitimate traffic. Though there has been a additional two scripts written where I won't go into the details, but each server request is given a "score" and if that score gets high enough, then a massive IP range gets nuked automatically. That script has to scan the logs repeatedly which takes a bit of CPU but its less CPU than fail2ban was using with all the now redundant jails anyway.

I found some of the IP addresses in some block lists, so I've added more lists, but I don't think their effectiveness helping at all any more. Most of the IPs we are being hit with are not in any lists at all ! So I don't know if such lists have just been given up on now with maintaining them. Also considering cloudflare seems to be in front of pretty much everything these days anyway..

TL;DR
Lots of optimisations have gone on today. The forum seems to be really snappy at the moment as well :thumbup: