Server updates

[exxos]

Several months later...

Got a reply from canonical that they now looking into why livepatch is killing nginx...

I just ended up manually doing updates every every couple days because it didn't crash that way. Been good if they figured the problem out though. Would save me a fair bit of messing about.

[exxos]

I've switched the wiki to a newer php version just.

I've also noticed some maintenance scripts wasn't working as they were defaulting to the wrong PHP version..

Also fixed the donation list on the wiki donate page, it hadn't updated since April. Also was PHP version problem.

Indeed thanks to everyone who chips in a few quid each month or made a donation to keep the whole forum/wiki/websites running etc ! It really help me out and keeps everything running for everyone !
:cheers:

:thanksyellow: :2k2:

[exxos]

Just updated the firewall rules because I have noticed slipping through the net.. Nothing major but was a quick fix.

[exxos]

The server had been sporadic over the weekend because of the attacks. I had not been able to load the forum since around 5pm (sunday night) others said it was working fine.. It had been taking 30 seconds or more to load sporadically before that . So as to who and who could not get on I have no idea.. I could not do anything until I got home about 7pm.. Just spent almost 4 hours solving the issues.

I had trouble even logging into the server terminal as I kept getting connection timed out.. Even when I did eventually login it was taking about two minutes for every keypress to register in the terminal.. I could not connect via FTP to download the logs.. So it took an incredibly long time to shut down services and get the logs to figure out what was going on.

It seems there was a lot of unfortunate sequence of events in all this.. fail2ban had such a large list of blocked IPs, that the blocked time had elapsed and it was trying to unban them all.. While at the same time trying to add hundreds of thousands of IP's from the abuse database, while also trying to block hundreds of thousands of IP is attacking the server at the same time..

The abuse block list generally gives around 300 ips every 3 hours, and it was jumping to more like 200,000 , so it wasn't just us being hit as those IPs would have had to be hitting 100's of other servers as well...

Now I know that iptables sucks. I timed it and it was only processing five IPs per second :roll: apparently it rebuilds the entire lot every time something is changed which was starting to cause a huge slowdown server. Basically the whole lot ground to halt :roll:

So GPT to the rescue in figuring out how to solve all these problems and writing new scripts and code for me to implement. Now instead of things taking literally hours to update they are done in literally seconds now.

Things should run a lot quicker and be more responsive now in general. I have seen a massive speedbump here.

Fail2ban is currently catching up with itself and should settle down over the next few hours.. There is basically...

Capture.PNG

187,604 IPs which were hitting us.. Actually a lot more as they were being reported and removed from the list in real time. But it was fighting a losing battle at that point. They will be reported to abuseipdb but I can only report a few thousand a day.. So could take a while for all that to clear out.

I think what might have happened is China have realised everyone is blocking them now when they are using proxies all over the world... I have been getting huge traffic from like Brazil, India mobile networks, Pakistan, Iraq.. and well you get the idea..
Should now be solved anyway :hide: Will have a look at it all again in the morning to make sure everything is functioning as expected.. We shall be a lot faster processing all the IPs now..So hopefully this problem will not happen again...

[exxos]

AI and I have been busy on a new script :)

All logged in users in the form over the past week are now automatically white listed in the service firewall... So make sure you login at least once a week to make sure your IP address is recorded into the firewall. The whitelist is updated every hour.

I guess @PhilC will be the ideal person to test the white list out as he seems to be the most banned person on the forum so far :lol:

Its found 2 users which got banned.

1.PNG

I checked one user, and seems they toggled between logic and register pages multiple times which my script saw as a unusual set of events and banned. I have made that script more tolerant now as well.

Current ban queue is

Capture.PNG

So another 80,000 odd banned since my last post.

I also found issues with the IP6 firewall this morning and fixed that. Also a IP6 server issue. Also a firewall program wasn't running which now is again.

[exxos]

300+ pages of signup bots I reported last night.. Its far from over..

https://www.abuseipdb.com/user/170367

I looked into Brazil attacks briefly..

A May 2025 report highlighted the BadBox 2.0 botnet, with 400,000 compromised devices in Brazil (out of 1.6 million globally), often pirate TV boxes infected at the factory. These are used for DDoS attacks, credential stuffing, and creating fake accounts

Also saw this..

https://news.ycombinator.com/item?id=43738603

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization.
The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all of Brazil (sorry).

A few days later this same web server was on fire again. I performed the same analysis on IPs and found a similar number of unique addresses, but spread across Turkey, Russia, Argentina, Algeria and many more countries. What is going on?! Eventually I think I found a pattern to identify the requests, in that they were using ancient Chrome user agents. Chrome 40, 50, 60 and up to 90, all released 5 to 15 years ago.

Exactly what I've been seeing..

I looked at server stats and we had about 3GB bandwidth per month at the start of the year.. Now it's over 100GB a month. The logs show most of traffic is PHP.. IE. Forum..

Also

https://blog.asper.us/botnets-no-brasil ... rometidos/

How 400,000 compromised devices in Brazil are being used in attacks.
These devices, mostly pirated TV Boxes, are already being used for denial-of-service (DDoS) attacks, malware distribution and digital fraud, compromising not only home users but also corporate networks.

I've also seen India mobile networks hammering the server..

[exxos]

Almost a backlog of half a million IPs to report. I can only report 10,000 a day :( It kinda becomes pointless reporting to abuseipdb at this point as it will take 50 days just to clear the backlog, assume zero IPs attacking for the next 50 days, which is unlikely somehow ! I did contact abuseipdb about upping the limits but didn't get anywhere. I mean the attacks this year are like 10 fold of last year.

Capture.PNG

Even the site HTTP bandwidth tells the same story..

Capture.PNG

Hopefully that traffic will start going down now I am blocking way more traffic.

[exxos]

I had to redo some of the firewall stuff last night.. Mostly because I was trying to white list all of Google's IP is to rule out all the fake bots.. But listing millions of IPs individually would probably take up some GB of RAM :lol: :roll: so had to convert the lists over to a hash format so they would accept ranges...

Then the AI said the firewall rules wouldn't persist over reboots :shrug: So stuff have to be setup for that as well.

All googles IPs were removed from the block lists. It took all night for that script to complete. The white listing script has finished adding Google's IP is to the white list now.

This morning I had to fix the site map issues because Google was blocked, and I have also noticed a few other issues which need fixing now :roll:

I have also updated the IP BAN CHECK page.. Now it asks for a forum username. Is not mandatory but it will help diagnose issues quicker that way.

When I get chance I will make it so that registered forum users will automatically get unbanned if they use that page (***).. As of course I'm not sat at the computer 24/7 ! "guests" will have to be manually evaluated and removed from the list which can take a few hours to a few days..

(***) Users which login each week will automatically be added to the white list anyway.. But if someone does login for a while and manages to trigger a firewall rule, then that user will get banned will have to request IP to be removed manually.. As always always investigate why the ban happened and will adjust the rules accordingly.

We are still being hit incredibly hard from Brazil ( but of course not limited to). Though the server is blocking those IPs fine. The block list backlog is now half a million IPs !!

[exxos]

Seems to be Pakistan and Kenya which seem to be attacking us currently..

kenya.PNG

pak.PNG

The abuseipdb support said I can use their bulk reporter for up to batches of 10,000.. I misunderstood that as I thought that the 10,000 was the limit per day which would not be any difference.. So I've been busy writing new scripts to do bulk reporting as there is now half a million backlog of IP's !

Is pretty weird because it seems like Grok is more creative about writing code but makes more mistakes.. While GPT seems to struggle a little bit of writing code, but seems to be better at error checking.. So I've been using Grok to write the initial code, then GPT to fix it :lol: GPT always seems to lose context when explaining stuff. But I noticed that before. It's like it will write you routines, but when you explain multiple things, it seems to "forget" some stuff which you asked it to do.

Grok has also fixed a lot of store code today for me as well. I've been busy fixing Google site maps and related stuff all day. Waiting for the Google validation thing to finish messing about so I can see if stuff is working better or not now...

[Steve]

It's very impressive how you've become such an internet server security expert in the last year or so, I hope one day you'll be able to sit back and relax without having to worry about the server all the time though.