Server updates

[exxos]

Our largest DOS attack yet.

Capture.PNG

Over 167,000 IPs on top of that. Seem to be coming from Asia.

General pattern from your list

Most of the large /16 blocks you’re blocking are consumer ISP address pools — especially:

PLDT / Philippines ISP networks (multiple 112.x and 49.x blocks)

Other APNIC‑allocated Asia blocks (e.g., 223.x, 14.x) likely belonging to regional carriers or ISPs.

This aligns with typical botnet behavior, where malware on consumer devices (home routers, PCs) in large ISP blocks generate traffic.

Started running my botnet scripts every half an hour in order to keep on top of it all.. The amount of IP's is just mind blowing along with the ranges of them..

It's bad enough with a mobile IPs because one moment they are good traffic and the next moment they are bad traffic and because of the sheer amount of them I just have no choice but to block them in order to get server online. They are basically infected networks in my book. Similar with all the malware in consumer devices like we have seen before. The actual home user might actually be a legitimate use above their network is infected where again I just have to block it. There's simply nothing I can do about infected networks.Even more evidently clear that pretty much every site I go on now is behind cloudflare. Sign of the times...

In other news, certificates which use GETSSL, stopped working. For some bizarre reason the binary had completely disappeared!! So just had to reinstall it! :crazy:

Also Grok suggested some fixes to make the SQL database queries a bit more efficiently seem to have helped a lot as well.

Well, that's the best part of my day gone again now :roll:

[exxos]

Just uncovered another botnet. Its constatnly hitting the wiki "forgot password" page 100's thousand times.. and some other links which I am now adding a ban rule for.

I don't know if this is happening for any particular user on the wiki, there is not really that many accounts. Not really sure what the point would be.. It is just a drain on SQL resources more than anything :shrug:

Capture.PNG

[exxos]

The current botnet is a tricky blighter.. Basically they are coming from Brazil, India, Africa, Middle East, SE Asia carriers etc

I am working on a new script to detect these new botnets, they are also detecting COMCAST and BT as botnets as well. I am going to have to be careful about what ranges I block..

I think it just got to the point where I will just have to start nuking whole countries, where they are unlikely to be accessing the forum or the wiki in the first place. It is literally like half of the Internet is now infected at this point !

[alexh]

Is it possible to have daily country quotas and after the daily quota is reached they are blocked until tomorrow?

[exxos]

Is it possible to have daily country quotas and after the daily quota is reached they are blocked until tomorrow?

A good thought, but the problem is the amount of IP's and the amount of RAM it takes to store them all to unban later. Thats part of the problem I've been fighting the past couple of weeks or so.

There's a lot of layers of things going on at the moment.. A lot of traffic is blocked on the nginx level and then blocked in the firewall. The problem is fail2ban can't store the IPs to "unban later", We have 4GB RAM in the server, and "last week" we hit 1.5GB in fail2ban RAM usage because of all the stored IPs it found and banned. Its why I had to stop doing that and go for a "one shot" ban only it does not eat up RAM.

The second layer problem to that, is that the sheer amount of IP addresses which get banned.. So a second script consolidates IP addresses into ranges. Later that consolidation is consolidated into a even bigger range. The individual IP addresses are removed from the firewall because that list gets massive very quickly as well.

Its pretty mind blowing the amount of IP addresses and ranges now. Even the botnets I blocked originally from Brazil are still hitting pages. But that traffic has significantly slowed down because of all the banning this been going on for the past few months. But I think it was pretty much the whole of Brazil which was infected going by the amount of IP addresses coming from there. Even my server host company said they just downright blocked the whole of Brazil and had done with it. I've been trying to not do that sort of thing. But now we are being hit from other countries just as hard..

It's not just dealing with one botnet from one country's multiple botnets worldwide from various other countries all at once..

I just created (well AI did) a script to ban the worst offenders.

For example

The current pass..

Code: Select all

 [DRY-RUN] BLOCK  45.0.0.0/8
  [DRY-RUN] BLOCK  177.0.0.0/8
  [DRY-RUN] BLOCK  103.0.0.0/8
  [DRY-RUN] BLOCK  187.0.0.0/8
  [DRY-RUN] BLOCK  186.0.0.0/8
  [DRY-RUN] BLOCK  179.0.0.0/8
  [DRY-RUN] BLOCK  190.0.0.0/8
  [DRY-RUN] BLOCK  181.0.0.0/8
  [DRY-RUN] BLOCK  197.0.0.0/8
  [DRY-RUN] BLOCK  102.0.0.0/8
  [DRY-RUN] BLOCK  37.0.0.0/8
  [DRY-RUN] BLOCK  189.0.0.0/8
  [DRY-RUN] BLOCK  191.0.0.0/8
  [DRY-RUN] BLOCK  41.0.0.0/8
  [DRY-RUN] BLOCK  170.0.0.0/8
  [DRY-RUN] BLOCK  49.0.0.0/8
  [DRY-RUN] BLOCK  201.0.0.0/8
  [DRY-RUN] BLOCK  5.0.0.0/8
  [DRY-RUN] BLOCK  185.0.0.0/8
  [DRY-RUN] BLOCK  180.75.0.0/16
  [DRY-RUN] BLOCK  85.254.0.0/16
  [DRY-RUN] BLOCK  152.59.0.0/16
  [DRY-RUN] BLOCK  152.58.0.0/16
  [DRY-RUN] BLOCK  106.219.0.0/16
  [DRY-RUN] BLOCK  79.116.0.0/16
  [DRY-RUN] BLOCK  79.117.0.0/16
  [DRY-RUN] BLOCK  161.142.0.0/16
  [DRY-RUN] BLOCK  223.181.0.0/16
  [DRY-RUN] BLOCK  223.185.0.0/16
  [DRY-RUN] BLOCK  92.208.0.0/16
  [DRY-RUN] BLOCK  180.74.0.0/16
  [DRY-RUN] BLOCK  51.36.0.0/16
  [DRY-RUN] BLOCK  27.125.0.0/16
  [DRY-RUN] BLOCK  143.105.0.0/16
  [DRY-RUN] BLOCK  138.226.0.0/16
  [DRY-RUN] BLOCK  59.103.0.0/16
  [DRY-RUN] BLOCK  38.154.161.0/24
  [DRY-RUN] BLOCK  14.192.215.0/24
  [DRY-RUN] BLOCK  121.121.57.0/24
  [DRY-RUN] BLOCK  14.192.213.0/24
  [DRY-RUN] BLOCK  118.101.170.0/24
  [DRY-RUN] BLOCK  77.223.252.0/24
  [DRY-RUN] BLOCK  14.192.214.0/24
  [DRY-RUN] BLOCK  159.148.126.0/24

My previous botnet script is still running every half an hour and is getting around 15 wide IP ranges each run. That is starting to slow down now as well.

Will check the logs again after dinner and see if this traffic has significantly slowed down..

So I cannot emphasise enough to the forum users that they need to use a fixed IP address and make sure you are logged in for a couple of hours so you get white listed.. Because the ranges of IPs getting banged is only ever increasing to keep on top of it all.. Which further and further increases the risk of collateral damage otherwise..

EDIT:

Single IP's do have a timeout where they get unbanned eventually. But if they get "promoted" to a range, then it won't help. But as soon as IP's get unbanned, the same ones come flooding back anyway. So temp bans for the most part don't really seem to help. The bad traffic just keep coming..

[alexh]

We have 4GB RAM in the server

What type of RAM does the server take? DDR3/DDR4/DDR5? Regular or LP? Full size DIMMs? SODIMMs? Do you have any free DIMM slots?

I designed the DDR4 RAM controller here, we've moved to DDR5LP and I have a phenomenal amount of DDR4 DIMMs just sitting here doing nothing. I am sure I could find 16GB or 32GB of DIMMs for you.

[exxos]

What type of RAM does the server take? DDR3/DDR4/DDR5? Regular or LP? Full size DIMMs? SODIMMs? Do you have any free DIMM slots?

I designed the DDR4 RAM controller here, we've moved to DDR5LP and I have a phenomenal amount of DDR4 DIMMs just sitting here doing nothing. I am sure I could find 16GB or 32GB of DIMMs for you.

Thanks.. but I don't own the server, its rented from the Netherlands..

[exxos]

CPU stress yesterday.. Very low now..

1.png

And since blocking those new ranges just...

2.PNG

There's hardly any activity in the logs now...

[exxos]

IP6 seems to have died :shrug: I'm pretty sure it was working last night..

I cannot find anything wrong with the server and I haven't touched it for some days.. So I have opened a ticket with a host to see if it's a problem there end first...

[alexh]

I was unable to connect from my house this morning (7am GMT).