Server updates

[exxos]

After a kernel update earlier, the server seems to now block all ports unless I set them as open.. I kinda did that anyway, but something changed and all ports were blocked totally. I even got locked out of the SSH !! So I couldn't even log into the server to change anything !

I'm slowly figuring out what ports to open up again. Currently trying to get email back up and running.. what a nightmare !!

IP6 should be working again now..

EDIT..

So AI helped me fix the issues. Seems there was 3 programs trying to restore the firewall on boot and potluck which one won. That's now been taken care of. My restore firewall script has been updated to make sure everything stays as it should after a reboot.

AI also found a bunch of other stuff which wasn't right. My script wasn't setting the chains in the right order which was also causing issues.

Anyway.. Should be all fixed now...

[exxos]

He was under a pretty heavy attack last night. I thought the former is a tiny bit sluggish but thought nothing of it until I checked my emails and I had been getting "CPU overload" alerts for over two hours

So with the help of AI it actually found out how to take a significant chunk of CPU time from processing the jails..

It seemed this was partly my fault because I had IP addresses being looked up on the white list before processing any jails as I thought that would be the most efficient way of doing things. But seems not... It seems all the constant wireless lookups were incredibly expensive!

Anyway, this is a CPU graph or basically before and after the change..

Capture.PNG (32.43 KiB) Viewed 142 times

Last night who was using about 90% CPU during the attacks. So I'm slightly happy for once that CPU now has a easier life in general.. Saved around 25% overall CPU time which is pretty good! So... oops on my part I guess

[exxos]

Had huge problems updating SQL just I had to ask AI because I had no idea as usual how to fix it (SQL broke totally)

Your MySQL update failed because the system had a mixed MySQL packaging state (Oracle-style mysql-common 8.0.43 vs Ubuntu/ESM MySQL 8.0.45), and apt refused to upgrade until mysql-common was downgraded to Ubuntu’s 5.8+ common package, allowing the MySQL 8.0.45 ESM server/client to install cleanly.

So we seem to be up and running again now...

We could have been down for several hours while I looked for solutions to all the issues. AI got me sorted in about 10 minutes ! At one point the installer wanted to know if I wanted to delete the database... That's enough stress for today for me....

[exxos]

So today there have been some weird attacks on the web store. Looks like lots of IPs was trying to run JS in the URL A new firewall rule will take care of all those bad requests.

I also found another firewall rule had a bug and might have banned some legitimate users. I found this out just because of a user unban request. It was part of the flood protection rules set. I think it might have been falsely triggering on people with incredibly fast Internet. I have disabled that rule for now anyway and unbanned all the IPs in that jail just in case.. That flood ban would have timed out automatically after 24hours anyway. So it was only a short temp ban.

A new flood protection rule will also be added as a forum spam safeguard later today.

[exxos]

I've had three more unban requests from users today, apologies folks..

The "malfunctioning jail" which I emptied yesterday, did not seem to remove the IP address fully from the blacklists..

So I just got AI to write a script to search the fail2ban logs for the past month and generate a list of IP addresses for that malfunctioning jail to un-ban directly from the block lists..

So that jail should generally be completely empty now...

I have verified that the unban request IPs were actually deleted out of the blacklists this time..

EDIT:

It also seems apparent that the script I had running which is supposed to un-ban users automatically every hour doesn't seem to be working properly either so I have updated that script with more debug information so I can keep track better than what is actually doing.. And the unbanning *should* actually work correctly this time as well..

I've also laxed off a couple of other rules in the firewall which should help with users coming from search engines better..

Also fail2bans unban logic wasn't working at all.. I actually changed from banning all IPs permanently because the list was getting to large. So I set to unban after X amount of time, depending on violation, but I never wrote the unban rule, as it wasn't needed until recently.. but it broke the user unban script anyway.. oops.. So when fail2ban said it worked, it actually didn't

Now there is a script running scanning the logs for all the past months "unban" lines because they would not have actually been removed from the blacklists...

There's almost 300,000 IPs in that list.. as to how many are forum users, i've no idea. It will likely take a few hours to remove all of them from the blacklists.