Server updates

[exxos]

I found some bugs in the cleantalk extension where the spam firewall was churning out huge lists of errors in my logs. This has been ongoing in the background for about two weeks now . Long story short, I ended up fixing the extension myself and those updates will make it into their future releases !

ct.PNG (13.39 KiB) Viewed 1124 times

So the new spam firewall has only been enabled for like two days ...

Capture.PNG (7.11 KiB) Viewed 1124 times

Hopefully this will clamp down on the spam posts which have been slipping through for a while now @Global moderators . IPs are outright blocked from even getting to the signup pages now.

Some will always slip through, it is going to be several thousand people less trying to register on the forum every week now. Mostly we was relying on real-time checking & questions and other stuff to make it difficult for bots to sign up. But I think legitimate humans are very determined and getting through those defences. So something is more "up front" now in blocking IPs. So at least the traffic hitting the signup page will be a few orders of magnitude less now!

I also have a new script running which detects multiple signup attempts in the nginx log. Like if someone is hitting the register page 50 times over then it's pretty safe to assume that is not a legitimate user! So such things get blocked there as well now.

I also asked about HTTP2 on Google community..

https://support.google.com/webmasters/thread/366659054/

Google only seem to crawl on HTTP2 on sites "which matter" so it seems to me. This doesn't work when I know communities are outright blocking HTTP1 now to deal with all the attacks.

Some communities are starting to close public access to a forums and others sre simply "dropping" bots because most use HTTP1 still. Which then results in communities vanishing of search engines !!

If Google would just allow HTTP2 even by a switch in robots.txt , it would solve a lot of problems.. It has been talked about in other threads across the Internet as I posted before for the past decade already. But it doesn't seem like they really care.

[exxos]

Today I did a deep dive into why some banned IPs were showing up as “already banned” in Fail2Ban but not actually appearing in the blacklist sets. After some investigation we found a couple of issues:

The Fail2Ban action script was pointing to the wrong binary paths for ipset, iptables, and conntrack. Because of that, bans were logged as successful but the IPs weren’t always being added to the kernel sets.

Fail2Ban itself keeps an internal memory of what it thinks is already banned. So if the actual ipset operation failed (due to the wrong path), Fail2Ban still “believed” the IP was banned even though it wasn’t.

Restarting Fail2Ban cleared out its internal cache and, with the fixed paths, the bans started applying correctly again.

I also confirmed that all traffic filtering is happening at the PREROUTING stage in iptables, so blocked IPs get dropped before they can even reach services like nginx.

Lastly, I set up a live check to automatically verify that any IP Fail2Ban bans really does get added to the blacklist. This confirmed everything is now working as expected.

TL;DR;
Bans are now landing in the firewall sets correctly, Fail2Ban’s state matches reality, and the system is reliably blocking all hostile traffic again.

[exxos]

I had to make some changes to the nginx config.. Please let me know if something seems broken now..

Mostly its how URL redirects are handled...

[exxos]

Just found out that my manual blocklists was not working properly. Those scripts check for "cluster attacks" basically.


EDIT:
This scripts are actually functioning fine, but the IP addresses was not been banned. From what the AI tells me, F2B needed a "polling" option set as it doesn't see IP lists when they simply change, only when added to.

So basically if there was 100 ips in the list, they would get banned on first run, then the file would get replaced by say 10 ips, and they wouldn't get banned.

There was basically a backlog of around 12,000 IPs.

[rubber_jonnie]

Just found out that my manual blocklists was not working properly. Those scripts check for "cluster attacks" basically.

[JezC]

Not quite sure what is going on...but when I just logged in, it was first showing dml instead of my forum name & avatar, then ijor.

Both times, the actual info under seemed to be for my account but I was seeing a mix of unread posts which I'd already read earlier today.

Will check it again a bit later, just though it worth mentioning...

EDIT : And right after posting the above, I was PhilC for a brief (any one might say worrying ) moment...

[exxos]

Not quite sure what is going on...but when I just logged in, it was first showing dml instead of my forum name & avatar, then ijor.

Both times, the actual info under seemed to be for my account but I was seeing a mix of unread posts which I'd already read earlier today.

Will check it again a bit later, just though it worth mentioning...

Thanks.. Yeah I been messing with the cache, but its not behaving, so just took out the changes... So you shouldn't see any problems again hopefully..

I will clear the cache on the forum now as well. ..

[exxos]

canonical finally got back to me about the server crashes when I was running canonical-livepatch. Basically they can't replicate and want me to try again..

So its likely the server may go down again randomly as it was crashing nginx last time..

@Global moderators @Administrators

[exxos]

canonical got back to me.. Basically they said there's some bug in nginx perl module causing the crash.

Annoyingly, when I looked into the error months back, I disabled the Perl module from loading but it didn't solve the issue . So now I've totally uninstalled Perl and so far so good.... But I don't think there's been any kernel updates yet anyway..

Buy anyway, live patch is enabled again.. So see how things go..

We still doing fine in relation to attacks. Traffic seems to triple over the weekend.. Then dies back a bit during the week. Seem to get around 5,000 IP per hour during the week and around 20,000 weekends.

At some point I need to reduce the jumper of jails i have and consolidate some rules as it takes a chunk out of CPU time processing all the jails. But it may not be worth the trouble. The web exploits jail is massive and takes up the most CPU time.. But with improvements in other jails, it's not really doing much as it once was. So I guess it could just be disabled if it becomes an issue.

Mostly the server CPU sits at around 5-10% now. It spikes up to about 50% because of faile2ban checking. But bad traffic generally dies out after a couple of hours now. Probably helps that after several IP hits on similar ranges that I ban the range now. So rather than waiting for other ips to hit the server, they are already banned by ranges.

Reported ips is fast heading to 6 million now. We was at 3million a couple weeks ago. I suppose that's nothing really these days. But sure can take its toll on small servers if they are not banning them somehow.

[exxos]

Cloudflare successfully defended against a record-breaking 11.5 terabits per second DDoS attack that lasted just 35 seconds. The massive UDP flood originated primarily from compromised resources on Google Cloud Platform and set a new industry high for network bandwidth consumed by malicious traffic.

https://aardwolfsecurity.com/cloudflare ... os-attack/

I noticed a load of traffic coming from google cloud this past week also. I was adjusting my scripts because I had cloud IPs whitelisted (mostly by mistake) . Those are blocked automatically now anyway.. Now if only cloudflare could block their own IPs from serving bad traffic! I mean who do cloudflare use for blocking traffic coming from their own servers ?