Server updates

[exxos]

It's very impressive how you've become such an internet server security expert in the last year or so, I hope one day you'll be able to sit back and relax without having to worry about the server all the time though.

I've learned a hell of a lot that's for sure. Some keep bugging me to use a proxy like Cloudflare which has caused me a lot of trouble by itself. I've seen stuff which would give most people nightmares some people on forums said cloudflare did nothing unless it was set to under attack mode. Probably just limits all requests.

Its still odd that nobody on the abuse site has reported the brazil attacks yet. I can only assume I'm the first one to adapt rules to catch them. There's articles on it like I posted before.. So it's a known thing..

I think the whole thing is a ticking timebomb though I can see the day coming where we will end up not being a public forum anymore... Like startdot is currently and the stos coders site. It's like the dead internet theory..

Currently all my detection rules are working well. Problem lately has been the overload of banning and reporting IPS.. That should be fixed now. New rules to detect the brazil attacks better...

We been hit with half a million IPs over the past few days. That's a whole year of "last years abuse" in a week. That should put it into perspective how bad attacks have become lately..

I've optimized nginx as well to not even send error messages on ips which get banned.. Saved a fair bit of CPU power there alone.. Its a good job I went with a 4 core server last move. No way 2 cores could keep up now.

[Steve]

If you took all of you knowledge, scripts, ban rules and configs - you could potentially go around private forums as a kind of contractor security expert

[exxos]

If you took all of you knowledge, scripts, ban rules and configs - you could potentially go around private forums as a kind of contractor security expert

Yeah. I did wonder something similar. It would be good if we had "small server unite" thing where they run my scripts and we all report IPS to a database similar to abuseipdb. Problem at the moment is I'm reporting IPS that nobody else is. So the confidence score is zero and it doesn't really help anyone else.. But if only I had the time and didn't have RSI.. My hands are killing me just fixing up my server.

But people are likely not going to give me root access to forums. Wouldn't be my first choice either . But trying to engage with people even to help them, always seems to be a waste of time anyway. I could say their servers under attack from brazil and if they think otherwise... Same problem as always. Nobody walks in my shoes a so nobody will understand whats going on or efforts or cures I've done.

Like when the German forum was up for the axe. I offered to host it.. They went with someone else, fair enough, but I didn't even get a thanks but no thanks message. I was left hanging until I saw announcement they hosting with someone else.. And ts just the theme with a lot of people.. I try and help and generally wish I just hadn't of bothered.

[exxos]

Done an overhaul of the IP BAN CHECK page. Forum users will be on band automatically from the firewall directly and fail2ban.. Ordinarily fail2ban unban would be enough... BUT..

When F2B we stars like on the server reboot or I changed the jails et cetera, it then gets stuck in a "restore ban loop" where it will not even give me the status of the client until its finished "restoring bans" . It is basically hangs indefinitely until its completed what it's doing.. Which can literally take several hours or longer now.

The problem seems to stem from F2B keeping an internal record of what was banned so when the server is rebooted, it automatically lads all the banned IP is back into IPtables etc.. Unfortunately when you have literally millions of IP addresses that becomes the total hindrance and is just not realistic to operate that way.

However you can set f2B to store a lesser sized database of a couple hours rather than 24hours as its currently set by default. So the fail2ban.sqlite3 database is now 14MB and not 416MB. So while the restore ban will still work up to 4 hours everything else is simply ignored now..

The firewall will automatically save and restore itself directly via a Demon script thingy. So there is no need for F2B to restore bans at all any more... But I left it as a couple of hours just in case anyway..

This also brings me back to my IP ban check script because that simply could not run and would hang for literally hours because it could not access the F2B jails.

We now have about 2.5 million IP addresses banned over the past week or so... I'm slowly working on reporting them to the abuse database much more efficiently.. So the list is rapidly going down now..

I have also optimised some php stuff. the PHP cache wasn't enabled when I upgraded PHP.. So thats now fixed. Normally PHP scripts are compiled in real-time as they are used which takes up CPU time. So they are all cached for 10 seconds now. Normally this is done for like a power or something, but because my scripts change so frequently, really don't want to wait longer than 10 seconds every time I change a script it should help reduce the server load a fair bit.

I also enabled a swap file, it is set to only be used when the server is critically low on RAM.

I also found nginx was struggling a fair bit for connections. So I have upped the limit on that as well.

I also realise that the UFW wasn't running.. I think I disabled that a while ago because it was causing no end of problems .. But I cannot remember what those problems were any more ... but in any case I got the AI to write a script to reset it all for me.. So will see how things go..

There's also been several other small fixes and updates here and there which I probably forgotten about already.

I got some other stuff I need to change as well yet, but I'm going to have to keep off the forum and the PC for the next few days because my hands are killing me again now

[exxos]

Over the past few days I've been optimising and tidying up a lot of the firewall related stuff again. I found better ways of doing some things and so we are a lot more efficient again now dealing with bad traffic. Too many things to list.....

Also noticed some attacks have been managing to bypass all my filter rules by doing something incredibly dumb in fact so I have fixed all that as well. Also tidied up several script so it doesn't output is much debugging information as its not really needed now.

Please note that in order for your IP to be whitelisted, you should be logged into the forum and leave the browser open on the index page for a few hours. So the script which runs about once an hour can pick up your current IP address and add it to the white list. Same script will also automatically unban any users which may have accidentally ended up on the blacklist somehow.

[Steve]

Server seems to be very responsive and fast

[exxos]

Server seems to be very responsive and fast

Hopefully will stop that way !

[exxos]

I asked ram node about all the attacks and they replied...

Hi Chris,

I understand you're experiencing bot attacks, with thousands of requests to dynamic URLs like /?filter=... or /?s=..., or add-to-cart primarily from Brazilian IPs (but also Singapore and Hong Kong)?

This is a known huge issue affecting many hosting companies all over the world and as far I know, I'm afraid there's still no definite solution.
Neither Cloudflare, nor Imunify360 WAF block these. We also tried Monarx, but they couldn't block those attacks either.

Our DDoS Filtering would definitely not help, as this is targeted towards protection from layer 4 volumetric attacks. Layer 7 attacks like these require a WAF such as Cloudflare.

On my servers, as a temporary measure I have completely blocked Brazil, Singapore and Hong Kong IPs via the Country block feature in Imunify360 as I don't expect any legit traffic from there. I understand however that this is not the case for everyone.

Certainly not simple to keep on top of.. I mean block China, Russia, Brazil etc , then your onto ones like Egypt, Pakistan, and loads of places I have never even heard of. I've pretty much end up having to block everywhere except UK and USA.. But I get hit a lot from USA anyway..

attacks.PNG (84.89 KiB) Viewed 1115 times

It all seems to be a pretty big mixed bag. I mean basically they are all attacking the server in similar ways, so they are relatively "easy" to block. The problem becomes with the sheer amount of traffic.. Enough to cripple small servers but not enough to get on cloudflares radar..

But anyway, I am constantly improving our defences against all this. Unfortunately it is soaking up a phenomenal amount of my free time at the moment, so I'm I haven't had much chance to do any Atari work etc lately.. But hopefully the last lot of updates will hold up and I can forget about it for a bit now...

[exxos]

Was just sent this page..

https://blog.cloudflare.com/perplexity- ... irectives/

We are observing stealth crawling behavior from Perplexity, an AI-powered answer engine. Although Perplexity initially crawls from their declared user agent, when they are presented with a network block, they appear to obscure their crawling identity in an attempt to circumvent the website’s preferences. We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring — or sometimes failing to even fetch — robots.txt files.

The Internet as we have known it for the past three decades is rapidly changing, but one thing remains constant: it is built on trust. There are clear preferences that crawlers should be transparent, serve a clear purpose, perform a specific activity, and, most importantly, follow website directives and preferences. Based on Perplexity’s observed behavior, which is incompatible with those preferences, we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling.

[exxos]

We are seeing a rise in spammers signing up lately. Mostly Cleantalk / signup questions / @rubber_jonnie catch them. But some are still managing to get to at least posting on the forum (but are catched by moderation before they go live).

We have a lot of signup questions which could now be "known". Mostly I think it is human behaviour which is manually signing up.. But as people have do a "bit of research" to get past signup questions, I really wonder why people go to all the effort in the first place.

Clean talk has a antispam firewall which I'm going to enable. Not sure this will have any impact on manual signup it is worth a try...

https://cleantalk.org/help/cleantalk-spam-firewall

CleanTalk SpamFireWall to Block Spam Traffic on Your Websites


The CleanTalk SpamFireWall manages and filters all inbound HTTP traffic to protect websites from spam bots and to reduce the load on the web servers. SpamFireWall is an additional and free option in the CleanTalk anti-spam plugins.

This solution to minimize in-house development and maintenance as much as possible while reducing the load on servers. You can be up and run within 60 seconds. Blocks of Spam Bots before it reaches your web server.


How It Works
Unlike other anti-spam solutions, CleanTalk SpamFireWall is able to reduce the load placed on your existing web server. The reporting allows administrators to control the process and to be sure of the accuracy of the work.


Our servers use the cloud-based technology of all websites to fight spam.
The visitor enters to your website.
HTTP request data is checked for the nearly 2 million of certain IP spambots
If it is an active spam bot, it will get a blank page, if it is a visitor then it proceeds to the site. That is completely transparent to the visitors.
Parameters are written to the log which can be viewed in the CleanTalk Dashboard.