ClaudeBot server attack.

[exxos]

I noticed last night the server was running a little slow but did not think anything of it. However this morning it was returning errors..

502.PNG

Upon looking on the Nginx log.

3.PNG

... and later the forum online list..

Capture.PNG

It becomes apparent that ClaudeBot from anthropic.com is opening up 100's of requests per second from various IP ranges. It just simply got to the point where the server 4 CPUs maxed out. It seems our old friend amazonaws.com is providing the bandwidth for it all again.

Blocking individual IP addresses would be a nightmare. I would have to block the entire 3.xxx.xxx.xxx range to stop those attacks. Plus as these attacks keep on happening, it is pointless to try and firewall them any more.

So what I have done is limit all bot limits to one request per second. Frankly, if they are requesting more than that then they are likely a bad bot anyway. But there is also a white list for things like googlebot etc, while they do a lot of requests, they are generally only once a second or thereabouts anyway. Of course I don't want to block bots completely because the forum will vanish off Google etc.. Been down that road before.. I have also installed all the latest updates for the server while it was down. At a glance I don't think anything has broken.. But there was 154 updates pending..!

I think while the web is often crawled for account information and such, I suspect this is now AI powered. In that the Internet gets scraped and then AI is now used to look through all the content for account information or personal information or usual stuff ..

I also see this has been affecting other people as well...

https://www.phpbb.com/community/viewtopic.php?t=2652265
https://community.cloudflare.com/t/sugg ... bot/635305

[exxos]

So rate-limiting was basically pointless then. There are just too many IPs hitting the server at once, nevermind connections per second from the same IP address :roll:

So basically I have just whitelisted "good bots" and every other bot which isn't in the list simply gets the connection dropped now. Some nice bots may well fall victim in all this, but realistically I cannot really do anything else.

[exxos]

Been working on this all day, what a total nightmare :roll: Another 12 hours of my life wasted messing with stuff I shouldn't really need to be messing with :roll:

It seems that there is no way to get Nginx to work with a blacklist AND a whitelist.

Unfortunately white whitelisting bots also seems to then cause problems with Google indexes..

440167072_1572660243310391_183110599679754138_n (1).png

So I just concentrated on the blacklist. I know that works because I blocked all user agents "chrome", so then I got the connection refused using chrome.

Capture.PNG

Long story short, it seems that the bots are trying to use forum links from exxoshost not exxosforum. This would mean that the bot would have to have all the links from outdated exxoshost forum links from 2+ years ago. Then it is trying to get those links from exxosforum.. So it cannot be obeying the redirect rules for starters.

In any case, I have set up the block on exxoshost and now the badbot list with ClaudeBot are now correctly been blocked.

Code: Select all

3.15.15.56 - - [25/Apr/2024:21:28:58 +0100] "GET /index.php?sid=f83932fc34520562a3f333d83f7d09d1 HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"

EG: index.php is not in the root folder for starters, its in the /forum/index.php and them blocking with 444 error.

Hopefully this will fix the problem.

I may have another go at the white list but if the blacklist does not have to be updated at often than I am probably just going to leave it like that for the time being now.

[exxos]

https://www.theguardian.com/technology/ ... ai-startup

So Amazon are funding the AI bots killing people's servers then :roll: explains why it's again, AWS servers causing all the problems. Seems a few people reported hits from 100s of IP addresses. I've clocked 1.1million hits over night from it :roll: :roll:

I may get my CPU monitor script to just shutdown the server on constant high CPU overloads. Always seems to kickoff during the night and no point running the server when it's going to be malfunctioning anyway..

[chronicthehedgehog]

Yes block those bots. Bezos has enough dosh already :D

[mfro]

did you try this:

https://developer.amazon.com/amazonbot

??

[exxos]

Yes block those bots. Bezos has enough dosh already :D

Yep. He owes me 12 hours of emergency I.T. work now.

[exxos]

did you try this:

https://developer.amazon.com/amazonbot

??

Amazonbot it isn't the problem ClaudeBot is. But others saying it's totally ignoring robots.txt anyway.

[exxos]

Looking at my stats there was a total of 904 different amazon IP addresses hammering my server all at once. Each IP had about 2,000 hits each. There was no delay in how fast the requests were being made either.

Capture.PNG

All files on my website had multiple hits.



As a side note, Operating Systems used last month..

1.PNG

2.PNG

[Cyprian]

What a nightmare,
What if you blocking all Amazon servers?