ClaudeBot server attack.

Latest Atari related news.
User avatar
exxos
Site Admin
Site Admin
Posts: 24063
Joined: Wed Aug 16, 2017 11:19 pm
Location: UK
Contact:

ClaudeBot server attack.

Post by exxos »

I noticed last night the server was running a little slow but did not think anything of it. However this morning it was returning errors..

502.PNG
502.PNG (6.4 KiB) Viewed 2746 times

Upon looking on the Nginx log.

3.PNG
3.PNG (671.83 KiB) Viewed 2746 times

... and later the forum online list..

Capture.PNG
Capture.PNG (152.75 KiB) Viewed 2742 times

It becomes apparent that ClaudeBot from anthropic.com is opening up 100's of requests per second from various IP ranges. It just simply got to the point where the server 4 CPUs maxed out. It seems our old friend amazonaws.com is providing the bandwidth for it all again.

Blocking individual IP addresses would be a nightmare. I would have to block the entire 3.xxx.xxx.xxx range to stop those attacks. Plus as these attacks keep on happening, it is pointless to try and firewall them any more.

So what I have done is limit all bot limits to one request per second. Frankly, if they are requesting more than that then they are likely a bad bot anyway. But there is also a white list for things like googlebot etc, while they do a lot of requests, they are generally only once a second or thereabouts anyway. Of course I don't want to block bots completely because the forum will vanish off Google etc.. Been down that road before.. I have also installed all the latest updates for the server while it was down. At a glance I don't think anything has broken.. But there was 154 updates pending..!

I think while the web is often crawled for account information and such, I suspect this is now AI powered. In that the Internet gets scraped and then AI is now used to look through all the content for account information or personal information or usual stuff ..

I also see this has been affecting other people as well...

https://www.phpbb.com/community/viewtopic.php?t=2652265
https://community.cloudflare.com/t/sugg ... bot/635305
https://www.exxosforum.co.uk/atari/ All my hardware guides - mods - games - STOS
https://www.exxosforum.co.uk/atari/store2/ - All my hardware mods for sale - Please help support by making a purchase.
viewtopic.php?f=17&t=1585 Have you done the Mandatory Fixes ?
Just because a lot of people agree on something, doesn't make it a fact. ~exxos ~
People should find solutions to problems, not find problems with solutions.
User avatar
exxos
Site Admin
Site Admin
Posts: 24063
Joined: Wed Aug 16, 2017 11:19 pm
Location: UK
Contact:

Re: ClaudeBot server attack.

Post by exxos »

So rate-limiting was basically pointless then. There are just too many IPs hitting the server at once, nevermind connections per second from the same IP address :roll:

So basically I have just whitelisted "good bots" and every other bot which isn't in the list simply gets the connection dropped now. Some nice bots may well fall victim in all this, but realistically I cannot really do anything else.
https://www.exxosforum.co.uk/atari/ All my hardware guides - mods - games - STOS
https://www.exxosforum.co.uk/atari/store2/ - All my hardware mods for sale - Please help support by making a purchase.
viewtopic.php?f=17&t=1585 Have you done the Mandatory Fixes ?
Just because a lot of people agree on something, doesn't make it a fact. ~exxos ~
People should find solutions to problems, not find problems with solutions.
User avatar
exxos
Site Admin
Site Admin
Posts: 24063
Joined: Wed Aug 16, 2017 11:19 pm
Location: UK
Contact:

Re: ClaudeBot server attack.

Post by exxos »

Been working on this all day, what a total nightmare :roll: Another 12 hours of my life wasted messing with stuff I shouldn't really need to be messing with :roll:

It seems that there is no way to get Nginx to work with a blacklist AND a whitelist.

Unfortunately white whitelisting bots also seems to then cause problems with Google indexes..

440167072_1572660243310391_183110599679754138_n (1).png
440167072_1572660243310391_183110599679754138_n (1).png (30.88 KiB) Viewed 2671 times

So I just concentrated on the blacklist. I know that works because I blocked all user agents "chrome", so then I got the connection refused using chrome.

Capture.PNG
Capture.PNG (15.18 KiB) Viewed 2671 times

Long story short, it seems that the bots are trying to use forum links from exxoshost not exxosforum. This would mean that the bot would have to have all the links from outdated exxoshost forum links from 2+ years ago. Then it is trying to get those links from exxosforum.. So it cannot be obeying the redirect rules for starters.

In any case, I have set up the block on exxoshost and now the badbot list with ClaudeBot are now correctly been blocked.

Code: Select all

3.15.15.56 - - [25/Apr/2024:21:28:58 +0100] "GET /index.php?sid=f83932fc34520562a3f333d83f7d09d1 HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
EG: index.php is not in the root folder for starters, its in the /forum/index.php and them blocking with 444 error.

Hopefully this will fix the problem.

I may have another go at the white list but if the blacklist does not have to be updated at often than I am probably just going to leave it like that for the time being now.
https://www.exxosforum.co.uk/atari/ All my hardware guides - mods - games - STOS
https://www.exxosforum.co.uk/atari/store2/ - All my hardware mods for sale - Please help support by making a purchase.
viewtopic.php?f=17&t=1585 Have you done the Mandatory Fixes ?
Just because a lot of people agree on something, doesn't make it a fact. ~exxos ~
People should find solutions to problems, not find problems with solutions.
User avatar
exxos
Site Admin
Site Admin
Posts: 24063
Joined: Wed Aug 16, 2017 11:19 pm
Location: UK
Contact:

Re: ClaudeBot server attack.

Post by exxos »

https://www.theguardian.com/technology/ ... ai-startup

So Amazon are funding the AI bots killing people's servers then :roll: explains why it's again, AWS servers causing all the problems. Seems a few people reported hits from 100s of IP addresses. I've clocked 1.1million hits over night from it :roll: :roll:

I may get my CPU monitor script to just shutdown the server on constant high CPU overloads. Always seems to kickoff during the night and no point running the server when it's going to be malfunctioning anyway..
https://www.exxosforum.co.uk/atari/ All my hardware guides - mods - games - STOS
https://www.exxosforum.co.uk/atari/store2/ - All my hardware mods for sale - Please help support by making a purchase.
viewtopic.php?f=17&t=1585 Have you done the Mandatory Fixes ?
Just because a lot of people agree on something, doesn't make it a fact. ~exxos ~
People should find solutions to problems, not find problems with solutions.
User avatar
chronicthehedgehog
Posts: 240
Joined: Sun May 08, 2022 6:11 pm
Location: The Midlands
Contact:

Re: ClaudeBot server attack.

Post by chronicthehedgehog »

Yes block those bots. Bezos has enough dosh already :D
User avatar
mfro
Posts: 123
Joined: Thu Dec 13, 2018 7:32 am

Re: ClaudeBot server attack.

Post by mfro »

And remember: Beethoven wrote his first symphony in C.
User avatar
exxos
Site Admin
Site Admin
Posts: 24063
Joined: Wed Aug 16, 2017 11:19 pm
Location: UK
Contact:

Re: ClaudeBot server attack.

Post by exxos »

chronicthehedgehog wrote: Thu Apr 25, 2024 11:29 pm Yes block those bots. Bezos has enough dosh already :D
Yep. He owes me 12 hours of emergency I.T. work now.
https://www.exxosforum.co.uk/atari/ All my hardware guides - mods - games - STOS
https://www.exxosforum.co.uk/atari/store2/ - All my hardware mods for sale - Please help support by making a purchase.
viewtopic.php?f=17&t=1585 Have you done the Mandatory Fixes ?
Just because a lot of people agree on something, doesn't make it a fact. ~exxos ~
People should find solutions to problems, not find problems with solutions.
User avatar
exxos
Site Admin
Site Admin
Posts: 24063
Joined: Wed Aug 16, 2017 11:19 pm
Location: UK
Contact:

Re: ClaudeBot server attack.

Post by exxos »

mfro wrote: Fri Apr 26, 2024 5:53 am did you try this:

https://developer.amazon.com/amazonbot

??
Amazonbot it isn't the problem ClaudeBot is. But others saying it's totally ignoring robots.txt anyway.
https://www.exxosforum.co.uk/atari/ All my hardware guides - mods - games - STOS
https://www.exxosforum.co.uk/atari/store2/ - All my hardware mods for sale - Please help support by making a purchase.
viewtopic.php?f=17&t=1585 Have you done the Mandatory Fixes ?
Just because a lot of people agree on something, doesn't make it a fact. ~exxos ~
People should find solutions to problems, not find problems with solutions.
User avatar
exxos
Site Admin
Site Admin
Posts: 24063
Joined: Wed Aug 16, 2017 11:19 pm
Location: UK
Contact:

Re: ClaudeBot server attack.

Post by exxos »

Looking at my stats there was a total of 904 different amazon IP addresses hammering my server all at once. Each IP had about 2,000 hits each. There was no delay in how fast the requests were being made either.

Capture.PNG
Capture.PNG (35.09 KiB) Viewed 2567 times

All files on my website had multiple hits.



As a side note, Operating Systems used last month..

1.PNG
1.PNG (104.39 KiB) Viewed 2567 times
2.PNG
2.PNG (29.7 KiB) Viewed 2567 times
https://www.exxosforum.co.uk/atari/ All my hardware guides - mods - games - STOS
https://www.exxosforum.co.uk/atari/store2/ - All my hardware mods for sale - Please help support by making a purchase.
viewtopic.php?f=17&t=1585 Have you done the Mandatory Fixes ?
Just because a lot of people agree on something, doesn't make it a fact. ~exxos ~
People should find solutions to problems, not find problems with solutions.
User avatar
Cyprian
Posts: 409
Joined: Fri Dec 22, 2017 9:16 am
Location: Poland

Re: ClaudeBot server attack.

Post by Cyprian »

What a nightmare,
What if you blocking all Amazon servers?
Lynx I / Mega ST 1 / 7800 / Portfolio / Lynx II / Jaguar / TT030 / Mega STe / 800 XL / 1040 STe / Falcon030 / 65 XE / 520 STm / SM124 / SC1435
DDD HDD / AT Speed C16 / TF536 / SDrive / PAK68/3 / Lynx Multi Card / LDW Super 2000 / XCA12 / SkunkBoard / CosmosEx / SatanDisk / UltraSatan / USB Floppy Drive Emulator / Eiffel / SIO2PC / Crazy Dots / PAM Net
http://260ste.atari.org
Post Reply

Return to “NEWS & ANNOUNCEMENTS”