I've been trying not to ban all the IPs because of the sheer amount of them.. (*) But I've actually gone back to this approach of sorts.. Because my aggregation script will scan the IP addresses every hour and ban permanently entire subsets anyway.
(*) Annoyingly most IPs only hit the server once a month. But there are 100's of thousands of IPs each day hitting various things which causes a significant load when the forum and wiki are hit. Life was a lot simpler with basic HTML pages.. Immensely lightweight and nobody cares about them any more !
I've also started dropping connections more in nginx. They hit the logs so failban will pick them up. But they don't get as far as hitting PHP and SQL which are the main resource hogs.. So now I think I have solved that problem, the CP was biking because of the traffic fail2ban is now having to process
but I suppose eventually it will settle down when enough ranges have been blocked..
Fail2ban isn't really helping with botnet attacks as they all look like legitimate traffic. Though there has been a additional two scripts written where I won't go into the details, but each server request is given a "score" and if that score gets high enough, then a massive IP range gets nuked automatically. That script has to scan the logs repeatedly which takes a bit of CPU but its less CPU than fail2ban was using with all the now redundant jails anyway.