Server updates

[exxos]

76 server updates now done. I have also (finally) removed all the old Apache stuff.

[exxos]

Today's updates include :

Preventing a massive registration attack on the wiki. There was over 40,000 registration attempts in just a few hours from multiple IP addresses. Cleantalk would have prevented those attempts. However, these attempts were consuming resources on the server! So another firewall rule has been set up. So if anyone tries multiple times to create a account, they get banned for 30 days. This has reduced a fair bit of CPU time etc.

Images are now (or should be) correctly cached in Nginx. Saves a lot of resources not having to serve content which users don't really need every page refresh.

Logs have been optimised not to log certain file types like images. This is making the log accesses a lot less. It also makes the files a lot smaller so that fail2ban doesn't have to spend as much time reading through them all.

I've increased the size of the SQL cache, so it *should* cache the entire database now in RAM. This will reduce direct SQL accesses to disk and hopefully help when server is having to deal with various attacks. Its always SQL which becomes the bottleneck and maxes out the CPU while dealing with requests. So I hope the database held in RAM will speed up read access to the database.

I also noticed there was actually a SQL IO bottleneck on writes to disk which has also hopefully being now resolved. IO to disk should be about 100x faster now.

Phpbb itself has a cache for rendering content from, So it caches the files where content is served. I have now set up a RAM disk where the forum cache is now held in RAM. Clearing the cache and rebuilding still takes several seconds, but this is way faster than it was operating from disk (which is Nvme IIRC). The bottleneck there is probably reading from the Nvme disk the forum files. But rebuilding the cache doesn't happen as often as data is served from the cache. So I don't plan on caching anything else here.

Nginx should also cache general content now. Though I am not sure this is really working correctly. I keep getting "cache miss" or nothing reported at all. But I have not configured the entire domain. Mostly I was trying to cache other forum content. But what I can tell, caches don't work with content relating to cookies or something. Likely because the content changes a lot and simply can't be cached. I'm not spending anymore time on that method now anyway.

TL:DR
Fighting attacks, and server optimisations. Something is probably broken now..

[EzdineG]

That's a lot of great administration work going on in the background. I CAN imagine all the go-nowhere rabbit holes you've recently gone down

[exxos]

That's a lot of great administration work going on in the background. I CAN imagine all the go-nowhere rabbit holes you've recently gone down

The best one was where I spent ages over the weekend looking to see if I could limit the CPU time for the forum only. Idea being that if the forum is under attack and maxing out a single core, the other 3 cores can still run the rest of the services like my store and the wiki.

So I found something (well ChatGPT did) where I could do just that. This was a mysql function which would do EXACTLY what I wanted. I could limit CPU power to SQL users which would solve a lot of problems. ChatGPT wrote a nice guide on how to set it all up.... I got a little way in the guide only to find out such a feature doesn't actually exist...

[exxos]

I have been looking at the stats for this year.. And it is easy to see at a glance how much traffic it was having to deal with, which was "all" basically "hacking" attempts or "bad bots" !

1.PNG (6.85 KiB) Viewed 111 times

All the new firewall rules I have set up and protection measures are seriously reducing the amount of server resources being used!

Last month php scripts served was 2.38GB...

11.PNG (61.13 KiB) Viewed 111 times

Compared to January which was 40.58GB!

Last December saw 126.16 GB! I've seen a lot higher in the past as well.

22.PNG (61.76 KiB) Viewed 111 times

I've also realised that PHP itself has a built-in cache which was disabled by default
So now this is enabled, and it keeps compiled versions of scripts held in RAM. So these do not have to be compiled continuously from disk, and with them being held in RAM the execution time is a lot faster!

It has only been enabled for about 10 minutes and already the "hits" show how much traffic its dealing with!

33.PNG (28.53 KiB) Viewed 111 times

[sporniket]

great work

I don't react on those posts, but thank you to dedicate time to make this space running.

[exxos]

great work

I don't react on those posts, but thank you to dedicate time to make this space running.

[exxos]

As a side note, which has just come to my attention...

Please don't "hit" the create account page multiple times on the wiki, you will get banned for 12 hours! (was 30 days , but just updated it, I may up this limit again later) Similar, if you are constantly logging in and out, you will also get banned for 12 hours. This is a systemwide IP ban, so you will lose access to everything on exxoshost / exxosforum / wiki servers until the ban time elapses.

Unfortunately I have to ban on multiple hits on these pages to mitigate spam bots trying to register and login via brute force more often than not. I had 40,000 attempts in just a few hours, hence the new rule being created to slow down such attacks.