Server updates

[exxos]

76 server updates now done. I have also (finally) removed all the old Apache stuff.

[exxos]

Today's updates include :

Preventing a massive registration attack on the wiki. There was over 40,000 registration attempts in just a few hours from multiple IP addresses. Cleantalk would have prevented those attempts. However, these attempts were consuming resources on the server! So another firewall rule has been set up. So if anyone tries multiple times to create a account, they get banned for 30 days. This has reduced a fair bit of CPU time etc.

Images are now (or should be) correctly cached in Nginx. Saves a lot of resources not having to serve content which users don't really need every page refresh.

Logs have been optimised not to log certain file types like images. This is making the log accesses a lot less. It also makes the files a lot smaller so that fail2ban doesn't have to spend as much time reading through them all.

I've increased the size of the SQL cache, so it *should* cache the entire database now in RAM. This will reduce direct SQL accesses to disk and hopefully help when server is having to deal with various attacks. Its always SQL which becomes the bottleneck and maxes out the CPU while dealing with requests. So I hope the database held in RAM will speed up read access to the database.

I also noticed there was actually a SQL IO bottleneck on writes to disk which has also hopefully being now resolved. IO to disk should be about 100x faster now.

Phpbb itself has a cache for rendering content from, So it caches the files where content is served. I have now set up a RAM disk where the forum cache is now held in RAM. Clearing the cache and rebuilding still takes several seconds, but this is way faster than it was operating from disk (which is Nvme IIRC). The bottleneck there is probably reading from the Nvme disk the forum files. But rebuilding the cache doesn't happen as often as data is served from the cache. So I don't plan on caching anything else here.

Nginx should also cache general content now. Though I am not sure this is really working correctly. I keep getting "cache miss" or nothing reported at all. But I have not configured the entire domain. Mostly I was trying to cache other forum content. But what I can tell, caches don't work with content relating to cookies or something. Likely because the content changes a lot and simply can't be cached. I'm not spending anymore time on that method now anyway.

TL:DR
Fighting attacks, and server optimisations. Something is probably broken now..

[EzdineG]

That's a lot of great administration work going on in the background. I CAN imagine all the go-nowhere rabbit holes you've recently gone down

[exxos]

That's a lot of great administration work going on in the background. I CAN imagine all the go-nowhere rabbit holes you've recently gone down

The best one was where I spent ages over the weekend looking to see if I could limit the CPU time for the forum only. Idea being that if the forum is under attack and maxing out a single core, the other 3 cores can still run the rest of the services like my store and the wiki.

So I found something (well ChatGPT did) where I could do just that. This was a mysql function which would do EXACTLY what I wanted. I could limit CPU power to SQL users which would solve a lot of problems. ChatGPT wrote a nice guide on how to set it all up.... I got a little way in the guide only to find out such a feature doesn't actually exist...

[exxos]

I have been looking at the stats for this year.. And it is easy to see at a glance how much traffic it was having to deal with, which was "all" basically "hacking" attempts or "bad bots" !

1.PNG (6.85 KiB) Viewed 709 times

All the new firewall rules I have set up and protection measures are seriously reducing the amount of server resources being used!

Last month php scripts served was 2.38GB...

11.PNG (61.13 KiB) Viewed 709 times

Compared to January which was 40.58GB!

Last December saw 126.16 GB! I've seen a lot higher in the past as well.

22.PNG (61.76 KiB) Viewed 709 times

I've also realised that PHP itself has a built-in cache which was disabled by default
So now this is enabled, and it keeps compiled versions of scripts held in RAM. So these do not have to be compiled continuously from disk, and with them being held in RAM the execution time is a lot faster!

It has only been enabled for about 10 minutes and already the "hits" show how much traffic its dealing with!

33.PNG (28.53 KiB) Viewed 709 times

[sporniket]

great work

I don't react on those posts, but thank you to dedicate time to make this space running.

[exxos]

great work

I don't react on those posts, but thank you to dedicate time to make this space running.

[exxos]

As a side note, which has just come to my attention...

Please don't "hit" the create account page multiple times on the wiki, you will get banned for 12 hours! (was 30 days , but just updated it, I may up this limit again later) Similar, if you are constantly logging in and out, you will also get banned for 12 hours. This is a systemwide IP ban, so you will lose access to everything on exxoshost / exxosforum / wiki servers until the ban time elapses.

Unfortunately I have to ban on multiple hits on these pages to mitigate spam bots trying to register and login via brute force more often than not. I had 40,000 attempts in just a few hours, hence the new rule being created to slow down such attacks.

[exxos]

People may notice more bots listed in the online users list.. The bots have always been there in the background.. Though as mentioned previously because they are not cookie friendly or not logged in users, they end up with "sid" in the URLs.. The problem with this is a lot of bots end up stuck in a loop because every time they visit the exact same page of the "sid" changes and they index the change, which causes a different "sid" again.. With the vast amount of bots out there this was starting to become a bit of a problem with all the traffic. So more common bots are now classed as proper users so they don't end up with sid's all over the place.

I also optimise multiple firewall rules and stopped bots from indexing "useless pages" such as the posting page, signup page etc etc I think not having these listed on search engines or indexed by bots can only be a good thing.

I also noticed a couple of pages "crashing" on the wiki so I have fix those as well.

Time for a rest now me thinks.....

[exxos]

Got a bit distracted these past few days because the version of fail2ban I had, had a "bug". I opened up a ticket on their own github page and long story short, the very next revision of F2B fixed that problem.. And guess what version in the ubuntu repo stopped updating to.. the version I had installed Basically my version was four years out of date . I had not realised they had stopped updating it in the official repository

So another long story short, I had to set up a test server and manually install the latest version, install loads of crap to simulate exxoshost etc. Because it was a rather large version jump, I wanted to make sure nothing broke before I did the updates on the actual live server. Another long story short, we are now running the latest version of fail2ban now. The bug I was seeing is now gone.

Have also been doing various tweaks to the nginx config and doing other tweaks on the server. Far too boring to list them all. Though I don't think anything seems broken at this point..

I also did some fixes to the store javascript and php code.. Mostly relating to the postage calculation.. So if anyone cannot seem to get a calculation to show up or cannot check out some reason, then you probably need to clear your browser cache as the old JS won't work with the updated php scripts.