Page 21 of 47

Re: Server updates

Posted: 23 Oct 2024 06:42
by russellnash
I don't know anything about Ubuntu server but if LTS is in the name it usually means you are using some old versions of various things in order to make it what they call stable.

Re: Server updates

Posted: 23 Oct 2024 09:41
by exxos
russellnash wrote: 23 Oct 2024 06:42 I don't know anything about Ubuntu server but if LTS is in the name it usually means you are using some old versions of various things in order to make it what they call stable.
Yep true the LTS always seems to lag a fair bit behind. Though the F2B repo is abandoned now :( I had to figure out how to compile another program from source a couple weeks back as another repo got abandoned :(

Re: Server updates

Posted: 20 Nov 2024 09:22
by exxos
@rubber_jonnie mentioned to me yesterday that images in particular were loading very slow or not at all.

It took some hours to diagnose the problem. Long story short, After opening a support ticket, Ramnode moved my server to another node on the host and things seem to be running fast again now.

I suspect there was some bad sectors on the SSD as I tested the drive speed before and after the move and afterwards the speed was like 10x faster. This also seems to explain the unusual high CPU usage lately also.

Re: Server updates

Posted: 20 Nov 2024 09:29
by rubber_jonnie
exxos wrote: 20 Nov 2024 09:22 @rubber_jonnie mentioned to me yesterday that images in particular were loading very slow or not at all.

It took some hours to diagnose the problem. Long story short, After opening a support ticket, Ramnode moved my server to another node on the host and things seem to be running fast again now.

I suspect there was some bad sectors on the SSD as I tested the drive speed before and after the move and afterwards the speed was like 10x faster. This also seems to explain the unusual high CPU usage lately also.
Glad I could help 😁

Re: Server updates

Posted: 20 Nov 2024 09:41
by HigashiJun
Yay ! The store is also back to normal.

:D

Re: Server updates

Posted: 20 Nov 2024 10:08
by exxos
HigashiJun wrote: 20 Nov 2024 09:41 Yay ! The store is also back to normal.
Was something wrong before ? I wasn't aware..

Re: Server updates

Posted: 20 Nov 2024 10:29
by HigashiJun
exxos wrote: 20 Nov 2024 10:08 Was something wrong before ? I wasn't aware..
Pictures were not loading and everything was slow.

Had issues too with basket refreshing.

I finally placed my order today.

Re: Server updates

Posted: 26 Jan 2025 12:02
by exxos
The server may intermittently go down for about half an hour over the next few weeks.

The reason for this is I am converting to full fibre and I will able to download and store full disk server backups on my home network. This should help too slightly reduce monthly operational costs as I will not have to store every disc back up in the cloud which of course costs money per month.

It should allow me to experiment with server updates for the forum on my home network as opposed to trying to do everything on the live server. It should ultimately make my life easier as updating the forum software always leads to something breaking and takes time to resolve. Doing this work on a live server puts pressure on me to fix it all quickly which I want to avoid in the future. Generally I have used multiple other servers for testing but that again costs more money.

Also the server was dealing with several thousand attacks from China and Singapore yesterday. I have not seen or heard of any server timeouts or slowdowns over the past couple of days. so my firewall rules are still holding up just fine. Fail2ban takes care of a lot of things but I had to write custom scripts because of the way the requests are made in order to avoid detection. Things like fake Google bots are constant nuisance and have to be scanned and IP verified individually. Things are done hourly among multiple other detection systems I have put in place.

Incidentally IPs are being reported to the abuse database regularly https://www.abuseipdb.com/user/170367 . It has a limit of 5, 000 reports per day which is how I knew about the attacks because I got to an email saying I had hit the limit for that day. The reporting is actually queued as to try and avoid hitting the reporting limits, but the attacks yesterday must have been that bad even the queueing wasn't enough. Things seem again "quiet" today.

Incidentally some real-time server updates using conical live patch system causes nginx to crash. I have opened up a bug report and support ticket but progress is incredibly slow / next to nonexistent. Generally @rubber_jonnie is on the ball in telling me when the server isn't responding. As long as I'm at home , I can restart nginx quickly. But it can be an issue if I am not at home, IE on weekends. In which case the server may be down until I get home Sunday nights.

Re: Server updates

Posted: 26 Jan 2025 12:08
by rubber_jonnie
exxos wrote: 26 Jan 2025 12:02 The server may intermittently go down for about half an hour over the next few weeks.

The reason for this is I am converting to full fibre and I will able to download and store full disk server backups on my home network. This should help too slightly reduce monthly operational costs as I will not have to store every disc back up in the cloud which of course costs money per month.

It should allow me to experiment with server updates for the forum on my home network as opposed to trying to do everything on the live server. It should ultimately make my life easier as updating the forum software always leads to something breaking and takes time to resolve. Doing this work on a live server puts pressure on me to fix it all quickly which I want to avoid in the future. Generally I have used multiple other servers for testing but that again costs more money.

Also the server was dealing with several thousand attacks from China and Singapore yesterday. I have not seen or heard of any server timeouts or slowdowns over the past couple of days. so my firewall rules are still holding up just fine. Fail2ban takes care of a lot of things but I had to write custom scripts because of the way the requests are made in order to avoid detection. Things like fake Google bots are constant nuisance and have to be scanned and IP verified individually. Things are done hourly among multiple other detection systems I have put in place.

Incidentally IPs are being reported to the abuse database regularly https://www.abuseipdb.com/user/170367 . It has a limit of 5, 000 reports per day which is how I knew about the attacks because I got to an email saying I had hit the limit for that day. The reporting is actually queued as to try and avoid hitting the reporting limits, but the attacks yesterday must have been that bad even the queueing wasn't enough. Things seem again "quiet" today.

Incidentally some real-time server updates using conical live patch system causes nginx to crash. I have opened up a bug report and support ticket but progress is incredibly slow / next to nonexistent. Generally @rubber_jonnie is on the ball in telling me when the server isn't responding. As long as I'm at home , I can restart nginx quickly. But it can be an issue if I am not at home, IE on weekends. In which case the server may be down until I get home Sunday nights.
I'll always try to let you know if there is an issue, no problems there.

Sounds like a plan to reduce costs, always a win there and being able to have a dev/uat environment is always a bonus. Nobody wants to break prod...

Re: Server updates

Posted: 10 Mar 2025 16:40
by exxos
I got Grok to optimise one of my fail2ban jails as its huge.. So if everyone suddenly gets banned.. thats why.. I will try and keep an eye on it all...