Server updates

[russellnash]

I don't know anything about Ubuntu server but if LTS is in the name it usually means you are using some old versions of various things in order to make it what they call stable.

[exxos]

I don't know anything about Ubuntu server but if LTS is in the name it usually means you are using some old versions of various things in order to make it what they call stable.

Yep true the LTS always seems to lag a fair bit behind. Though the F2B repo is abandoned now I had to figure out how to compile another program from source a couple weeks back as another repo got abandoned

[exxos]

@rubber_jonnie mentioned to me yesterday that images in particular were loading very slow or not at all.

It took some hours to diagnose the problem. Long story short, After opening a support ticket, Ramnode moved my server to another node on the host and things seem to be running fast again now.

I suspect there was some bad sectors on the SSD as I tested the drive speed before and after the move and afterwards the speed was like 10x faster. This also seems to explain the unusual high CPU usage lately also.

[rubber_jonnie]

@rubber_jonnie mentioned to me yesterday that images in particular were loading very slow or not at all.

It took some hours to diagnose the problem. Long story short, After opening a support ticket, Ramnode moved my server to another node on the host and things seem to be running fast again now.

I suspect there was some bad sectors on the SSD as I tested the drive speed before and after the move and afterwards the speed was like 10x faster. This also seems to explain the unusual high CPU usage lately also.

Glad I could help

[HigashiJun]

Yay ! The store is also back to normal.

[exxos]

Yay ! The store is also back to normal.

Was something wrong before ? I wasn't aware..

[HigashiJun]

Was something wrong before ? I wasn't aware..

Pictures were not loading and everything was slow.

Had issues too with basket refreshing.

I finally placed my order today.

[exxos]

The server may intermittently go down for about half an hour over the next few weeks.

The reason for this is I am converting to full fibre and I will able to download and store full disk server backups on my home network. This should help too slightly reduce monthly operational costs as I will not have to store every disc back up in the cloud which of course costs money per month.

It should allow me to experiment with server updates for the forum on my home network as opposed to trying to do everything on the live server. It should ultimately make my life easier as updating the forum software always leads to something breaking and takes time to resolve. Doing this work on a live server puts pressure on me to fix it all quickly which I want to avoid in the future. Generally I have used multiple other servers for testing but that again costs more money.

Also the server was dealing with several thousand attacks from China and Singapore yesterday. I have not seen or heard of any server timeouts or slowdowns over the past couple of days. so my firewall rules are still holding up just fine. Fail2ban takes care of a lot of things but I had to write custom scripts because of the way the requests are made in order to avoid detection. Things like fake Google bots are constant nuisance and have to be scanned and IP verified individually. Things are done hourly among multiple other detection systems I have put in place.

Incidentally IPs are being reported to the abuse database regularly https://www.abuseipdb.com/user/170367 . It has a limit of 5, 000 reports per day which is how I knew about the attacks because I got to an email saying I had hit the limit for that day. The reporting is actually queued as to try and avoid hitting the reporting limits, but the attacks yesterday must have been that bad even the queueing wasn't enough. Things seem again "quiet" today.

Incidentally some real-time server updates using conical live patch system causes nginx to crash. I have opened up a bug report and support ticket but progress is incredibly slow / next to nonexistent. Generally @rubber_jonnie is on the ball in telling me when the server isn't responding. As long as I'm at home , I can restart nginx quickly. But it can be an issue if I am not at home, IE on weekends. In which case the server may be down until I get home Sunday nights.

[rubber_jonnie]

The server may intermittently go down for about half an hour over the next few weeks.

The reason for this is I am converting to full fibre and I will able to download and store full disk server backups on my home network. This should help too slightly reduce monthly operational costs as I will not have to store every disc back up in the cloud which of course costs money per month.

It should allow me to experiment with server updates for the forum on my home network as opposed to trying to do everything on the live server. It should ultimately make my life easier as updating the forum software always leads to something breaking and takes time to resolve. Doing this work on a live server puts pressure on me to fix it all quickly which I want to avoid in the future. Generally I have used multiple other servers for testing but that again costs more money.

Also the server was dealing with several thousand attacks from China and Singapore yesterday. I have not seen or heard of any server timeouts or slowdowns over the past couple of days. so my firewall rules are still holding up just fine. Fail2ban takes care of a lot of things but I had to write custom scripts because of the way the requests are made in order to avoid detection. Things like fake Google bots are constant nuisance and have to be scanned and IP verified individually. Things are done hourly among multiple other detection systems I have put in place.

Incidentally IPs are being reported to the abuse database regularly https://www.abuseipdb.com/user/170367 . It has a limit of 5, 000 reports per day which is how I knew about the attacks because I got to an email saying I had hit the limit for that day. The reporting is actually queued as to try and avoid hitting the reporting limits, but the attacks yesterday must have been that bad even the queueing wasn't enough. Things seem again "quiet" today.

Incidentally some real-time server updates using conical live patch system causes nginx to crash. I have opened up a bug report and support ticket but progress is incredibly slow / next to nonexistent. Generally @rubber_jonnie is on the ball in telling me when the server isn't responding. As long as I'm at home , I can restart nginx quickly. But it can be an issue if I am not at home, IE on weekends. In which case the server may be down until I get home Sunday nights.

I'll always try to let you know if there is an issue, no problems there.

Sounds like a plan to reduce costs, always a win there and being able to have a dev/uat environment is always a bonus. Nobody wants to break prod...

[exxos]

I got Grok to optimise one of my fail2ban jails as its huge.. So if everyone suddenly gets banned.. thats why.. I will try and keep an eye on it all...