nginx connections puzzle

[exxos]

Odd one for anyone who fancies a puzzle. My nginx connection count sat spiky between 300 and 1200 per minute for about a week, then dropped to a flat 100 at 11am one morning and has stayed there since.

Ruled out, in order:

Traffic drop. Request volume was identical either side, 4,216 at 11:00 against 4,493 at 12:00. Distinct IPs actually went up, 2,818 to 3,297.

A single source leaving. Compared five minute windows either side. Only one IP vanished and it accounted for 28 requests, nowhere near enough.

Nginx restarting. Last start was three days earlier, nothing in the journal for that window.

An attack holding sockets open. All connections were idle keep-alive with empty send and receive queues. client_header_timeout is 5 seconds with reset_timedout_connection on, so a connection that opens and sends nothing is killed almost immediately, it cannot accumulate.

The metric being wrong. Cross-checked stub_status against ss three times, they agree within about ten. It is a genuine peak across sixty one-second samples, not a single reading landing on a burst.

Squid, and a couple of specific crawlers. All steady across the drop.

The one thing that has trended alongside it is protocol mix. HTTP/2 was 93% of requests a week ago and is 33% now, with HTTP/1.1 taking up the slack. That fits fewer multiplexed browser sessions and proportionally more crawler traffic, and explains a gradual rise in connections, but not a cliff at 11am.

Everything that actually matters is healthy. PHP listen queue has been zero throughout and CPU peaks around 50%.

TL;DR: I keep seeing spikes of hundreds of nginx connections, but there is nothing in the logs showing what that traffic actually was. It looks like a botnet, except the logged request volume is identical between 10am and 12pm either side of the drop. So what were those connections doing? This one generally has me beat :(

Capture.PNG

[exxos]

I finally figured it out...

Capture.PNG

The bottom rate spikes were PHP-FPM triggering.. So they SHOULD be in the log but not all of them was. :roll:

Seems at some point along the way they look problem was introduced into the nginx config :shrug: This *might* have been my own doing a while back for *reasons*.

So now my log is a lot more active now it has been fixed.. So it should be a lot more efficient in dealing with all the traffic now the log is actually showing what all the traffic is ! :stars: