Ruled out, in order:
Traffic drop. Request volume was identical either side, 4,216 at 11:00 against 4,493 at 12:00. Distinct IPs actually went up, 2,818 to 3,297.
A single source leaving. Compared five minute windows either side. Only one IP vanished and it accounted for 28 requests, nowhere near enough.
Nginx restarting. Last start was three days earlier, nothing in the journal for that window.
An attack holding sockets open. All connections were idle keep-alive with empty send and receive queues. client_header_timeout is 5 seconds with reset_timedout_connection on, so a connection that opens and sends nothing is killed almost immediately, it cannot accumulate.
The metric being wrong. Cross-checked stub_status against ss three times, they agree within about ten. It is a genuine peak across sixty one-second samples, not a single reading landing on a burst.
Squid, and a couple of specific crawlers. All steady across the drop.
The one thing that has trended alongside it is protocol mix. HTTP/2 was 93% of requests a week ago and is 33% now, with HTTP/1.1 taking up the slack. That fits fewer multiplexed browser sessions and proportionally more crawler traffic, and explains a gradual rise in connections, but not a cliff at 11am.
Everything that actually matters is healthy. PHP listen queue has been zero throughout and CPU peaks around 50%.
TL;DR: I keep seeing spikes of hundreds of nginx connections, but there is nothing in the logs showing what that traffic actually was. It looks like a botnet, except the logged request volume is identical between 10am and 12pm either side of the drop. So what were those connections doing? This one generally has me beat
- Capture.PNG (182.55 KiB) Viewed 55 times
This *might* have been my own doing a while back for *reasons*. 
