Last modified: 2014-07-07 18:03:26 UTC
When emery got decomissioned, erbium around mid-April 2014 had it's udp2log instance turned from a multicast consumer into a consumer of only varnishncsa. Hence, since then, erbium only receives logs for the traffic the ssl terminators cause on the caches, but not the logs for traffic arriving at the ssl terminators. However, to correctly count ssl traffic, webstatscollector's filter (which runs on erbium) relies on seeing the logs at the ssl terminators. So as a result, ssl traffic was not counted in webstatscollector's output since mid-April 2014. Discussing a bit with ottomata, it seems the best way forward [1] [2] is to try moving webstatscollector's filter to a host that still consumes the udp2log multicast. oxygen consumes the udp2log multicast, and does not look overloaded, so we could try moving webstatscollector's filter there. ---- [1] We could instead turn multicast consumption on erbium on again. But that would also affect all the other udp2log filters that meanwhile got added to erbium. So it would cause further issues. [2] We could update webstatscollector to stop dropping local http requests from the ssl terminators. That would be nice as it would mean that the overall picture of the webstatscollector pipeline gets simpler. However, people are afraid to touch webstatscollector code, and we want to phase it out anyways.
Change 143775 had a related patch set uploaded by QChris: Feed logs from ssl terminators again into webstatscollector's filter https://gerrit.wikimedia.org/r/143775
As tomorrow (2014-07-04) is a holiday in the US, today (2014-07-03) is the last working day of this week, and it's bad karma to deploy on the last working day of a week. So it means that merging the fix will have to wait until next week. :-/
Change 143775 merged by Gage: Feed logs from ssl terminators again into webstatscollector's filter https://gerrit.wikimedia.org/r/143775
The first hourly webstatscollector files after the above merge contain ssl requests again. Thanks Gage!
