Last modified: 2014-04-21 18:47:28 UTC
It seems fixing bug 58876 made the logged requests from ssl terminators jump from ~50/day to 1.3M/day [1] ... which is actually good. As the ~1.3M/day seems more plausible than ~50/day ... Did some of the (maybe) relevant changes https://gerrit.wikimedia.org/r/#/c/105449/ https://gerrit.wikimedia.org/r/#/c/105450/ https://gerrit.wikimedia.org/r/#/c/105451/ really unbreak edit.tsv writing for ssl terminators as a side-effect? If so, how could they have unbroken storing requests from ssl terminators ... should not x-encoded requests have gotten logged nonetheless? (They for sure were in the stream before, as I checked before writing the changes above, and some are visible in the edit tsvs for the previous dates.) Was there some network issue beforehand? Was udp-filter (see [4] for its invocation) misbehaving? Once we understand what went on, would we need fixes elsewhere? Is the 1.3M/day now finally about correct? Some more pointers: * The above gerrit changes got deployed to ssl1001 on 2014-01-14 ~18:56 [2]. * Heavy ssl traffic in the edit tsvs starts on 2014-01-14T18:59:11.634 and is limited to ssl1001 at first. On 2014-01-14T19:06:23.209 other machines kick in. * There was a deployment window on 2014-01-14 19:00-21:00 upgrading group1 1.23wmf9->1.23wmf10 [3], but it looks unrelated, as we're seeing ssl requests pour in for unaffected wikis. [1] ___________________________________________________________ qchris@stat1002 // 0 // 22:13:50 cwd: ~ zgrep -c '^ssl' /a/squid/archive/edits/edits.tsv.log-201401*.gz [...] /a/squid/archive/edits/edits.tsv.log-20140112.gz:47 /a/squid/archive/edits/edits.tsv.log-20140113.gz:18 /a/squid/archive/edits/edits.tsv.log-20140114.gz:17 /a/squid/archive/edits/edits.tsv.log-20140115.gz:557830 /a/squid/archive/edits/edits.tsv.log-20140116.gz:1307081 /a/squid/archive/edits/edits.tsv.log-20140117.gz:1336017 /a/squid/archive/edits/edits.tsv.log-20140118.gz:1345793 /a/squid/archive/edits/edits.tsv.log-20140119.gz:1379590 /a/squid/archive/edits/edits.tsv.log-20140120.gz:1343811 /a/squid/archive/edits/edits.tsv.log-20140121.gz:1358968 [2] See http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/20140114.txt around 18:54, look for ottomata & qchris. [3] https://wikitech.wikimedia.org/wiki/Deployments/Archive/2014 [4] https://git.wikimedia.org/blob/operations%2Fpuppet/aa4f6383ac7fc8dc5515d8466574368658f959ed/templates%2Fudp2log%2Ffilters.oxygen.erb#L13
Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/cards/1398
