Last modified: 2014-10-22 10:54:53 UTC
None of the webrequest partitions [1] for 2014-10-21T11/1H have been been marked successful. What happened? [1] _________________________________________________________________ qchris@stat1002 // jobs: 0 // time: 10:12:12 // exit code: 0 cwd: ~ ~/cluster-scripts/dump_webrequest_status.sh +------------------+--------+--------+--------+--------+ | Date | bits | mobile | text | upload | +------------------+--------+--------+--------+--------+ [...] | 2014-10-21T09/1H | . | . | . | . | | 2014-10-21T10/1H | . | . | . | . | | 2014-10-21T11/1H | X | X | X | X | | 2014-10-21T12/1H | . | . | . | . | | 2014-10-21T13/1H | X | X | . | . | [...] +------------------+--------+--------+--------+--------+ Statuses: . --> Partition is ok M --> Partition manually marked ok X --> Partition is not ok (duplicates, missing, or nulls) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(2014-10-21T13/1H is handled in bug 72353) The affected period is 2014-10-21T11:41:19/2014-10-21T11:59:09. It affected only ulsfo caches, but all ulsfo caches. The affected period shows ~5M duplicates, which are worth * 4 minutes of ulsfo data, or * 36 seconds of total data. The affected period shows ~6M missing lines, which are worth * 5 minutes of ulsfo data, or * 41 seconds of total data. Ops reported [1] that the ulsfo->eqiad connection again caused issues. According to the IRC logs [2], the connection issues started around 10:30, which matches the affected period. [1] https://lists.wikimedia.org/mailman/private/ops/2014-October/042427.html [2] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/20141021.txt