Last modified: 2014-10-21 11:06:32 UTC
For the hour 2014-08-23T20:xx:xx, none [1] of the the four sources' bucket was marked successful. What happened? [1] _________________________________________________________________ qchris@stat1002 // jobs: 0 // time: 10:06:45 // exit code: 0 cwd: ~/cluster-scripts ./dump_webrequest_status.sh +---------------------+--------+--------+--------+--------+ | Date | bits | text | mobile | upload | +---------------------+--------+--------+--------+--------+ [...] | 2014-08-23T18:xx:xx | . | . | . | . | | 2014-08-23T19:xx:xx | . | . | . | . | | 2014-08-23T20:xx:xx | X | X | X | X | | 2014-08-23T21:xx:xx | . | . | . | . | | 2014-08-23T22:xx:xx | . | . | . | . | [...] +---------------------+--------+--------+--------+--------+ Statuses: . --> Partition is ok X --> Partition is not ok (duplicates, missing, or nulls)
Neither ulsfo, nor eqiad hosts are affected Only esams hosts are affected. Some hosts are having missing duplicates (like amssq57.esams.wikimedia.org), some hosts are having missing lines (like cp3016.esams.wmnet).
Closing, as the issue is too far back to properly debug it.