Last modified: 2014-01-29 18:44:57 UTC
Looks like bugzilla 59979 all over again. Search for "~2014" and the date shows 15 January 2014. That page was edited on 16, 17, 18 and 19 January.
I'm unsure what was going on but I've restarted the incremental indexer as documented here: https://wikitech.wikimedia.org/wiki/Search#Main_indexer_on_searchidx2.2Fsearchidx1001_is_stuck Hopefully it'll recover.
It ran OK on or about 19 January, but the index updating did not get done on 21 January.
The indexing machine is currently thrashing its disk something fierce. I'd dig into what is causing it but the service is end of life.... At this point I'm going to advise waiting to see if it gets through this rough patch. Thrashing: Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 11.00 196.00 177.67 12.84 13.04 141.82 172.99 504.55 70.14 983.78 2.68 100.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 7.00 285.00 198.67 20.93 15.47 154.11 173.22 436.21 49.78 990.56 2.07 100.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 8.00 234.67 180.33 15.13 14.01 143.83 172.03 390.09 66.87 810.71 2.41 100.00 Adding Andrew Otto as a CC in case this could use some love from ops.
Index looks like it's been updating fairly regularly now. Closing this as I think we're done here. Please reopen if we find things failing to update again.
