Last modified: 2013-09-20 22:00:17 UTC
When one of the replica servers went down on Wednesday[1], one of my userDBs (p50380g50729__grantsbot) went with it. That db and the data in it looks to be gone forever. It sounds like wipe-and-reset is the standard practice when these replicas go down, so I imagine that these dbs will be deleted any time there's a hardware malfuction. Fortunately for me, there wasn't much data there yet (I'm still migrating). However, it's still lost work. My bots runs joins on replicas and store data in custom tables, so I can't host it on tools-db. This is a pretty common workflow for bots in general. Therefore, there should be backup systems in place for restoring those userDBs after a failure. I can and will run daily backups of all my tables from now on, but that doesn't solve the problem: there will still be potential for data loss any time there's a hardware failure. For the many, many other bots that run more frequently than once a day, the data loss could be even more significant and disruptive. Bots are essential to the maintenance of our projects. Can we get better backups systems for userDBs hosted on replicas? - Jonathan [1]http://lists.wikimedia.org/pipermail/labs-l/2013-September/001668.html
