Last modified: 2014-02-26 12:53:48 UTC
In E3, we recently ran in to a situation where post-hoc analysis revealed logging that should in theory have been impossible given the user experience and bucketing logic (bug 47590). It would be relatively easy to set up alerts if actions like logging a test condition impression for a user id bucketed as control occurs, letting us know about errors as they happen. Since each experimental design is different, this may require setting up custom alerts per schema/log.
Here's one: count(page-impression) >= count(page-save-attempt) >= count(page-save-success)
Super-simple approach: a repository which consists of a single script and a 'sql' subdirectory. The subdirectory contains SQL queries, one to a file. Each query should return a boolean. When the script is launched, it iterates through these queries and executes each in turn against the database. If any query returns 0, an e-mail alert is sent. A cron job ensures the script is invoked at regular intervals. If we had this, Steven and Dario would be able to add and remove assertions very easily, using git rm / git add and git review. I'd use puppet to configure a cron job to keep the repository state up-to-date.
[moving from MediaWiki extensions to Analytics product - see bug 61946]
