Wikimedia Bugzilla is closed!

To understand how Notifications impact the behavior of new editors in Wikipedia, we are planning an A/B test the week of June 10, which will let us study their productivity by comparing two cohorts, as described below, and on this page:

http://meta.wikimedia.org/wiki/Research:Notifications/Experiment_1

The research question we're trying to answer is "Do the new Echo Notifications help people become more productive?" This answer will be determined primarily by analyzing total edits from each cohort -- and number of edits that were not reverted within a week.

To that end, we need to accomplish two goals this week:

1) divide new users who register after June 11 into two buckets:
a) Pre-Echo: All notification types disabled (legacy talk notification enabled)
b) Echo-Current: Existing Echo notification defaults for new users

Users will be bucketed based on the last digit of their userID (even digits = pre-Echo cohort; odd digits = Echo-Current cohort). 

2) modify the 'hidden preference' to disable all email notifications from Echo, and to restore the previous talk page messages users were getting before Echo. 

As a result, the first 'Pre-Echo' cohort will have both web and email preferences disabled (including Echo emails), using a 'hidden preference' that is currently part of the Echo code; they will see the OBOD for a one-week period, to match conditions prior to Echo. The second 'Echo-Current' cohort will get exactly what they get today (with edit reverts disabled for new users, both on web and email).

After a one-month period, we will change the 'hidden preference' for the first cohort, so that they get the same 'Echo-Current' settings as new users do today.

Note that this study needs to be conducted the week of June 10, before the Visual Editor research starts impacting our ability to bucket users into cohorts, which is why this ticket has a high priority. The reason we want to conduct this study right away is that we want to have some productivity data available when discussing the deployment of Echo on other language wikis later this summer.

Dario Taraborelli and Aaron Halfaker are the primary investigators for this research, and may provide more details on this thread, as needed.

P.S.: If clicktracking data is available by next week, we would like to also study the clickthrough rate of the two cohorts as part of this test, though that is not a hard requirement.