Last modified: 2012-10-26 09:06:11 UTC
beta robots.txt has been made to disallow anything (bug 37189). The Apaches are currently serving the file from /home/wikipedia/common/robots.txt which is tracked in git. We should make the Apache conf to alias robots.txt to another, labs only, file.
https://gerrit.wikimedia.org/r/#/c/21602/
Was merged a few time ago already. Note that each wiki can use [[Mediawiki:robots.txt]] to override the default generated robots.txt.