Last modified: 2014-02-09 12:39:37 UTC
Currently all crawling is disabled; it's probably just some stupid default value, I can't imagine any reason why (it's all public logs, loading some static HTML can't kill a server). Considering that labs services must not be relied on, at the very least you should whitelist the Wayback machine https://archive.org/about/exclude.php . I don't know if empty disallow or Allow: / is best. User-agent: ia_archiver Disallow: