Last modified: 2014-02-12 23:53:35 UTC
A friend showed me their Android phone where the following could be reproduced: searching for certain terms in Google provided a link to the relevant page under en.m.wikipedia.org, which, when clicked, first sent the browser to the relevant page, which then redirected to http://wikimediafoundation.org/wiki/Missing_wiki , which does not exist. Given the number of cache poisoning bugs we've had in the mobile varnishes, this sequence did not surprise me. What did somewhat surprise me was that when I grepped for the relevant URL in the configuration files, I couldn't find it. But the request logs on emery confirm that this is happening very regularly, e.g. sq61.wikimedia.org 3860736 2013-04-28T06:52:16.463 0 1.2.3.4 TCP_NEGATIVE_HIT/404 5619 GET http://wikimediafoundation.org/wiki/Missing_wiki NONE/- text/html http://www.google.com.au/search?rlz=...&ei=...&q=key+to+hunting+fox+qith+bow&oq=key+to+hunting+fox+qith+bow&gs_l=... - Mozilla/5.0%20(Linux;%20U;%20Android%204.1.2;%20en-au;%20GT-I9300%20Build/JZO54K)%20AppleWebKit/534.30%20(KHTML,%20like%20Gecko)%20Version/4.0%20Mobile%20Safari/534.30 en-AU,%20en-US - Potentially personally identifying information has been stripped from the referrer and IP field of this log line. You can find the original data in sampled-1000.tsv.log on emery. missing.php generates redirects to http://meta.wikimedia.org/wiki/Missing_wiki , which also appears in the logs for mobile browsers. Perhaps something is rewriting that redirect to wikimediafoundation.org.
Thanks Tim I'll try to recreate
Related URL: https://gerrit.wikimedia.org/r/61519 (Gerrit Change I3d02804170f7e502300329740cba9f45437a24fa)
https://gerrit.wikimedia.org/r/61519 (Gerrit Change I3d02804170f7e502300329740cba9f45437a24fa) | change APPROVED and MERGED [by Faidon]
Related URL: https://gerrit.wikimedia.org/r/61537 (Gerrit Change I18950df30ad96f2ab7f21333e8c7b316f05e1379)
https://gerrit.wikimedia.org/r/61537 (Gerrit Change I18950df30ad96f2ab7f21333e8c7b316f05e1379) | change APPROVED and MERGED [by Tim Starling]
Assigned Mingle Ticket https://mingle.corp.wikimedia.org/projects/mobile/cards/749
Related URL: https://gerrit.wikimedia.org/r/61539 (Gerrit Change I17989fc4124bcb296e809ba0a13f41c043af7d0d)
The logs gathered with I18950df3 show that most of the remaining hits on http://meta.wikimedia.org/wiki/Missing_wiki are due to invalid hostnames, such as "www.en.wiktionary.org". Such hits will be converted to 404s when I17989fc4 is deployed. The rate of requests to http://wikimediafoundation.org/wiki/Missing_wiki has dropped by a factor of 50 or so. About a third of the remaining log entries are from Optus client IPs, suggesting a carrier issue.
Now that I17989fc4 has been deployed, is this fixed? Thanks for taking care of this, Tim!
The root cause was fixed in Varnish with I3d028041, we immediately verified this. In the course of debugging this, Tim also found a couple of related bugs, like the redirect to wikimediafoundation.org instead of meta, that were fixed (among other cleanups) with I17989fc4.
