Last modified: 2013-12-13 15:18:29 UTC
The search indexer instance attempts to reach the *.beta.wmflabs.org which is pointing to a public IP part of labs. That does not work. A hacky quick solution would be to rewrite any request sent from the search indexer for the squid public IP (208.80.153.219) to use the internal squid instance private IP.
Iptable rule would be: iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.0.17
Needs to be puppetized.
Mailled ops to figure out how to get the iptables rule to be puppetized.
I have no idea how to puppetize the iptables rule mentionned in comment #1. So I have filled RT #4824 that list the mail exchanges on ops mailing list.
same issue happens on deployment-upload.pmtpa.wmflabs which is an internal proxy for thumbnails generation.
The text cache has been migrated out of deployment-squid [10.4.0.17] to a varnish instance deployment-cache-text1 [10.4.1.133] The iptables command is thus: iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133
(In reply to comment #5) > same issue happens on deployment-upload.pmtpa.wmflabs which is an internal > proxy for thumbnails generation. That was unrelated. The thumb handler points directly to the varnish cache via its private IP.
Rephrasing summary. Wikidata is hit by the same issue (was bug 49300) when some script attempt to access: http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&prop=info&redirects=1&converttitles=1&format=json&titles=Keyboard+Cat The RT is https://rt.wikimedia.org/Ticket/Display.html?id=4824 The workaround is to use an iptables rule to rewrite networking packet: iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133
*** Bug 49300 has been marked as a duplicate of this bug. ***
The iptables command for all the beta public IP: iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133 iptables -t nat -I OUTPUT --dest 208.80.153.242 -j DNAT --to-dest 10.4.0.211 iptables -t nat -I OUTPUT --dest 208.80.153.243 -j DNAT --to-dest 10.4.0.51 iptables -t nat -I OUTPUT --dest 208.80.153.244 -j DNAT --to-dest 10.4.0.48 iptables -t nat -I OUTPUT --dest 208.80.153.243 -j DNAT --to-dest 10.4.1.82
https://gerrit.wikimedia.org/r/#/c/101192/ converts the above iptables rules to ferm. They can be applied on instances using the puppet class role::beta::natfixup. I have applied the class on the following instances: deployment-apache32 deployment-apache33 deployment-bastion deployment-jobrunner08 deployment-parsoid2 deployment-video06
Change 101209 had a related patch set uploaded by Hashar: beta: ferm on appservers must allow port 80 https://gerrit.wikimedia.org/r/101209
Change 101210 had a related patch set uploaded by Hashar: role::parsoid::beta must allow port 8080 https://gerrit.wikimedia.org/r/101210
Change 101209 merged by ArielGlenn: beta: ferm on appservers must allow port 80 https://gerrit.wikimedia.org/r/101209
Change 101210 merged by ArielGlenn: role::parsoid::beta must allow port 8000 https://gerrit.wikimedia.org/r/101210
Everything working again now. I will close the related RT #4824.
