Last modified: 2014-08-27 22:25:20 UTC
Created attachment 16127 [details] Script to inject needed DNAT rules to connect to labsdb After rebooting wikimetrics-dev1, the instance could not connect to the databases. The reason was missing DNATs in firewall configuration. I could not find this requirement documented, nor puppetized. Is it documented somewhere? If not ... let's puppetize DNAT rules (if they are not yet). (Since I needed something right away, I wrote setup_dnat_rules.sh (see attachment), which contains the (somewhat redundant) DNAT rules from staging. Maybe it helps someone else in the future)
I do not remember us having to do this when we set up neither dev or staging when we fist set them up, which indicates that something might have changed on labs setup. I do not think this should be a bug on our end but we should confirm with otto 1st whether this needs to be puppetized.
(In reply to nuria from comment #1) > I do not remember us having to do this when we set up neither dev or staging > when we fist set them up, [...] It might be a new thing or not. I do not know. But we're fighting it once in a while: * at least on 2014-07-24 there was also need to do it [1], * when I rebooted a machine some days ago, * just now (2014-08-05), we again needed to reboot a machine. Tim's bug that I linked above is from 2014-02-25. So I doubt it's a new thing. > I do not think this should be a bug on our end [...] I wanted a place to track it on our end. And I wanted a place to put the DNAT script. Hence, I filed it for us for now, and linked Tim's bug. [1] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-analytics/20140724.txt [13:53:26] <milimetric> qchris: do you have any idea how to iptables-restore this: http://paste.ubuntu.com/7847723/
AFAIK, the replica DB servers were never accessible under the enwiki.labsdb:$STANDARDPORT scheme without additions to /etc/hosts and iptables on the client side.
This bug hit us again today. See http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-analytics/20140815.txt starting on [12:05:53]
Automatic loading of iptables settings is getting implemented in https://gerrit.wikimedia.org/r/#/c/156599/ Once that has been merged, the issue decreases to how to create /etc/iptables.conf automatically.
