Last modified: 2014-10-30 21:31:46 UTC
Dan reported through email that we're seeing [Sun Sep 28 14:24:23 2014] [error] [client 10.68.16.65] client denied by server configuration: /srv/wikimetrics/wikimetrics/api.wsgi lines. It's on wikimetrics1.eqiad.wmflabs. Every 30 seconds, there are two lines like the above in /var/log/apache2/error.metrics.log. . What is that?
The first occurrence I could find in the logs has a timestamp of [Sat Sep 20 14:37:58 2014] ... almost two weeks back. I could neither find relevant changes in the wikimetrics module nor the wikimetrics role around that time that look related. Does anyone happen to know what got deployed/changed on 2014-09-20?
The webserver's config around api.wsgi is from 2014-01-20, so that should not have changed in the last two weeks.
D'oh! The IP is not the machine's IP, but that of dynamicproxy-gateway.eqiad.wmflabs.
Seems the issue is triggered by pointing a browsing to https://metrics.wmflabs.org/server-status
Then is a lawful error, I think there is no harm on that one returning a 403. Loads of interesting things on server-status: http://www.apache.org/server-status
While we could ignore them, those requests are coming very controlled and reliably. Instead of expecting two erroneous requests every 30 seconds, which turns our logs into noise collectors, let's just turn them off, or fix them :-) So I took another look. It's apache_status.py [1], which we get through the default puppet apache setup, which includes apache::monitoring. This status script should report to monitoring daemons. This status script requests http://localhost:80/server-status (which should be whitelisted). However, the wikimetrics apache-config redirects /all/ http requests (so also this one) to https. Hence, the script follows the redirect to https://metrics.wmflabs.org/server-status which turns the internal request to an external, as it's coming the the ssl terminator. And so we finally (correctly) deny it. That should be easy to fix ... [1] https://git.wikimedia.org/blob/operations%2Fpuppet.git/a1aa79fb6bd6e9ab329736acef788fe241d97b2f/modules%2Fapache%2Ffiles%2Fapache_status.py
Change 164583 had a related patch set uploaded by QChris: Allow local server-status requests on http https://gerrit.wikimedia.org/r/164583
Change 164583 merged by Ottomata: Allow local server-status requests on http https://gerrit.wikimedia.org/r/164583
deployment train picked this up choo choo!
