Last modified: 2014-05-19 04:48:51 UTC
Several users have reported seeing a redirect loop error page when attempting to access http://logstash.wikimedia.org. Using the https protocol seems to correct the issue. I have personally not been able to recreate the redirect loop in my browser, but I can recreate it using curl from the command line: curl -v -L http://logstash.wikimedia.org Expected: * GET / HTTP/1.1 * 301 Moved Permanently; Location: https://logstash.wikimedia.org/ * GET / HTTP/1.1 * 401 Authorization Required Got: * GET / HTTP/1.1 * 301 Moved Permanently; Location: https://logstash.wikimedia.org/ * GET / HTTP/1.1 * 301 Moved Permanently; Location: https://logstash.wikimedia.org/ * ... Behavior seems to be intermittent. During testing to file this bug I could recreate the looping behavior for several requests in a row followed by several proper redirects followed again by several redirect loops. This seems to indicate that a single node in a load balanced cluster is misbehaving.
Ori suggested assigning to Faidon for triage/resolution.
It looks like Varnish is caching the 301 response under some circumstances: < HTTP/1.1 301 Moved Permanently < Server: nginx/1.1.19 < Date: Mon, 27 Jan 2014 21:12:13 GMT < Content-Type: text/html; charset=iso-8859-1 < Transfer-Encoding: chunked < Connection: keep-alive < Location: https://logstash.wikimedia.org/ < Vary: Accept-Encoding < X-Varnish: 2005611465 2005610948 < Age: 102 < Via: 1.1 varnish < X-Cache: cp1044 hit (450) Maybe a cache-control header needs to be added when sending the 301 to switch protocols?
I think there are two possible solutions to this: 1. Send a Cache-Control header along with the 301 response to keep Varnish from caching the redirect 2. Add the X-Forwarded-Proto header to the hash inside Varnish sub vcl_hash { if (req.http.X-Forwarded-Proto) { set req.hash += req.http.X-Forwarded-Proto; } }
Can X-Forwarded-Proto by used with a Vary header?
(In reply to comment #4) > Can X-Forwarded-Proto by used with a Vary header? *be used
Change 110393 had a related patch set uploaded by BryanDavis: varnish: Add X-Forwarded-Proto to Vary on redirects https://gerrit.wikimedia.org/r/110393
Change 110396 had a related patch set uploaded by Ori.livneh: kibana: Set Vary: X-Forwarded-Proto on HTTP -> HTTPS 301s https://gerrit.wikimedia.org/r/110396
Change 110396 merged by Ori.livneh: kibana: Set Vary: X-Forwarded-Proto on HTTP -> HTTPS 301s https://gerrit.wikimedia.org/r/110396
Change 110393 abandoned by BryanDavis: varnish: Add X-Forwarded-Proto to Vary on redirects Reason: Ori's workaround in Id2cc29f911fa36805320cdb606a5da1226c9d230 is less yucky https://gerrit.wikimedia.org/r/110393
Ori fixed this with a cool bit of Apache config: RewriteRule ^/(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,E=ProtoRedirect] Header always merge Vary X-Forwarded-Proto env=ProtoRedirect This works around an annoying bit of esoteric Apache internals where headers are stripped during redirect generation and have to be reapplied later. See http://stackoverflow.com/a/3711110/8171 for a description of the underlying "feature" that is being addressed.
