Last modified: 2014-07-24 21:00:40 UTC
Created attachment 16004 [details] 503 error from Parsoid on test2 Seen on test2wiki several times 22 July, see screen shot.
I pull production logs onto bast1001 and looked for anything odd, but couldn't find anything for the page in the screenshot. ssastry@bast1001:~/logs$ grep test2wiki * | grep 0.229 wtp1015:[info][test2wiki/0.229214833204894_Moved] starting parsing wtp1015:[warning][test2wiki/0.229214833204894_Moved?oldid=66911] Cache MISS: http://10.2.2.29/test2wiki/0.229214833204894_Moved?oldid=66910 wtp1015:[info][test2wiki/0.229214833204894_Moved?oldid=66911] completed parsing in 227 ms Requires more investigation.
Load graphs also look normal here: https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Parsoid+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 (the network traffic spike is me pulling the production logs).
I have definitely seen Parsoid 503 several times today on test2wiki.
I believe you. I should have clarified that I am just recording information here as I look around.
I also verified that all parsoid nodes have the same version of the code (just to rule out the possibility of a bad deploy) by running "dsh -g parsoid -M curl http://127.0.0.1:8000/_version" on tin.
From #mediawiki-pdfhack @ 12:07 CST "<mwalker> cscott, morning; I have three patches for you :) also -- I've been getting a lot of 503s from parsoid lately" We are now wondering if this could be from varnish that is sitting in front of the parsoid cluster.
I just dumped a couple of 503 responses; they look like this: { "code": 503, "headers": { "server": "Varnish", "content-type": "text/html; charset=utf-8", "retry-after": "5", "content-length": "419", "accept-ranges": "bytes", "date": "Wed, 23 Jul 2014 18:00:14 GMT", "x-varnish": "1859100242", "age": "0", "via": "1.1 varnish", "connection": "close", "x-cache": "cp1045 frontend miss (0)" }, "body": "\n<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\n \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n<html>\n <head>\n <title>503 Service Unavailable</title>\n </head>\n <body>\n <h1>Error 503 Service Unavailable</h1>\n <p>Service Unavailable</p>\n <h3>Guru Meditation:</h3>\n <p>XID: 1859100242</p>\n <hr>\n <p>Varnish cache server</p>\n </body>\n</html>\n" }
Change 148924 had a related patch set uploaded by GWicke: Bump Parsoid backend connection limit https://gerrit.wikimedia.org/r/148924
Change 148924 merged by Cmjohnson: Bump Parsoid backend connection limit https://gerrit.wikimedia.org/r/148924
