Last modified: 2013-04-23 14:31:46 UTC
Gerrit with the shipped Jetty servlet container causes problems, when the client connection pool fills up. Then gerrit's log get spammed with messages like [2013-02-19 14:55:54,592] WARN org.eclipse.jetty.io.nio : Dispatched Failed! SCEP@44fb799e{l(/127.0.0.1:47357)<->r(/127.0.0.1:8080),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@59ed9773,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@3699d37e This spamming renders Gerrit extremely unresponsive. When the problem hit us around 2013-02-19 12:55:22,464 UTC, spamming through above messages amounted to about 1Gb of logs in ~2 hours. The problem is known and documented upstream: http://code.google.com/p/gerrit/wiki/Scaling#Jetty The problem hit us again at 2013-02-21 ~10:50 UTC, and also in one of the last days. Possible mitigation strategies are: * hand connection limiting off to apache https://gerrit.wikimedia.org/r/#/c/50591 * Trading Jetty for another container (e.g.: Tomcat)
This has been merged & deployed.
We haven't had this problem for the last week, so...knock on wood...I think it's resolved. Going to go ahead and mark this as FIXED. Will reopen in a heartbeat if we have the problem again.
