Last modified: 2013-04-03 07:06:12 UTC
On gallium, I use a local repository as a reference when doing a git clone: Example: git clone --reference /var/lib/jenkins/git/mediawiki/core \ -- https://gerrit.wikimedia.org/r/p/mediawiki/core.git \ /tmp/DeleteThis Under Jenkins, that eventually fails with a RPC error: 15:54:17 [exec] error: RPC failed; result=22, HTTP code = 502 Under shell it seems to works: $ git clone --reference /var/lib/jenkins/git/mediawiki/core -- https://gerrit.wikimedia.org/r/p/mediawiki/core.git /tmp/DeleteThis Initialized empty Git repository in /tmp/DeleteThis/.git/ remote: Counting objects: 380902, done remote: Finding sources: 100% (380902/380902) remote: Getting sizes: 100% (12861/12861) remote: Compressing objects: 100% (12841/12841) remote: Total 380902 (delta 322620), reused 373266 (delta 321052) Receiving objects: 100% (380902/380902), 96.91 MiB | 3.02 MiB/s, done. Resolving deltas: 100% (325746/325746), done. So might be related to Jenkins
Example: https://integration.mediawiki.org/ci/job/Ext-AFTv5/4/console
Possible strack trace reported by Chad : [2012-08-28 15:48:11,637] WARN /r : Internal error during upload-pack org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:154) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:98) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.eclipse.jgit.util.TemporaryBuffer.doFlush(TemporaryBuffer.java:151) at org.eclipse.jgit.http.server.UploadPackServlet$1.flush(UploadPackServlet.java:167) at org.eclipse.jgit.transport.SideBandOutputStream.flush(SideBandOutputStream.java:125) at org.eclipse.jgit.storage.pack.PackOutputStream.flush(PackOutputStream.java:135) at org.eclipse.jgit.storage.pack.PackWriter.writePack(PackWriter.java:895) at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1197) at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1055) at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:611) at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:521) at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:176) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:237) at com.google.gerrit.httpd.GitOverHttpServlet$UploadFilter.doFilter(GitOverHttpServlet.java:222) at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235) at org.eclipse.jgit.http.server.UploadPackServlet$Factory.doFilter(UploadPackServlet.java:141) at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235) at org.eclipse.jgit.http.server.RepositoryFilter.doFilter(RepositoryFilter.java:147) at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235) at org.eclipse.jgit.http.server.NoCacheFilter.doFilter(NoCacheFilter.java:80) at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235) at org.eclipse.jgit.http.server.glue.UrlPipeline.service(UrlPipeline.java:215) at org.eclipse.jgit.http.server.glue.SuffixPipeline.service(SuffixPipeline.java:101) at org.eclipse.jgit.http.server.glue.MetaFilter.doFilter(MetaFilter.java:163) at org.eclipse.jgit.http.server.glue.MetaServlet.service(MetaServlet.java:133) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at com.google.gerrit.pgm.http.jetty.ProjectQoSFilter.doFilter(ProjectQoSFilter.java:132) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gerrit.httpd.ProjectDigestFilter.doFilter(ProjectDigestFilter.java:111) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:921) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:856) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handleAsync(Server.java:397) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:601) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:410) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450) at java.lang.Thread.run(Thread.java:679)
I hit the same error multiple times trying: git clone --quiet --recursive https://gerrit.wikimedia.org/r/p/mediawiki/core.git -b fundraising/1.20 /srv/www-staging/org/wikimedia/payments
We made this a little better by upping Apache's proxy timeout. Dropping the --recursive also helps, as we've discussed :) I'd like to mark this FIXED unless we're still hitting it regularly.
Being bold, this no more happen in my experience though someone using a slow connection might eventually have the issue.
This should be doubly fixed now: https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log&diff=65291&oldid=65286
For history purpose, copy pasting from wikitech: ^demon: gerrit: finished doing jgit gc on mediawiki/core. Repo size 3G -> 323M. Fresh clone time to <5m to localhost, <1.5m to other hosts inside wmf. I rock. Backup's in /home/demon/core.git in case something goes wrong. Bed time. Congratulations Chad :-]
