Last modified: 2014-02-12 04:03:06 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T62940, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 60940 - Gerrit throwing internal server error with "status:closed" queries
Gerrit throwing internal server error with "status:closed" queries
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
Git/Gerrit (Other open bugs)
wmf-deployment
All All
: Unprioritized major (vote)
: ---
Assigned To: Nobody - You can work on this!
https://code.google.com/p/gerrit/issu...
: upstream
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-02-06 04:47 UTC by MZMcBride
Modified: 2014-02-12 04:03 UTC (History)
7 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description MZMcBride 2014-02-06 04:47:53 UTC
Something goofy is going on with "status:closed" queries in Gerrit lately.

Works (status:abandoned):

$ curl "https://gerrit.wikimedia.org/r/changes/?q=status:abandoned&n=1"
)]}'
[
  {
    "kind": "gerritcodereview#change",
    "id": "mediawiki%2Fextensions%2FFlow~master~Iade500a4c2fb8b96e88e5074c7ceff484f02cce9",
    "project": "mediawiki/extensions/Flow",
    "branch": "master",
    "change_id": "Iade500a4c2fb8b96e88e5074c7ceff484f02cce9",
    "subject": "Implement former SimpleAntiSpam extension in \\Flow\\View",
    "status": "ABANDONED",
    "created": "2014-02-06 04:22:23.000000000",
    "updated": "2014-02-06 04:30:14.000000000",
    "mergeable": true,
    "_sortkey": "002af04e0001b469",
    "_number": 111721,
    "owner": {
      "name": "Legoktm"
    },
    "_more_changes": true
  }
]


Fails (status:closed):

$ curl "https://gerrit.wikimedia.org/r/changes/?q=status:closed&n=1"
Internal server error

You can reproduce the same behavior by going to <https://gerrit.wikimedia.org> and inputting "status:closed" into the search input. Doing so results in:

---
Code Review - Error

500 Internal server error

[Continue]
---

This is a breaking change in the Gerrit Web API, causing [[mw:Gerrit/Reports]] to stop updating.
Comment 1 Antoine "hashar" Musso (WMF) 2014-02-06 14:50:36 UTC
Same in the search box (which uses the REST API I guess).

Since status:closes is equivalent to is:merged OR is:abandoned I tried that, and it fails as well.
Comment 2 Chad H. 2014-02-06 18:43:52 UTC
Pretty sure this is the exception we're seeing (swap merged for abandoned or closed and you'll find some more):

[2014-02-06 14:50:28,927] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /r/changes/?q=is:merged+OR+is:abandoned&n=25&O=1
java.sql.SQLException: setMaxRows() out of range. 2147483647 > 50000000.
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:988)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:974)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
        at com.mysql.jdbc.StatementImpl.setMaxRows(StatementImpl.java:2745)
        at com.google.gerrit.reviewdb.client.Change_Access_changes_GwtOrm$$13.allClosedNext(Unknown Source)
        at com.google.gerrit.server.query.change.SqlRewriterImpl$13.scan(SqlRewriterImpl.java:340)
        at com.google.gerrit.server.query.change.SqlRewriterImpl$PaginatedSource.scan(SqlRewriterImpl.java:645)
        at com.google.gerrit.server.query.change.SqlRewriterImpl$ChangeSource.read(SqlRewriterImpl.java:608)
        at com.google.gerrit.server.query.change.OrSource.read(OrSource.java:43)
        at com.google.gerrit.server.query.change.AndSource.readImpl(AndSource.java:110)
        at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:94)
        at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:261)
        at com.google.gerrit.server.query.change.QueryChanges.query0(QueryChanges.java:153)
        at com.google.gerrit.server.query.change.QueryChanges.query(QueryChanges.java:141)
        at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:108)
        at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:41)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:306)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:365)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
Comment 3 MZMcBride 2014-02-07 02:36:09 UTC
Chad or someone else who likely remembers where the Gerrit bug tracker is and has an account over there: can you please check if there's already an upstream ticket filed about this issue and if not, copy and paste this one over there and cross-reference?
Comment 4 Andre Klapper 2014-02-07 09:30:58 UTC
(In reply to comment #3)
> someone else who likely remembers where the Gerrit bug tracker is

https://www.mediawiki.org/wiki/Bug_management/Upstream_bugtrackers
Comment 5 Bartosz Dziewoński 2014-02-07 12:28:30 UTC
Upstream bug seems to be https://code.google.com/p/gerrit/issues/detail?id=2392
Comment 6 christian 2014-02-07 12:49:16 UTC
I can reproduce on a local gerrit that uses MySQL, so it's not limited
to our wmf gerrit.

Changing the database to H2 makes the error disappear.

It seems the issue is limited to the MySQL backend, as the error is
thrown from the MySQL connector, whose hard limit for 50000000 rows is
below 2^31-1.

I checked gerrit upstreams stable-2.8 branch, and it does not seem to
have a fix for the issue. But it might actually be a gwtorm issue,
where the setMaxRows gets called :-(


P.S.: Although the bug is of course valid nonetheless, searches for
"status:closed" have never been working IIRC, as they boil down to
using "OR". Previously they just gave the empty result set more often
than not :-(
Comment 7 christian 2014-02-07 12:54:07 UTC
Although it of course sucks, you can work around the issues, by resolving the
"OR" part of the query by hand:

Like instead of

  curl "https://gerrit.wikimedia.org/r/changes/?q=status:closed"

do

  curl "https://gerrit.wikimedia.org/r/changes/?q=status:merged"
  curl "https://gerrit.wikimedia.org/r/changes/?q=status:abandoned"

.
Comment 8 Gerrit Notification Bot 2014-02-08 11:40:06 UTC
Change 112201 had a related patch set uploaded by QChris:
Keep status:closed's limit below MySQL Connector/J's hard limit

https://gerrit.wikimedia.org/r/112201
Comment 9 Gerrit Notification Bot 2014-02-10 22:31:50 UTC
Change 112201 merged by QChris:
Keep status:closed's limit below MySQL Connector/J's hard limit

https://gerrit.wikimedia.org/r/112201
Comment 10 christian 2014-02-11 10:04:51 UTC
New, fixed gerrit has been deployed yesterday.
Comment 11 Bartosz Dziewoński 2014-02-11 12:18:24 UTC
Thank you!
Comment 12 MZMcBride 2014-02-12 04:03:06 UTC
\o/

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links