Last modified: 2014-06-10 02:01:29 UTC
Created attachment 12255 [details] Screenshot of error page Location: https://gerrit.wikimedia.org/r/#/c/62207/ ------- HTTP ERROR: 500 Problem accessing /r/. Reason: Guice provision errors: 1) Cannot open ReviewDb at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:71) while locating com.google.gerrit.reviewdb.server.ReviewDb 1 error Powered by Jetty:// -------
Looks like this has escalated. Most users don't even get an error, it just times out. Gerrit has become completely unresponsive.
Seems to affect SSH too.
Gerrit was restared, https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log&diff=68980&oldid=68979. Seems to be working now.
Reopening since restarting isn't a solution. Will take a look at the logs.
In case this becomes relevant at some point of investigating the root cause: ~2 hours before this happened MatmaRex told me that gerrit was really slow to respond. Not having access to logs, I checked ganglia, and the graphs showed the usual increase around that time of the day. The now reported error message surfaces if gerrit could connect to the database at startup but now cannot open new connections to the database. Do the logs show a root cause?
Chad: Do the logs show a root cause?
Here's the full stacktraces from the failure: http://p.defau.lt/?B4I7N8sdqihW2Dj7tQOUNw
(In reply to comment #3) > Gerrit was restared, Seems to be working now. Lowering priority.
Seen again with exactly the same error message, on https://gerrit.wikimedia.org/r/#/c/63824/ .
Created attachment 12371 [details] Proxy error, usually accompanies the previously captured Guice error The problem is back.
Restart of Gerrit restored functionality for now.
The following error appeared today (a few minutes ago): Guice provision errors: 1) Cannot open ReviewDb at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:71) while locating com.google.gerrit.reviewdb.server.ReviewDb 1 error
[Lowering priority; no recent incidents.]
And its back coming to eat more innocent souls: Guice provision errors: 1) Cannot open ReviewDb at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) while locating com.google.gerrit.reviewdb.server.ReviewDb 1 error
There's no one bug here so I'm not sure what this is supposed to accomplish. "Don't break the database connection with bad config, networking hiccups or otherwise crash the ability of Gerrit to talk to its DB?"
