Last modified: 2014-11-18 14:56:47 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T75555, the corresponding Phabricator task for complete and up-to-date bug report information.

Bug 73555 - [OPS] db1020.eqiad.wmnet is down (impacts Gerrit, OTRS, EventLogging)


Summary:	[OPS] db1020.eqiad.wmnet is down (impacts Gerrit, OTRS, EventLogging)

Status:	RESOLVED FIXED

Product:	Wikimedia
Classification:	Unclassified
Component:	Git/Gerrit (Other open bugs)
Version:	wmf-deployment
Hardware:	All All

Importance:	High blocker (vote)
Target Milestone:	---
Assigned To:	Nobody - You can work on this!

URL:
Whiteboard:
Keywords:	ops

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2014-11-18 14:44 UTC by Antoine "hashar" Musso (WMF)
Modified:	2014-11-18 14:56 UTC (History)
CC List:	6 users (show)

See Also:
Web browser:	---
Mobile Platform:	---
Assignee Huggle Beta Tester:	---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Antoine "hashar" Musso (WMF) 2014-11-18 14:44:06 UTC

Guice provision errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error


Interactions with git over port 29418 is fine though.

Comment 1 Antoine "hashar" Musso (WMF) 2014-11-18 14:48:40 UTC

According to puppet, Gerrit uses the DB 'm2-master.eqiad.wmnet'

From site.pp:

  ## m2 shard
  node /^db10(20)\.eqiad\.wmnet/ {

Mysqld died:

mysqld processes on db1020 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld  


The DB is also used by EventLogging and OTRS.

Comment 2 Antoine "hashar" Musso (WMF) 2014-11-18 14:56:09 UTC

<springle>	 !log fail over m2 to m2-slave (db1046); investigating db1020

Comment 3 Antoine "hashar" Musso (WMF) 2014-11-18 14:56:47 UTC

Gerrit is back, using the other mysql server now. Sean Pringle is investigating db1020 failure.  Not much help to do on this bug ticket.

Wikimedia Bugzilla is closed!

Search

Personal tools

Navigation

Links