Last modified: 2013-10-10 15:36:10 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T57498, the corresponding Phabricator task for complete and up-to-date bug report information.

Bug 55498 - tools-webserver-01 is down


Summary:	tools-webserver-01 is down

Status:	RESOLVED FIXED

Product:	Wikimedia Labs
Classification:	Unclassified
Component:	tools (Other open bugs)
Version:	unspecified
Hardware:	All All

Importance:	Immediate critical
Target Milestone:	---
Assigned To:	Marc A. Pelletier

URL:
Whiteboard:
Keywords:

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2013-10-09 06:02 UTC by bgwhite
Modified:	2013-10-10 15:36 UTC (History)
CC List:	6 users (show)

See Also:
Web browser:	---
Mobile Platform:	---
Assignee Huggle Beta Tester:	---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description bgwhite 2013-10-09 06:02:24 UTC

Showing a 500 Internal Server Error error on any page I've tried except the main page.  Went down around 10:40pm Pacific

Comment 1 Andre Klapper 2013-10-09 10:31:51 UTC

Thanks for taking the time to report this!

Which webserver? A URL to reproduce is welcome in bug reports. :)

Comment 2 metatron 2013-10-09 11:38:50 UTC

php-sites on tools-webserver retrun Error 500 i.e.

https://tools.wmflabs.org/catnap/

Comment 3 Addshore 2013-10-09 11:56:17 UTC

This error relates to tools-webserver-01

Comment 4 Addshore 2013-10-09 13:35:51 UTC

tools-webserver-01 seems to be fixed and is no longer constantly throwing a 500 error.

The other two webservers are still up. Only about 1/3 tools were affected.

Logs show there was a burst of activity with glamtools that lasted a couple of hours.

Comment 5 Andre Klapper 2013-10-10 09:25:50 UTC

If this was about http://tools.wmflabs.org/ then it was also brought up on http://lists.wikimedia.org/pipermail/wikitech-l/2013-October/072336.html

Comment 6 Addshore 2013-10-10 09:48:33 UTC

tools-webserver-01 is experiencing the same problems as above, reopening.

Comment 7 Marc A. Pelletier 2013-10-10 12:54:36 UTC

The current scheme is, annoyingly, vulnerable to single tools consuming every resources.  There is a new system in place that is more robust and more performing for all but the very simplest of tools:

https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb

While the system is currently opt-in, I would very much recommend that the heavier duty move to it as soon as possible; this will lighten the load on the apaches (alleviating the issue) and insulate those tools from the others.

Wikimedia Bugzilla is closed!

Search

Personal tools

Navigation

Links