Last modified: 2014-02-06 23:52:40 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T62970, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 60970 - HTTP 5xx Wikimedia Errors
HTTP 5xx Wikimedia Errors
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
wmf-deployment
All All
: Highest critical (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-02-06 16:38 UTC by PiRSquared17
Modified: 2014-02-06 23:52 UTC (History)
3 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
Error 5xx rate today, for the records (111.87 KB, image/png)
2014-02-06 23:52 UTC, Nemo
Details

Description PiRSquared17 2014-02-06 16:38:04 UTC
on https://meta.wikimedia.org/wiki/Main_Page
Comment 1 PiRSquared17 2014-02-06 16:38:22 UTC
also on https://en.wikipedia.org/wiki/Hildegund
Comment 2 Andre Klapper 2014-02-06 16:39:07 UTC
I don't see any error. In general, describing the error is a good first step for a bug report. :) See https://www.mediawiki.org/wiki/How_to_report_a_bug
Comment 3 Kunal Mehta (Legoktm) 2014-02-06 16:39:50 UTC
It's being investigated in -operations right now.
Comment 4 Andre Klapper 2014-02-06 16:41:25 UTC
Good spike in https://gdash.wikimedia.org/dashboards/reqerror/

Assuming this is about
"WIKIMEDIA FOUNDATION
Error
Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.",

Operations is looking at it in #wikimedia-operations on IRC.
Comment 5 Andre Klapper 2014-02-06 16:42:28 UTC
Also worth to watch https://wikitech.wikimedia.org/wiki/Server_admin_log
Comment 6 Andre Klapper 2014-02-06 17:05:59 UTC
Issue should be fixed now:

<Reedy> What caused it is fairly obvious (trying to run 3 versions of MW simultaneously). Why that was happening is a differnt matter
<Reedy> It should be fine now...
Comment 7 PiRSquared17 2014-02-06 19:38:57 UTC
Getting it again.
Comment 8 Zell Faze 2014-02-06 21:20:10 UTC
Appears to have been caused by some issue with the wmf12 deploy.  Everything has been reverted to wmf11.  The root cause of the issue was not identified.  Can anyone else add anymore details?
Comment 9 PiRSquared17 2014-02-06 23:23:59 UTC
(In reply to comment #8)
> Appears to have been caused by some issue with the wmf12 deploy.  Everything
> has been reverted to wmf11.  The root cause of the issue was not identified. 
> Can anyone else add anymore details?
Due to Math extension (texvc) apparently.
Comment 10 Nemo 2014-02-06 23:52:40 UTC
Created attachment 14515 [details]
Error 5xx rate today, for the records

Basically we had two partial outages lasting 10-20 minutes each on random pages served, if the 5xx error rate graph is to be trusted. I don't know if someone is able to calculate what portion of users/viewers was affected: 125k errors/s is three times the total pageviews/s and the same as total requests, as reported by https://gdash.wikimedia.org/dashboards/reqsum/

(In reply to comment #9)
> Due to Math extension (texvc) apparently.

http://lists.wikimedia.org/pipermail/wikitech-l/2014-February/074321.html
When some conditions piled up.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links