Last modified: 2013-11-08 14:55:17 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T58545, the corresponding Phabricator task for complete and up-to-date bug report information.

Bug 56545 - purges failing in pmtpa+esams?


Summary:	purges failing in pmtpa+esams?

Status:	RESOLVED FIXED

Product:	Wikimedia
Classification:	Unclassified
Component:	General/Unknown (Other open bugs)
Version:	wmf-deployment
Hardware:	All All

Importance:	High major (vote)
Target Milestone:	---
Assigned To:	Nobody - You can work on this!

URL:
Whiteboard:
Keywords:	ops

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2013-11-04 01:05 UTC by jeremyb
Modified:	2013-11-08 14:55 UTC (History)
CC List:	14 users (show)

See Also:
Web browser:	---
Mobile Platform:	---
Assignee Huggle Beta Tester:	---

Attachments
Ganglia data: sporadic HTCP delivery to esams (83.99 KB, image/png) 2013-11-04 02:53 UTC, Tim Starling	Details
Add an attachment (proposed patch, testcase, etc.)

Description jeremyb 2013-11-04 01:05:11 UTC

forking to new bug

(In reply to bug 54647 comment #35)
> Hi,
> it just happened again. To be able to see the updated version in IE, I had to
> use Firefox and do a hard reload. Only then IE showed the recent version.
> It's
> like before, with emptied caches and so on.

(In reply to bug 54647 comment #36)
> Here are two examples:
> https://de.wikipedia.org/wiki/Massenmedien
> and:
> https://de.wikipedia.org/wiki/Massendrucksache
> Edited as user (being logged in) with Opera. Afterwards viewed with other
> browsers, namely Opera, IE and Firefox, and not logged in.
> The problem is back. Only I don't know the reason. What do you think?
> Greetings,
> Volker

Comment 1 jeremyb 2013-11-04 01:05:26 UTC

$ egrep -e '^< Last-Modified: ' bug54647-2013-11-04-3/*_Massenmedien.* | perl -lpe 's|^[^/]+/([a-z]+\..*?)\.\d+:|$1:|;' | sort | uniq -c | sort -nr
      5 ulsfo._wiki_Massenmedien:< Last-Modified: Sun, 03 Nov 2013 23:25:45 GMT
      5 pmtpa._wiki_Massenmedien:< Last-Modified: Fri, 01 Nov 2013 10:30:13 GMT
      5 esams._wiki_Massenmedien:< Last-Modified: Fri, 01 Nov 2013 10:30:13 GMT
      5 eqiad._wiki_Massenmedien:< Last-Modified: Sun, 03 Nov 2013 23:25:45 GMT

Comment 2 jeremyb 2013-11-04 01:27:01 UTC

Confirmed with a 2nd, arbitrary page pulled out of recent changes.

[[de:Balduin Baas]]

Comment 3 Bryan Davis 2013-11-04 02:07:19 UTC

I never heard the root cause of the bug this was forked from. The proximal cause was a failure for the HTCP UDP packets to be relayed from equiad to esams. Looking at the ganglia graphs makes me think a similar event is occurring: https://ganglia.wikimedia.org/latest/?r=custom&cs=11%2F03%2F2013+22%3A00+&ce=11%2F04%2F2013+02%3A00+&m=vhtcpd_inpkts_recvd&s=descending&c=Text+caches+esams&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4

Comment 4 Tim Starling 2013-11-04 02:52:22 UTC

HTCP delivery to esams relies on multicast routing from eqiad apaches to the udpmcast.py in pmtpa. This eqiad-pmtpa multicast service appears to be flapping.

Comment 5 Tim Starling 2013-11-04 02:53:50 UTC

Created attachment 13682 [details]
Ganglia data: sporadic HTCP delivery to esams

Comment 6 Tim Starling 2013-11-04 03:17:54 UTC

This should be fixed now. I moved htcpmcast to eqiad to work around the pmtpa network issue.

Comment 7 Andre Klapper 2013-11-08 14:55:17 UTC

Closing as per comment 6. Please reopen if this is still a problem.

Wikimedia Bugzilla is closed!

Search

Personal tools

Navigation

Links