Last modified: 2014-11-19 21:47:00 UTC
Since I4e71671c8, WikibaseClient's OutputPageParserOutput & ParserAfterParse hook handlers call WikibaseClient::getDefaultInstance()->getLangLinkSiteGroup(). Since $wgWBClientSettings['languageLinkSiteGroup'] is unset, it defaults toWikibaseClient::getSite, which calls SiteSQLStore::getSites, which requires retrieving a huge object from memcached, with a predictable impact on the cluster: see <http://noc.wikimedia.org/~ori/SiteSQLStore.html> -- you can guess when I4e71671c8 was deployed.
Change 93648 had a related patch set uploaded by Ori.livneh: Set enwiki's languageLinkSiteGroup to 'wikipedia' https://gerrit.wikimedia.org/r/93648
Change 93648 merged by Ori.livneh: Set enwiki's languageLinkSiteGroup to 'wikipedia' https://gerrit.wikimedia.org/r/93648
Didn't improve things a whole lot, since the call to getSite in WikibaseClient.hooks.php's onSkinTemplateOutputPageBeforeExec hook handler is executed much more frequently.
(In reply to comment #3) > Didn't improve things a whole lot, since the call to getSite in > WikibaseClient.hooks.php's onSkinTemplateOutputPageBeforeExec hook handler is > executed much more frequently. That's Ie17f2af09, to be specific.
Change 93661 merged by jenkins-bot: Re-introduce siteGroup setting for performance reasons https://gerrit.wikimedia.org/r/93661
Change 93767 had a related patch set uploaded by Aude: Re-introduce siteGroup setting for performance reasons https://gerrit.wikimedia.org/r/93767
Change 93767 merged by jenkins-bot: Re-introduce siteGroup setting for performance reasons https://gerrit.wikimedia.org/r/93767
Change 93769 had a related patch set uploaded by Aude: Update Wikibase, use siteGroup setting instead of doing lookup https://gerrit.wikimedia.org/r/93769
Change 93772 had a related patch set uploaded by Aude: Update Wikibase, use siteGroup setting instead of doing lookup https://gerrit.wikimedia.org/r/93772
Change 93769 merged by jenkins-bot: Update Wikibase, use siteGroup setting instead of doing lookup https://gerrit.wikimedia.org/r/93769
Change 93772 merged by jenkins-bot: Update Wikibase, use siteGroup setting instead of doing lookup https://gerrit.wikimedia.org/r/93772
Change 93773 had a related patch set uploaded by Aude: Add siteGroup setting for Wikibase https://gerrit.wikimedia.org/r/93773
Change 93773 merged by Ori.livneh: Add siteGroup setting for Wikibase https://gerrit.wikimedia.org/r/93773
This is once again an issue; it is loading on every request. Impact: http://i.imgur.com/v9ebld6.png
This wasn't fixed, so I'm not sure why the bug was closed.
This causes: https://ganglia.wikimedia.org/latest/graph.php?r=year&z=xlarge&h=mc1005.eqiad.wmnet&m=network_report&s=by+name&mc=2&g=network_report&c=Memcached+eqiad https://ganglia.wikimedia.org/latest/graph.php?r=year&z=xlarge&h=mc1011.eqiad.wmnet&m=network_report&s=by+name&mc=2&g=network_report&c=Memcached+eqiad https://ganglia.wikimedia.org/latest/graph.php?r=year&z=xlarge&h=mc1014.eqiad.wmnet&m=cpu_report&s=by+name&mc=2&g=network_report&c=Memcached+eqiad Top keys are: enwiki:sites/SiteList#2014-03-17+Site:2013-01-23 (54MB/s) wikidatawiki:sites/SiteList#2014-03-17+Site:2013-01-23 (20MB/s) commonswiki:sites/SiteList#2014-03-17+Site:2013-01-23 (18MB/s) This has been pointed out as early as September 2013, and again March 2014 and again September 2014 and is still happening. Having e.g. 80% of mc1005's total network bandwidth being a single wikidata key, or SiteList keys being consistently on the top of memcached bandwidth output by multiple factors compared to the rest, is frankly indicative of a serious design failure and unacceptable. I don't understand why this bug was closed either.
Can we just have a current version hash stored in memcached and used to validate server-local CDB caches (made on the fly, with a special key holding the hash of the other key/values). This would reduce the memcached I/O to a minuscule amount.
Change 174113 had a related patch set uploaded by Aude: Lazy initialize OtherProjectsSidebarGenerator in hook handlers https://gerrit.wikimedia.org/r/174113
my patch (https://gerrit.wikimedia.org/r/#/c/174113/) ensures the memcached lookup of SiteList is confined to users with the other projects beta feature enabled. This should help quite a lot to reduce memcached access for the SiteList. the SiteList is used in similar functionality as the interwiki data, used to add links to related sister projects in the sidebar. to roll out the feature more widely, we should have local caching (json, like i18n?) of the site list data and may want to have memcached store the hash (like done for i18n), per Aaron's suggestion.
