Wikimedia Bugzilla is closed!

WFM, do you have any other context?

19:59 <       ori> i think this is the key: 
testwikidatawiki/WBL-1.24wmf21:WikiPageEntityRevisionLookup:Q33
19:59 <       ori> i'm going to delete it

test2 then came up, but test.wikipedia has this stack trace now:

Unexpected non-MediaWiki exception encountered, of type "UnexpectedValueException"
[758e9f98] /wiki/Main_Page Exception from line of : Error while unserializing
Backtrace:
#0 [internal function]: SplObjectStorage->unserialize(string)
#1 [internal function]: Memcached->getByKey(string, string, NULL, NULL)
#2 /srv/mediawiki/php-1.24wmf21/includes/objectcache/MemcachedPeclBagOStuff.php(124): Memcached->get(string, NULL, NULL)
#3 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/lib/includes/store/CachingEntityRevisionLookup.php(107): MemcachedPeclBagOStuff->get(string)
#4 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/lib/includes/store/CachingEntityRevisionLookup.php(146): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer)
#5 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/lib/includes/store/CachingEntityRevisionLookup.php(126): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision(Wikibase\DataModel\Entity\ItemId, integer)
#6 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/lib/includes/store/RevisionBasedEntityLookup.php(41): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId)
#7 [internal function]: Wikibase\Lib\Store\RevisionBasedEntityLookup->getEntity(Wikibase\DataModel\Entity\ItemId)
#8 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/lib/includes/store/EntityRedirectResolvingDecorator.php(79): call_user_func_array(array, array)
#9 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/lib/includes/store/RedirectResolvingEntityLookup.php(50): Wikibase\Lib\Store\EntityRedirectResolvingDecorator->__call(string, array)
#10 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/client/includes/ClientSiteLinkLookup.php(110): Wikibase\Lib\Store\RedirectResolvingEntityLookup->getEntity(Wikibase\DataModel\Entity\ItemId)
#11 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/client/includes/ClientSiteLinkLookup.php(82): Wikibase\Client\ClientSiteLinkLookup->getItem(Title)
#12 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/client/includes/hooks/LanguageLinkBadgeDisplay.php(84): Wikibase\Client\ClientSiteLinkLookup->getSiteLink(Title, string)
#13 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/client/includes/hooks/SidebarHookHandlers.php(262): Wikibase\Client\Hooks\LanguageLinkBadgeDisplay->assignBadges(Title, Title, array)
#14 /srv/mediawiki/php-1.24wmf21/extensions/Wikidata/extensions/Wikibase/client/includes/hooks/SidebarHookHandlers.php(141): Wikibase\Client\Hooks\SidebarHookHandlers->doSkinTemplateGetLanguageLink(array, Title, Title)
#15 /srv/mediawiki/php-1.24wmf21/includes/Hooks.php(206): Wikibase\Client\Hooks\SidebarHookHandlers::onSkinTemplateGetLanguageLink(array, Title, Title)
#16 /srv/mediawiki/php-1.24wmf21/includes/GlobalFunctions.php(3984): Hooks::run(string, array, NULL)
#17 /srv/mediawiki/php-1.24wmf21/includes/skins/SkinTemplate.php(204): wfRunHooks(string, array)
#18 /srv/mediawiki/php-1.24wmf21/includes/skins/SkinTemplate.php(512): SkinTemplate->getLanguages()
#19 /srv/mediawiki/php-1.24wmf21/includes/skins/SkinTemplate.php(274): SkinTemplate->prepareQuickTemplate()
#20 /srv/mediawiki/php-1.24wmf21/includes/OutputPage.php(2171): SkinTemplate->outputPage()
#21 /srv/mediawiki/php-1.24wmf21/includes/MediaWiki.php(594): OutputPage->output()
#22 /srv/mediawiki/php-1.24wmf21/includes/MediaWiki.php(435): MediaWiki->main()
#23 /srv/mediawiki/php-1.24wmf21/index.php(46): MediaWiki->run()
#24 /srv/mediawiki/w/index.php(3): include(string)
#25 {main}

The unserialize method for one of the objects included in the memcached array with key "testwikidatawiki/WBL-1.24wmf21:WikiPageEntityRevisionLookup:Q33" is broken, but I'm not sure which.

Created attachment 16448 [details]
Value of "testwikidatawiki/WBL-1.24wmf21:WikiPageEntityRevisionLookup:Q33" as raw binary data

Generated using memccat:

  memccat --servers=localhost:11211 "testwikidatawiki/WBL-1.24wmf21:WikiPageEntityRevisionLookup:Q33" --file=serialized_value.raw

Change 159934 had a related patch set uploaded by Ori.livneh:
Update Wikidata to tip of master for I23b7eb54b8e

https://gerrit.wikimedia.org/r/159934

Change 159934 merged by Ori.livneh:
Update Wikidata to tip of master for I23b7eb54b8e

https://gerrit.wikimedia.org/r/159934

Reducing importance/etc as this is no longer breaking production (test wikis). But we're in a reverted state, so keeping it higher than normal so we can get wikidata back on the 1.24wmf21 train (hopefully).

Plan is to revert to a known good state on Monday morning.
https://wikitech.wikimedia.org/wiki/Deployments#Week_of_September_15th

 14:13 logmsgbot: aude Started scap: Put test.wikidata back on mw1.24-wmf19 extension branch
 14:50 logmsgbot: aude Finished scap: Put test.wikidata back on mw1.24-wmf19 extension branch (duration: 37m 27s)
 14:54 logmsgbot: aude Synchronized wmf-config/Wikibase.php: Bump wikibase memcached key for test.wikidata, test, test2 (duration: 00m 16s)

I think we're good. Closing. If not, please re-open :)

I can easily reproduce the issues that we had on beta, and bumping the cache key we use for entities seems the solution.  I think the solution is to have the cache key changed (generated in our build) for beta and then deploy to beta again.

I did reproduce this error "Error at offset 0 of 6 bytes" from test2 on vagrant/hhvm on friday but can't reproduce it now.

My guess is maybe a race condition, someone hitting the main page of test2 causing incorrect data in memcached, or possibly an issue with hhvm.  I can try to reproduce some more. If I can't reproduce and beta is ok, then we try test.wikidata again (maybe next week?). If the issue happens again, I would try bumping the shared cache key again:

$wgWBRepoSettings['sharedCacheKeyPrefix'] = 'wikidatawiki/mw1.24-wmf20';

and

$wgWBClientSettings['sharedCacheKeyPrefix'] = 'wikidatawiki/mw1.24-wmf20';

anyway, we are good for now and no new code for wikidata this week. (except maybe a bug fix)

*** Bug 71461 has been marked as a duplicate of this bug. ***

Any reason to reopen this? Should be fixed

(In reply to Marius Hoch from comment #12)
> Any reason to reopen this? Should be fixed

See https://bugzilla.wikimedia.org/show_bug.cgi?id=71461