Last modified: 2014-06-06 00:29:05 UTC
Trace from osmium: Warning: could not uncompress value in /usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/MemcachedPeclBagOStuff.php on line 124 #0 Memcached->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/MemcachedPeclBagOStuff.php:124] #1 MemcachedPeclBagOStuff->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:126] #2 BagOStuff->mergeViaCas(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:111] #3 BagOStuff->merge(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/jobqueue/JobQueueGroup.php:408] #4 JobQueueGroup->executeReadyPeriodicTasks(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:76] #5 RunJobs->execute(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/doMaintenance.php:109] #6 include(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:281] #7 include(), called at [/usr/local/apache/common-local/multiversion/MWScript.php:97] #0 Memcached->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/MemcachedPeclBagOStuff.php:124]\n #1 MemcachedPeclBagOStuff->get(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:126]\n #2 BagOStuff->mergeViaCas(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/objectcache/BagOStuff.php:111]\n #3 BagOStuff->merge(), called at [/usr/local/apache/common-local/php-1.24wmf6/includes/jobqueue/JobQueueGroup.php:408]\n #4 JobQueueGroup->executeReadyPeriodicTasks(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:76]\n #5 RunJobs->execute(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/doMaintenance.php:109]\n #6 include(), called at [/usr/local/apache/common-local/php-1.24wmf6/maintenance/runJobs.php:281]\n #7 include(), called at [/usr/local/apache/common-local/multiversion/MWScript.php:97]
I ran the job loop with tcpdump running in the background to capture memcached gets. Here's a key that triggers the warning: srwiki:jobqueuegroup:taskruns:v1
Minimal testcase: $memc = new Memcached; $memc->addServer('localhost', 11211); $memc->get('srwiki:jobqueuegroup:taskruns:v1'); I saved the raw key via memccat to osmium:/tmp/srwiki_jobqueuegroup_taskruns_v1.raw Is it using igbinary serialization? I ran out of time.
the flags field is 84, which means MEMC_VAL_COMPRESSED and MEMC_VAL_IS_SERIALIZED
When setting the same value in memcached under HHVM, the resultant key has a flags value of 20 (vs. PHP's 84). I think the figures represent: MEMC_VAL_COMPRESSED | MEMC_VAL_COMPRESSION_FASTLZ | (MEMC_VAL_IS_SERIALIZED & MEMC_VAL_TYPE_MASK) === 84 MEMC_VAL_COMPRESSED | (MEMC_VAL_IS_SERIALIZED & MEMC_VAL_TYPE_MASK) === 20 So this fingers fastlz compression as the culprit, I think.
HHVM doesn't support fastlz compression. Switching PHP's compression to zlib and re-saving the key still doesn't result in a value HHVM can read. Under HHVM, ZLIB's uncompress() is returning Z_DATA_ERROR, indicating that the input data is corrupted.
I ported "new-style" zlib/fastlz decompression to HHVM, adapting https://github.com/php-memcached-dev/php-memcached/blob/master/php_memcached.c#L3288 into https://github.com/wikimedia/operations-software-hhvm-dev/commit/5b32fd599e8ce76bfe3d13723fd9d66c6e22dccf Will submit a PR upstream
Submitted: https://github.com/atdt/hhvm/commit/3e97781e6c2e3b52c8647957909ec9308d09dfe6
