Last modified: 2014-07-07 20:16:57 UTC
Running /usr/local/bin/jobs-loop.sh on osmium with WIKIDEBUG=1 floods the output buffer with the following error message: Notice: Unable to unserialize: [ERR value is not an integer or out of range]. Expected ':' but got 'R'. in /usr/local/apache/common-local/php-1.24wmf4/includes/jobqueue/JobQueueRedis.php on line 798 Quick Googling shows that "ERR value is not an integer or out of range" is coming from Redis itself. Note that HHVM uses a pure-PHP redis client library that is supposed to be compatible with php-redis (but may not be). The implementation is here: https://github.com/facebook/hhvm/blob/master/hphp/system/php/redis/Redis.php
strace shows: sendto(9, "*9\r\n$7\r\nEVALSHA\r\n$40\r\n46eccb844b00c898d68cb06502b13a906ff3d6a8\r\n$46\r\nfrwiktionary:jobqueue:refreshLinks:l-unclaimed\r\n$45\r\nfrwiktionary:jobqueue:refreshLinks:h-sha1ById\r\n$45\r\nfrwiktionary:jobqueue:refreshLinks:h-idBySha1\r\n$44\r\nfrwiktionary:jobqueue:refreshLinks:z-claimed\r\n$45\r\nfrwiktionary:jobqueue:refreshLinks:h-attempts\r\n$41\r\nfrwiktionary:jobqueue:refreshLinks:h-data\r\n$10\r\n1400461767\r\n", 389, 0, NULL, 0) = 389 recvfrom(9, "-ERR value is not an integer or out of range\r\n", 8192, MSG_DONTWAIT, NULL, NULL) = 46 If an ERR response is received, Redis::evalSha() should return false and the last error set: <https://github.com/nicolasff/phpredis/blob/b9a16b5ad5d8c80eac53f530f37e89761ec2c720/library.c#L1758> Obviously this is not being done; the error string is being returned from evalSha() as if it were the script result. JobQueueRedis::doPop() then goes into an infinite loop. As for the actual server error: comparison with a valid request from phpredis shows that the "numkeys" parameter is missing, so the first script parameter, in this case "frwiktionary:jobqueue:refreshLinks:l-unclaimed", is interpreted as the number of keys.
This is https://github.com/facebook/hhvm/pull/2555 (a fixed bug).
Rather, the missing numkeys was #2555, the error handling issue was fixed in https://github.com/facebook/hhvm/pull/2603
I saw lots of: 2014-06-03 19:12:08 osmium plwiki: Could not unserialize ParsoidCacheUpdateJobOnDependencyChange job. today...is that a similar issue?
There was another bug in the Redis client which also caused failures to unserialize: https://github.com/facebook/hhvm/pull/2874
(In reply to Aaron Schulz from comment #4) > I saw lots of: > > 2014-06-03 19:12:08 osmium plwiki: Could not unserialize > ParsoidCacheUpdateJobOnDependencyChange job. > > today...is that a similar issue? Similar but not identical. I submitted a fix here: https://github.com/facebook/hhvm/pull/3127
