Last modified: 2014-08-25 16:19:50 UTC
For some reason the hhvm process on deployment-mediawiki02 ends up hitting memcached server on port 11211 instead of 11212 (used by nutcracker). I haven't found the root cause for this.
The MediaWiki configuration cache is in /tmp/mw-cache-master which might end up having null size configuration files whenever / is filled up. Might cause some side effect causing MediaWiki to use some obsolete configuration or the wrong settings.
From mediawiki-config/wmf-config/mc-labs.php: $wgObjectCaches['memcached-pecl']['servers'] => array( '127.0.0.1:11212' ) Was this maybe caused by the full disk and empty config cache problem that was noticed on the same day?
Yeah I though it was related to the /tmp/mw-cache-master being dirty (0 bytes files). But the issue is still showing up in the memcached-serious log bucket: deployment-bastion:/data/project/logs$ tail -f memcached-serious.log 2014-08-25 16:17:15 deployment-mediawiki02 enwiki: Memcached error: Error connecting to 127.0.0.1:11211: Connection refused 2014-08-25 16:17:15 deployment-mediawiki02 enwiki: Memcached error: Error connecting to 127.0.0.1:11211: Connection refused 2014-08-25 16:17:15 deployment-mediawiki02 enwiki: Memcached error: Error connecting to 127.0.0.1:11211: Connection refused Seems our configuration somehow fallback to MediaWiki build-in default: includes/DefaultSettings.php:$wgMemCachedServers = array( '127.0.0.1:11211' ); On bastion, I looked at the object cache configuration with: $ mwscript eval.php --wiki=enwiki > var_dump( $wgObjectCaches ); All seems fine. mwscript does not exist on mediawiki02 though, could not check there.
