Last modified: 2013-10-23 20:14:55 UTC
We use 'redis' for wgMainCacheType. That means there are several places where data is stored to the Redis cache without any expiration, using either wgMemc or CACHE_ANYTHING. These include: ResourceLoader - https://git.wikimedia.org/blob/mediawiki%2Fcore.git/d987f43b24a24a6f3fcd5886fedaf483fa3c6891/includes%2Fresourceloader%2FResourceLoader.php#L180 User cache - https://git.wikimedia.org/blob/mediawiki%2Fcore.git/d987f43b24a24a6f3fcd5886fedaf483fa3c6891/includes%2FUser.php#L368 The size of the Vagrant Redis database is rather low (16 MB), even considering the default size of the box is 768 MB. Since Redis is also set to persist, the 16 MB can fill up. Redis allows multiple policies for expiration (https://raw.github.com/antirez/redis/2.6/redis.conf). We don't override this currently, so it uses the default of 'volatile-lru'. This means, when max memory is hit, it will try to pick a key with an expiration using an approximated least recently used algorithm (this is further configurable with maxmemory-samples). The problem is that as stated above, a lot of keys don't have an expiration set. My Redis database actually says none do currently, which is probably because it has been at maxmemory for a while and thus has evicted all the ones that could expire. I'm not sure the ideal solution, and it would be helpful for someone more familiar with MW's caching to weigh in. However, so far it seems like I think increasing it to 32 MB (still less than 5% of the overall RAM) and using allkeys-lru (remove any key, using least recently used) should be an improvement (at least then it won't just stay filled and block all writes). However, there might be a better way that makes it less likely to evict important stuff like ongoing sessions (MediaWiki-Vagrant also uses Redis for sessions). Production uses 'memcached-pecl' for the main cache. I'm not sure exactly how that expiration is configured, but prod doesn't currently deal with Redis here.
Basically, with the current setup it can degenerate into a full (at the memory limit) db where all the keys are non-expiring, and thus nothing can be evicted or written.
The commit is https://gerrit.wikimedia.org/r/#/c/91127/ .
Change 91127 had a related patch set uploaded by Legoktm: redis: 16M -> 64M, volatile-lru -> allkeys-lru https://gerrit.wikimedia.org/r/91127
Change 91127 merged by jenkins-bot: redis: 16M -> 64M, volatile-lru -> allkeys-lru https://gerrit.wikimedia.org/r/91127
