Last modified: 2013-09-30 08:18:04 UTC
deployment-memc0 and deployment-memc1 have 16GB of memory. They are being applied the role::memcached class which boot a single memcached instance having 90MB of memory. We end up with only 180MB of memcache space. ps -u nobody f PID TTY STAT TIME COMMAND 995 ? SLl 0:08 /usr/bin/memcached -m 89088 -p 11211 -u nobody -l 0.0.0.0 -c 25000 -k -n 5 -f 1.05 -D : -o slab_reassign
Created attachment 13199 [details] ganglia graph for memc0 showing 1 month of memcached memory usage Apparently memcached grows up by itself as needed: From top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1058 nobody 20 0 1397m 1.2g 2424 S 0 7.4 48:38.38 memcached Though according to the ganglia graph, it is limited at 1GB of memory.
Created attachment 13200 [details] stats output for memc0 Maximum cache size: STAT limit_maxbytes 93415538688 ~ 93G Currently used cache: STAT bytes 1019756272 ~ 1G Apparently it is not memory limited. STAT evictions 0 Stats documentation is in the protocol definition: https://github.com/memcached/memcached/blob/master/doc/protocol.txt So apparently memcached memory usages grows up and maybe we dont need more memory ?
So the real issue is that memcached is passed -m 89088, which would let the instances allocate roughly 89GB of memory when they only have 16GB. Should limit to 15GB.
Change 81905 had a related patch set uploaded by Hashar: tweak memcached limit on beta (89GB -> 15GB) https://gerrit.wikimedia.org/r/81905
Change 81905 merged by Faidon Liambotis: tweak memcached limit on beta (89GB -> 15GB) https://gerrit.wikimedia.org/r/81905
I have restarted memcached on deployment-memc{0,1} ps shows: /usr/bin/memcached -m 15000 .. Fixed!
