Last modified: 2014-04-20 14:38:55 UTC
Traceback (most recent call last): File "../toredis.py", line 41, in <module> method(channel, data) File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1063, in publish return self.execute_command('PUBLISH', channel, message) File "/usr/lib/python2.7/dist-packages/redis/client.py", line 256, in execute_command connection.send_command(*args) File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 197, in send_command self.send_packed_command(self.pack_command(*args)) File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 180, in send_packed_command self.connect() File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 127, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting tools-redis:6379. Connection refused. Also reported by legoktm in the SAL @ https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL 08:59 legoktm: grrrit-wm: 2014-04-20T08:28:15.889Z - error: Caught error in redisClient.brpop: Redis connection to tools-redis:6379 failed - connect ECONNREFUSED
The /var partition was full due to Redis using 1.6 GByte in /var/lib/redis. After verifying that using role::labs::lvm::mnt wouldn't cause any existing partition to be overwritten, I tried to use it with "$lvm_mount_point = /var/lib/redis", but in modules/labs_lvm/manifests/volume.pp this defines a file resource with "ensure => directory" that collides with a file resource with more detailed parameters in modules/redis/manifests/init.pp. I /think/ that volume.pp's file resource should be replaced with a "resource collector" ("User <| title == 'luke' |>"). Once that is done, we should consider defining the partitions in tools::redis so they don't get lost and can be smaller than the whole 68 GBytes that are allocated at the moment.
I should have added what /worked/ :-): Setting $lvm_mount_point to "/var/lib", running Puppet (leaves an empty /var/lib), "umount /var/lib && mv -i /var/lib /var/lib2 && puppetd -tv && mv -i /var/lib2/* /var/lib/ && rmdir /var/lib2" and a reboot for good measure.
