Last modified: 2013-07-12 07:43:10 UTC
http://ganglia.wmflabs.org/ used to list all the projects. It is now down to 17 and miss my favorite project 'deployment-prep'. It would be nice to have it fixed up to list all the labs project.
There's a crontab, run as root: /usr/local/sbin/generate-ganglia-conf.py When run manually it regenerates all ganglia configuration files. Restarting /etc/init.d/ganglia-monitor-aggrs re-adds the sources. The cron seems to not be running. More worrying, though, is that ganglia seems to just break itself every once in a while. I'm assuming that puppet is doing so.
Output from the cron job trying to run on aggregator1: ----- Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service ganglia-monitor restart Since the script you are attempting to invoke has been converted to an Upstart job, you may also use the restart(8) utility, e.g. restart ganglia-monitor /etc/init.d/ganglia-monitor: 73: start: not found Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service ganglia-monitor restart Since the script you are attempting to invoke has been converted to an Upstart job, you may also use the restart(8) utility, e.g. restart ganglia-monitor /etc/init.d/ganglia-monitor: 73: start: not found ----- In the meantime new members of the project can't get onto the instance because it can't create their home directories (likely it's using the gluster home dir and that's read-only).
Home directories were set to use nfs, for some reason, and there was an issue with the automount config for lucid. It was fixed by https://gerrit.wikimedia.org/r/#/c/72936/. The homedirs are now accessible.
Homedirs were set to nfs by me in an attempt to get them to write my homedir to something other than a read-only gluster fs (I guess it's ro so people move off ot it). We couldn't figure out a good way to test that gerrit patch, thanks for pushing that through.
We found that Default PATH for cron environment on Debian/Ubuntu does not contain /sbin as a valid path. Therefore we added /sbin as part of the PATH. Fixed with https://gerrit.wikimedia.org/r/#/c/73278/
Nice! Thank you everyone :-]
