Last modified: 2014-03-23 06:05:02 UTC
During the NFS outage, the qacct transfer jobs pestered the roots' mailboxes every five minutes. Though such an outage of course will never ever happen again :-), it sucked nonetheless. The transfer job is a service and if we would monitor it as one, we would get better behaviour as well: A nice green or red icon on a web dashboard, and only one (or none?) ping by mail when the status *changes*. So we should set up Icinga monitoring for that: a) The transfer job directs all stdout/stderr to a file, saves its exit code in another and periodically these files are queried by Icinga. b) The transfer job passes its output and exit code directly to an Icinga sentinel that passes it somewhere up the chain. Whether a) or b) are preferable (or possible for that matter), I haven't figured out yet, but this bug will track the progress on that.
http://blog.endpoint.com/2012/04/monitoring-cronjob-exit-codes-with.html has an example how to monitor cron jobs.
The qacct transfer cron job has been replaced with symlinks that make copying the accounting file unnecessary (cf. Gerrit change #114950 and Gerrit change #118120).
