Last modified: 2014-10-16 13:00:19 UTC
http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=cvn&style=detail One of my instances was outdated for 2 days because of a stuck lock file that got left behind when the instance rebooting during a scheduled puppet run. Took a while to figure out why that one instance wasn't mounting the new /srv. [07:37 UTC] $ sudo puppet agent -t Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) [07:37 UTC] $ l /var/lib/puppet/state/agent_catalog_run.lock -rw-r--r-- 1 root root 0 Jun 12 01:53 agent_catalog_run.lock syslog Jun 13 07:23:01 cvn-app5 CRON[27536]: (root) CMD (timeout -k 300 1800 puppet agent --onetime --verbose --no-daemonize --splay --splaylimit 60 --show_diff >> /var/log/puppet.log 2>&1) Jun 13 07:23:02 cvn-app5 puppet-agent[27538]: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) syslog.2.gz Jun 12 06:23:01 cvn-app5 CRON[5136]: (root) CMD (timeout -k 300 1800 puppet agent --onetime --verbose --no-daemonize --splay --splaylimit 60 --show_diff >> /var/log/puppet.log 2>&1) Jun 12 06:23:03 cvn-app5 puppet-agent[5138]: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) and going back further: Jun 12 01:53:01 cvn-app5 CRON[12568]: (root) CMD (timeout -k 300 1800 puppet agent --onetime --verbose --no-daemonize --splay --splaylimit 60 --show_diff >> /var/log/puppet.log 2>&1) Jun 12 01:53:02 cvn-app5 puppet-agent[12570]: Sleeping for 39 seconds (splay is enabled) Jun 12 01:53:41 cvn-app5 puppet-agent[12570]: Retrieving plugin Jun 12 01:53:42 cvn-app5 puppet-agent[12570]: Loading facts in /var/lib/puppet/lib/facter/puppet_config_dir.rb.... Jun 12 01:55:02 cvn-app5 kernel: imklog 5.8.6, log source = /proc/kmsg started. Jun 12 01:55:02 cvn-app5 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="807" x-info="http://www.rsyslog.com"] start Jun 12 01:55:02 cvn-app5 rsyslogd: rsyslogd's groupid changed to 103 Jun 12 01:55:02 cvn-app5 rsyslogd: rsyslogd's userid changed to 101 Jun 12 01:55:02 cvn-app5 rsyslogd-2039: Could not open output pipe '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ] Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Initializing cgroup subsys cpuset Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Initializing cgroup subsys cpu Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Linux version 3.2.0-59-virtual (buildd@toyol) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #90-Ubuntu SMP Tue Jan 7 23:02:33 UTC 2014 (Ubuntu 3.2.0-59.90-virtual 3.2.54) Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Command line: root=UUID=3cbf0c17-65ab-4d22-a2f1-69d1697cf8d0 ro quiet splash console=ttyS0 Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] KERNEL supported cpus:
(Not strictly relevant for "Icinga" (as in: alerting without user action), but you can see the Puppet status of your instances under [[wikitech:Special:NovaInstance]]. AFAIK you can't query for those with SMW, though.)
