Last modified: 2012-07-29 19:17:29 UTC
There seems to be an issue with the LDAP backend: root@deployment-apache32:~# puppetd -tv info: Loading facts in /var/lib/puppet/lib/facter/apt.rb info: Loading facts in /var/lib/puppet/lib/facter/projectgid.rb info: Loading facts in /var/lib/puppet/lib/facter/default_gateway.rb err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed when searching for node i-0000031a.pmtpa.wmflabs: LDAP Search failed warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run
OpenDJ was running but did not listen on 389. Nothing was in the logs (besides the usual virt1 replication errors that kept going). I restarted OpenDJ and everything seems to be working again. Assigning to Ryan who knows OpenDJ better, maybe he can find the root cause.
Works for me now :)
We had a Nagios alert mentioning virt0 did not listen to LDAP.
Hm. It's possible the iptables NAT rules were somehow removed. Restarting will clear them and re-add them. I can't really check now. We should check for that if it happens again.
still getting errors and can't successfully create a new instance. 12:32 <+nagios-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds 12:45 <+nagios-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.023 seconds Jul 29 13:43:07 i-00000373 puppet-agent[3935]: Starting Puppet client version 2.7.11 Jul 29 13:43:07 i-00000373 puppet-agent[3935]: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed when searching for node i-00000373.pmtpa.wmflabs: LDAP Search failed Jul 29 13:43:07 i-00000373 puppet-agent[3935]: Using cached catalog Jul 29 13:43:07 i-00000373 puppet-agent[3935]: Could not retrieve catalog; skipping run and a bunch of dhclient stuff, repeatedly retrying (don't know if it's normal, doubt it)
Seems it happened again. I'm going to investigate the server failure and why the clients failed to failover properly.
