Last modified: 2013-04-17 20:05:43 UTC
Created attachment 12108 [details] puppetd -tv log when applying puppetmaster::self I have applied the puppetmaster::self class on deployment-cache-upload-test7 . Turns out it fails to generate some certificate which cause the puppet client to not be able to access the local puppetmaster. err: /Stage[main]/Puppetmaster::Ssl/Exec[generate hostcert]/returns: change from notrun to 0 failed: /usr/bin/puppet cert generate deployment-cache-upload-test7.pmtpa.wmflabs returned 23 instead of one of [0] at /etc/puppet/manifests/puppetmaster.pp:54 see attached the puppetd -tv log Then when trying to run puppet we get: root@deployment-cache-upload-test7:~# puppetd -tv info: Creating a new SSL key for deployment-cache-upload-test7.pmtpa.wmflabs info: Creating a new SSL certificate request for deployment-cache-upload-test7.pmtpa.wmflabs info: Certificate Request fingerprint (md5): D7:16:67:1F:B9:C6:8F:A0:75:32:42:71:EA:CF:6B:0C Exiting; no certificate found and waitforcert is disabled root@deployment-cache-upload-test7:~#
Confirmed! And, it's not a race... I tried several different attempts with different timing, and it fails every time.
err: /Stage[main]/Puppetmaster::Ssl/Exec[generate hostcert]/returns: change from notrun to 0 failed: /usr/bin/puppet cert generate deployment-cache-upload-test7.pmtpa.wmflabs returned 23 instead of one of [0] at /etc/puppet/manifests/puppetmaster.pp:54 That is where I would love puppet to log the stderr ...
strace -estat puppetd -tv gives some hints: stat("/var/lib/puppet/server/ssl/public_keys/deployment-cache-upload-test7.pmtpa.wmflabs.pem", {st_mode=S_IFREG|0644, st_size=251, ...}) = 0 stat("/var/lib/puppet/server/ssl/public_keys/deployment-cache-upload-test7.pmtpa.wmflabs.pem", {st_mode=S_IFREG|0644, st_size=251, ...}) = 0 stat("/var/lib/puppet/server/ssl/private_keys/deployment-cache-upload-test7.pmtpa.wmflabs.pem", {st_mode=S_IFREG|0600, st_size=887, ...}) = 0 stat("/var/lib/puppet/server/ssl/private/password", 0x7fff78f319b0) = -1 ENOENT (No such file or directory) stat("/var/lib/puppet/server/ssl/certs/ca.pem", {st_mode=S_IFREG|0644, st_size=818, ...}) = 0 stat("/var/lib/puppet/server/ssl/certs/deployment-cache-upload-test7.pmtpa.wmflabs.pem", 0x7fff78f31fe0) = -1 ENOENT (No such file or directory) stat("/var/lib/puppet/server/ssl/certs", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 stat("/var/lib/puppet/server/ssl/certs/deployment-cache-upload-test7.pmtpa.wmflabs.pem", 0x7fff78f2f110) = -1 ENOENT (No such file or directory) stat("/usr/lib/ssl/certs/03b4993c.0", 0x7fff78f2d760) = -1 ENOENT (No such file or directory) stat("/var/lib/puppet/server/ssl/certificate_requests/deployment-cache-upload-test7.pmtpa.wmflabs.pem", {st_mode=S_IFREG|0640, st_size=582, ...}) = 0 stat("/var/lib/puppet/server/ssl/certs/ca.pem", {st_mode=S_IFREG|0644, st_size=818, ...}) = 0 stat("/var/lib/puppet/server/ssl/certs/deployment-cache-upload-test7.pmtpa.wmflabs.pem", 0x7fff78f31220) = -1 ENOENT (No such file or directory) stat("/var/lib/puppet/server/ssl/certs", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 stat("/var/lib/puppet/server/ssl/certs/deployment-cache-upload-test7.pmtpa.wmflabs.pem", 0x7fff78f2e350) = -1 ENOENT (No such file or directory) stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=254, ...}) = 0 stat("/usr/lib/ssl/certs/03b4993c.0", 0x7fff78f2c9a0) = -1 ENOENT (No such file or directory) stat("/var/lib/puppet/server/ssl/certs/ca.pem", {st_mode=S_IFREG|0644, st_size=818, ...}) = 0 stat("/var/lib/puppet/server/ssl/certs/deployment-cache-upload-test7.pmtpa.wmflabs.pem", 0x7fff78f31fe0) = -1 ENOENT (No such file or directory) stat("/var/lib/puppet/server/ssl/certs", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 stat("/var/lib/puppet/server/ssl/certs/deployment-cache-upload-test7.pmtpa.wmflabs.pem", 0x7fff78f2f110) = -1 ENOENT (No such file or directory) stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=254, ...}) = 0 stat("/usr/lib/ssl/certs/03b4993c.0", 0x7fff78f2d760) = -1 ENOENT (No such file or directory) Most probably a cert is missing somewhere :( It looks for /usr/lib/ssl/certs/03b4993c.0 but in that dir (ordered by time) I do not have that cert: -rw-r--r-- 1 root root 1017 Apr 7 03:48 ssl-cert-snakeoil.pem lrwxrwxrwx 1 root root 21 Apr 7 03:48 736caeca -> ssl-cert-snakeoil.pem -r--r--r-- 1 root root 1525 Apr 15 15:15 wmf-ca.pem lrwxrwxrwx 1 root root 25 Apr 15 15:15 dda55890.0 -> /etc/ssl/certs/wmf-ca.pem -r--r--r-- 1 root root 1151 Apr 15 15:15 wmf-labs.pem lrwxrwxrwx 1 root root 27 Apr 15 15:15 9779bdc4.0 -> /etc/ssl/certs/wmf-labs.pem Maybe some hashing algorithm has been changed.
The certificate is not being generated properly. For a while this was due to puppet dependencies not being specified properly. I spend a ton of time trying to figure out why today, only to realize that virt0's /etc/puppet files weren't being updated when I pushed, which threw me off of the trail for a while. I should have this working again tomorrow. Will keep you updated.
Phew, ok this should be fixed. I had actually fixed it yesterday (my test instance was a little whacky from all the poking yesterday). New instances use puppetmaster::self just fine. Let me know if you have more problems. I'm going to close this bug.
Ah thanks for fixing the virt0 /etc/puppet fetching :-] Cause me headhaches yesterday. I have deleted my instance deployment-cache-upload-test7 created a new one deployment-cache-upload-test8 and will see what happens there. Thanks for the fix :-}
I have created yet another instance: deployment-cache-upload-test9 debug: Finishing transaction 69846343294080 debug: Using cached certificate for ca err: Could not request certificate: getaddrinfo: Name or service not known Exiting; failed to retrieve certificate and waitforcert is disabled I guess that is related so reopening :-D
On deployment-cache-upload-test9 /etc/puppet/puppet.conf : [main] logdir=/var/log/puppet vardir=/var/lib/puppet ssldir=/var/lib/puppet/ssl rundir=/var/run/puppet factpath=$vardir/lib/facter templatedir=$confdir/templates prerun_command=/etc/puppet/etckeeper-commit-pre postrun_command=/etc/puppet/etckeeper-commit-post [master] # These are needed when the puppetmaster is run by passenger # and can safely be removed if webrick is used. ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY I am not sure why it has a [master] section, that looks wrong since that instance does not have any class applied. On the instance deployment-cache-mobile01 I got: # This file is managed by Puppet! [main] logdir = /var/log/puppet vardir = /var/lib/puppet ssldir = /var/lib/puppet/ssl rundir = /var/run/puppet factpath = $vardir/lib/facter [agent] server = virt0.wikimedia.org certname = i-0000059a.pmtpa.wmflabs configtimeout = 960 splay = true prerun_command = /etc/puppet/etckeeper-commit-pre postrun_command = /etc/puppet/etckeeper-commit-post pluginsync = false report = false So I guess something is screwed up in puppet for labs :/
Hmm, this is just a new instance creation, right? Are you using role::puppet::self or puppetmaster::self stuff right now? I had some weird problems with brand new instances too.
That is a brand new instance using the Precise image :(
Comment #7 to Comment #10 can be dismissed. That was some unrelated cause. I have created a new instance and applied puppetmaster::self to it, that works! Sorry for the false alarm !
