Last modified: 2014-01-11 08:53:02 UTC
hashar@integration-selenium-driver:~$ curl https://git.wikimedia.org/ curl: (60) SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. hashar@integration-selenium-driver:~$
With curl in verbose mode: hashar@integration-selenium-driver:~$ curl -v https://git.wikimedia.org/ * About to connect() to git.wikimedia.org port 443 (#0) * Trying 208.80.154.241... connected * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS alert, Server hello (2): * SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed * Closing connection #0 curl: (60) SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. hashar@integration-selenium-driver:~$ I guess labs is missing some certificates :(
Change 106771 had a related patch set uploaded by Hashar: star.wikimedia.org cert chain fix https://gerrit.wikimedia.org/r/106771
Found out the same issue on production machine lanthanum.eqiad.wmnet. It is lacking the Rapid SSL CA cert: $ ll /etc/ssl/certs/*apid* ls: cannot access /etc/ssl/certs/*apid*: No such file or directory $ No chained either since that machine never had any certificate installed.
According to the history of Jenkins job https://integration.wikimedia.org/ci/job/mwext-browsertests-UniversalLanguageSelector-phantomjs/ (which uses https://git.wikimedia.org/ ). That stopped working between Dec 10 2013 17:15 and Dec 11 2013 13:40UTC.
Damn I was wrong! The job last success was Jan 8th 13:20 , first failure Jan 10th 3:37 UTC.
Trying on lanthanum: hashar@lanthanum:~$ openssl s_client -connect git.wikimedia.org:443 ... Certificate chain 0 s:/serialNumber=06QcQ9dUSZqu5ru7oQSfeCpXiBccrCyh/C=US/O=*.wikimedia.org/OU=GT11518520/OU=See www.rapidssl.com/resources/cps (c)10/OU=Domain Control Validated - RapidSSL(R)/CN=*.wikimedia.org i:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA 1 s:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority $ Equifax is wrong, should be Geotrust :(
And on my machine the chain is: Certificate chain 0 s:/serialNumber=06QcQ9dUSZqu5ru7oQSfeCpXiBccrCyh/C=US/O=*.wikimedia.org/OU=GT11518520/OU=See www.rapidssl.com/resources/cps (c)10/OU=Domain Control Validated - RapidSSL(R)/CN=*.wikimedia.org i:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA 1 s:/C=US/O=GeoTrust, Inc./CN=RapidSSL CA i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA (correct)
Change 106785 had a related patch set uploaded by RobH: fixes star.wikimedia.org intermidite certificate chain https://gerrit.wikimedia.org/r/106785
Change 106785 merged by RobH: fixes star.wikimedia.org intermidite certificate chain https://gerrit.wikimedia.org/r/106785
Change 106842 had a related patch set uploaded by RobH: install rapidssl_ca_2.pem https://gerrit.wikimedia.org/r/106842
Change 106842 merged by RobH: install rapidssl_ca_2.pem https://gerrit.wikimedia.org/r/106842
Ok, So the two patchsets I submitted for this are linked in ticket here. Turns out the wildcard rapidssl has two intermediate certificates, versus the non-wildcard rapidssl certs that use the single one we've had on cluster and in puppet for awhile. I've added the second cert to our repo and included it in use for misc-web-lb.eqiad. Now curls return correctly, folks shouldn't be getting certificate errors anymore.
Works for me, thanks Rob.
