Last modified: 2014-09-09 20:29:35 UTC
Fetching origin error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing https://git.wikimedia.org/git/mediawiki/core.git/info/refs fatal: HTTP request failed error: Could not fetch origin
To reproduce: | [tim@passepartout ~]$ for HOST in tools-{dev,login}-eqiad.wmflabs.org; do ssh "$HOST" 'git clone https://git.wikimedia.org/git/pywikibot/compat.git $(mktemp -d)'; done | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | Cloning into '/tmp/tmp.sNGj8sMBod'... | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | Cloning into '/tmp/tmp.e4O2n9lnBB'... | error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing https://git.wikimedia.org/git/pywikibot/compat.git/info/refs | fatal: HTTP request failed | [tim@passepartout ~]$ So tools-dev-eqiad works, tools-login-eqiad fails. But the problem doesn't seem to lie with curl: | [tim@passepartout ~]$ for HOST in tools-{dev,login}-eqiad.wmflabs.org; do ssh "$HOST" 'curl https://git.wikimedia.org/git/pywikibot/compat.git > /dev/null'; done | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | % Total % Received % Xferd Average Speed Time Time Time Current | Dload Upload Total Spent Left Speed | 100 1389 100 1389 0 0 84243 0 --:--:-- --:--:-- --:--:-- 113k | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | % Total % Received % Xferd Average Speed Time Time Time Current | Dload Upload Total Spent Left Speed | 100 1389 100 1389 0 0 78147 0 --:--:-- --:--:-- --:--:-- 99214 | [tim@passepartout ~]$
Only difference beneath /etc/ssl is in /etc/ssl/certs/java/cacerts which shouldn't affect git clone: | [tim@passepartout ~]$ for CMD in 'sudo find /etc/ssl -not -type d -print0 | xargs -0r sudo md5sum'; do diff -u <(ssh tools-login.eqiad.wmflabs "$CMD") <(ssh tools-dev.eqiad.wmflabs "$CMD"); done | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | --- /dev/fd/63 2014-03-09 17:25:44.783519345 +0000 | +++ /dev/fd/62 2014-03-09 17:25:44.784519334 +0000 | @@ -286,7 +286,7 @@ | c9048f79a8f1da62f89b3eeb8c493689 /etc/ssl/certs/b42ff584.0 | 8a2b0f016146ed5f78f8bdd828772803 /etc/ssl/certs/NetLock_Qualified_=Class_QA=_Root.pem | f130d662fbfeb1ddc4c35d2e0c67a357 /etc/ssl/certs/Camerfirma_Global_Chambersign_Root.pem | -750061a18276cd2b4fc8debd90cd947f /etc/ssl/certs/java/cacerts | +321edf0746699c5ac1158632a9ad4ea3 /etc/ssl/certs/java/cacerts | e0a3a4ecbfc76649d2c9f4f0d2773565 /etc/ssl/certs/a2df7ad7.0 | 47efdfb0853adc341e39d422c96fb36f /etc/ssl/certs/TC_TrustCenter__Germany__Class_2_CA.pem | 485bce6d706a2c6ef08e0d8cfd51760d /etc/ssl/certs/3c860d51.0 | [tim@passepartout ~]$
No differences in relevant packages: | [tim@passepartout ~]$ for CMD in 'sudo dpkg -l'; do diff <(ssh tools-login.eqiad.wmflabs "$CMD") <(ssh tools-dev.eqiad.wmflabs "$CMD"); done | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances | 28d27 | < ii automake 1:1.11.3-1ubuntu2 Tool for generating GNU Standards-compliant Makefiles | 75d73 | < ii dh-autoreconf 5ubuntu1 debhelper add-on to call autoreconf and clean up after the build | 1137a1136 | > ii nmap 5.21-1.1ubuntu1 The Network Mapper | 1215c1214 | < ii python-coverage 3.4-1ubuntu1 code coverage tool for Python | --- | > ii python-coverage 3.6-1 Code coverage measurement for Python | 1382a1382 | > ii terminatord 1.0.6.0ppa2 Terminator daemon | [tim@passepartout ~]$
Ran "strace -f", extracted the following list of filenames, and all are identical on tools-{dev,login}-eqiad: | /etc/gai.conf | /etc/gcrypt/fips_enabled | /etc/gitconfig | /etc/gnutls/pkcs11.conf | /etc/host.conf | /etc/hosts | /etc/ld.so.cache | /etc/ld.so.nohwcap | /etc/ld.so.preload | /etc/nsswitch.conf | /etc/pkcs11/modules | /etc/pkcs11/pkcs11.conf | /etc/resolv.conf | /etc/ssl/certs/ca-certificates.crt | /lib/x86_64-linux-gnu/libcom_err.so.2 | /lib/x86_64-linux-gnu/libcrypt.so.1 | /lib/x86_64-linux-gnu/libc.so.6 | /lib/x86_64-linux-gnu/libdl.so.2 | /lib/x86_64-linux-gnu/libgcrypt.so.11 | /lib/x86_64-linux-gnu/libgpg-error.so.0 | /lib/x86_64-linux-gnu/libkeyutils.so.1 | /lib/x86_64-linux-gnu/libnss_dns.so.2 | /lib/x86_64-linux-gnu/libnss_files.so.2 | /lib/x86_64-linux-gnu/libpthread.so.0 | /lib/x86_64-linux-gnu/libresolv.so.2 | /lib/x86_64-linux-gnu/librt.so.1 | /lib/x86_64-linux-gnu/libz.so.1 | /proc/sys/crypto/fips_enabled | /usr/bin/git | /usr/lib/git-core/git-remote-https | /usr/lib/locale/locale-archive | /usr/lib/x86_64-linux-gnu/libasn1.so.8 | /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 | /usr/lib/x86_64-linux-gnu/libgnutls.so.26 | /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 | /usr/lib/x86_64-linux-gnu/libgssapi.so.3 | /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 | /usr/lib/x86_64-linux-gnu/libheimbase.so.1 | /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 | /usr/lib/x86_64-linux-gnu/libhx509.so.5 | /usr/lib/x86_64-linux-gnu/libidn.so.11 | /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 | /usr/lib/x86_64-linux-gnu/libkrb5.so.26 | /usr/lib/x86_64-linux-gnu/libkrb5.so.3 | /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 | /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 | /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 | /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 | /usr/lib/x86_64-linux-gnu/libroken.so.18 | /usr/lib/x86_64-linux-gnu/librtmp.so.0 | /usr/lib/x86_64-linux-gnu/libsasl2.so.2 | /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 | /usr/lib/x86_64-linux-gnu/libtasn1.so.3 | /usr/lib/x86_64-linux-gnu/libwind.so.0 | /usr/share/git-core/templates/ | /usr/share/git-core/templates/branches | /usr/share/git-core/templates/config | /usr/share/git-core/templates/description | /usr/share/git-core/templates/hooks | /usr/share/git-core/templates/hooks/applypatch-msg.sample | /usr/share/git-core/templates/hooks/commit-msg.sample | /usr/share/git-core/templates/hooks/post-update.sample | /usr/share/git-core/templates/hooks/pre-applypatch.sample | /usr/share/git-core/templates/hooks/pre-commit.sample | /usr/share/git-core/templates/hooks/prepare-commit-msg.sample | /usr/share/git-core/templates/hooks/pre-rebase.sample | /usr/share/git-core/templates/hooks/update.sample | /usr/share/git-core/templates/info | /usr/share/git-core/templates/info/exclude | /usr/share/locale/en/LC_MESSAGES/git.mo | /usr/share/locale/en_US/LC_MESSAGES/git.mo | /usr/share/locale/en_US.utf8/LC_MESSAGES/git.mo | /usr/share/locale/en_US.UTF-8/LC_MESSAGES/git.mo | /usr/share/locale/en.utf8/LC_MESSAGES/git.mo | /usr/share/locale/en.UTF-8/LC_MESSAGES/git.mo | /usr/share/locale-langpack/en/LC_MESSAGES/git.mo | /usr/share/locale-langpack/en_US/LC_MESSAGES/git.mo | /usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/git.mo | /usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/git.mo | /usr/share/locale-langpack/en.utf8/LC_MESSAGES/git.mo | /usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/git.mo | /usr/share/locale/locale.alias | /var/run/nscd/socket
Happened on pronunciationrecording.eqiad.wmflabs too.
I can no longer reproduce this on tools-login or tools-dev. I'm not aware that someone consciously fixed this, so resolving as WORKSFORME.
