Last modified: 2014-10-02 08:18:06 UTC
Jobs running on Ubuntu Trusty for repositories with submodules are currently broken due to a bug in the Jenkins GIT Plugin. We use Jenkins GIT Plugin 1.5.0, which uses git-client-plugin 1.0.6: https://github.com/jenkinsci/git-client-plugin/blob/git-client-1.0.6/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L401-L412 args.add("submodule", "foreach"); if (recursive) { args.add("--recursive"); } args.add("git reset"); if (hard) { args.add("--hard"); } https://integration.wikimedia.org/ci/job/mwext-VisualEditor-npm/4684/console Cleaning workspace Resetting working tree FATAL: Command "submodule foreach --recursive git reset --hard" returned status code 1: stdout: Entering 'lib/ve' stderr: /usr/lib/git-core/git-submodule: 552: /usr/lib/git-core/git-submodule: git reset: not found Stopping at 'lib/ve'; script returned non-zero status. Trying to reproduce: jenkins-deploy@integration-slave1007:/mnt/jenkins-workspace/workspace/mwext-VisualEditor-npm$ git submodule foreach --recursive git reset --hard > works fine The version of git on Ubunty and Precise both support this command. The problem lies in the quoting and separation of the arguments. jenkins-deploy@integration-slave1007:/mnt/jenkins-workspace/workspace/mwext-VisualEditor-npm$ git submodule foreach --recursive 'git reset' --hard > /usr/lib/git-core/git-submodule: 552: /usr/lib/git-core/git-submodule: git reset: not found
Release notes of git 1.9 detailing this change in git-submodule semantics: https://raw.githubusercontent.com/git/git/master/Documentation/RelNotes/1.9.0.txt Basically, it used to interpret the argument as a long string (similar to ssh), and letting bash separate the command and arguments. And it now does this at the git level.
Created attachment 16642 [details] git submodule with GIT_TRACE=1 I found that bug while testing a job for pywikibot/core.git which needs to be run on Trusty (it uses python3.4 which is not on Precise). It yielded: /usr/lib/git-core/git-submodule: 552: /usr/lib/git-core/git-submodule: git reset: not found I have added to the job a parameter GIT_TRACE=1 and ran it again (see attached console output for https://integration.wikimedia.org/ci/job/pywikibot-core-tox-nose34-trusty/9/console ). The git command shown: stderr: trace: exec: 'git-submodule' 'foreach' '--recursive' 'git reset' '--hard' Note how git reset is enclosed in quote. Since git version 1.9.0, that is passed as a single argument to git-submodule which can find such executable 'git reset'. I eventually had an audio call and had to be back home. The Jenkins git-client-plugin needs to be patched to split 'git reset' to be two args, then compiled via maven and uploaded to Jenkins. You can take backup of the existing version by looking on gallium under /var/lib/jenkins/plugins/ for some .jpi files. Note: last time I have upgraded the Jenkins git plugin, the whole CI setup went mad :-/ I gave up investigating the java code change and just stuck to the version we currently have.
Created attachment 16643 [details] IRC log of a diagnostic on #pywikibot earlier (time is UTC+2) For what it is worth, attached is an IRC log from earlier today on #pywikibot. That resulted in the previous attachment showing the 'git reset' not found issue.
(In reply to Krinkle from comment #0) > Jobs running on Ubuntu Trusty for repositories with submodules are currently > broken due to a bug in the Jenkins GIT Plugin. > > We use Jenkins GIT Plugin 1.5.0, which uses git-client-plugin 1.0.6: > Actually, we while GIT Plugin requires git-client-plugin v1.0.5, we currently use git-client-plugin v1.4.6 -- https://github.com/wikimedia/git-client-plugin/commits/git-client-1.4.6%2Bwmf1 https://github.com/wikimedia/git-client-plugin/commit/5d4c940519352a0a38038154f8c9690a7a5dba58#diff-4a13310f7c84a9d6e133def48c0aed5aR558
https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Jenkins_Plugin Deployed the .jpi file to Jenkins.
Thank you Timo! I testing out with the job that shown the issue yesterday: https://integration.wikimedia.org/ci/job/pywikibot-core-tox-nose34-trusty/ Test case: * Precise fresh workspace > OK * Precise, rebuild on existing workspace > OK * Trusty fresh workspace > OK * Trusty, rebuild on existing workspace > OK
