Last modified: 2014-01-09 18:52:39 UTC
Could we use the same subject heading for these emails so they grouped into one thread? e.g. QA Report and move the current subject into the email body itself? Currently I am getting lots of mails in the form: Build failed in Jenkins: MobileFrontend-en.m.wikipedia.org-linux-firefox #97 and it is very hard to group and filter them. As a result I tend to ignore them due to too much noise which is bad :).
We should find a Jenkins plugin that should allow us to customize e-mail notification messages. The default messages are not really useful.
I used this plugin in a previous job and found it to be very flexible (among other things, it provides for customized e-mail subject strings): https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin
This is how a typical Jenkins report looks like (see attached file).
Created attachment 13911 [details] typical Jenkins report
If you do not have an e-mail client, this is how an e-mail notification for a failed Jenkins job looks like: From: jenkins-no-reply@cloudbees.com To: Date: Tue, 26 Nov 2013 19:02:57 +0100 Subject: Build failed in Jenkins: VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox #172 See <https://wmf.ci.cloudbees.com/job/VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox/172/changes> Changes: [roan.kattouw] Share code between conflict resolution and the edit source switch [roan.kattouw] Don't build out save dialog when initializing editor [roan.kattouw] Use InputWidget facilities in byte limit counter [krinklemail] Remove the toolbar tracker ------------------------------------------ Started by user Chris Mcmahon Building remotely on s-e4476767 in workspace <https://wmf.ci.cloudbees.com/job/VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox/ws/> Cloning the remote Git repository Cloning repository https://gerrit.wikimedia.org/r/mediawiki/extensions/VisualEditor git --version git version 1.7.11.7 Checking out Revision 07464e20a535fe1efdbd7f8e862ff8f7b3825722 (origin/master) [VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox] $ /bin/sh -xe /tmp/hudson495896231200628031.sh + export MEDIAWIKI_USER=Selenium_user + MEDIAWIKI_USER=Selenium_user + export MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + export BROWSER_LABEL=firefox + BROWSER_LABEL=firefox + export MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ + MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ + curl -s -o use-ruby https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/ruby/use-ruby + RUBY_VERSION=2.0.0-p247 + source ./use-ruby ++ [[ -z 2.0.0-p247 ]] ++ ruby187='1.8.7-*' ++ [[ 2.0.0-p247 == 1.8.7-* ]] ++ BASE=/scratch/jenkins/addons ++ mkdir -p /scratch/jenkins/addons ++ '[' '!' -f /scratch/jenkins/addons/use.sh ']' ++ wget -q -P /scratch/jenkins/addons https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/lib/use.sh ++ . /scratch/jenkins/addons/use.sh ++ mkdir -p /scratch/jenkins/addons/ruby +++ addon_architecture +++ uname -m ++ ruby_name=ruby-2.0.0-p247-x86_64 ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64.tar.bz2 ']' +++ addon_platform ++++ head -n 1 ++++ cat /etc/issue +++ issue='Fedora release 17 (Beefy Miracle)' +++ fedora_regex='Fedora release 17*' +++ [[ Fedora release 17 (Beefy Miracle) == Fedora release 17* ]] +++ echo fc17 +++ return ++ wget -q -P /scratch/jenkins/addons/ruby https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/ruby/fc17/ruby-2.0.0-p247-x86_64.tar.bz2 ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin/ruby ']' ++ tar xjf /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64.tar.bz2 -C /scratch/jenkins/addons/ruby ++ export RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ export PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ unset GEM_PATH ++ unset GEM_HOME ++ unset BUNDLE_PATH + gem install bundler --no-ri --no-rdoc Successfully installed bundler-1.3.5 1 gem installed + '[' -d modules/ve-mw/test/browser/ ']' + cd modules/ve-mw/test/browser/ + bundle install Fetching gem metadata from https://rubygems.org/.......... Fetching gem metadata from https://rubygems.org/.. Installing builder (3.2.2) Installing ffi (1.9.3) Installing childprocess (0.3.9) Installing diff-lcs (1.2.4) Installing multi_json (1.8.2) Installing gherkin (2.12.2) Installing multi_test (0.0.2) Installing cucumber (1.3.8) Installing i18n (0.6.5) Installing faker (1.2.0) Installing yml_reader (0.2) Installing data_magic (0.16.1) Installing json (1.8.1) Installing net-http-persistent (2.9) Installing page_navigation (0.9) Installing rubyzip (1.0.0) Installing websocket (1.0.7) Installing selenium-webdriver (2.37.0) Installing watir-webdriver (0.6.4) Installing page-object (0.9.3) Installing mime-types (2.0) Installing rest-client (1.6.7) Installing rspec-expectations (2.14.3) Installing syntax (1.0.0) Installing mediawiki-selenium (0.1.12) Using bundler (1.3.5) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed. + bundle exec cucumber --verbose --profile ci --tags @en.wikipedia.beta.wmflabs.org Code: * features/support/env.rb * features/support/hooks.rb * features/support/modules/url_module.rb * features/support/pages/cursor_test_page.rb * features/support/pages/login_page.rb * features/support/pages/visual_editor_page.rb * features/step_definitions/bullets_steps.rb * features/step_definitions/common_steps.rb * features/step_definitions/cursor_steps.rb * features/step_definitions/general_markup_steps.rb * features/step_definitions/headings_steps.rb * features/step_definitions/links_steps.rb * features/step_definitions/references_steps.rb * features/step_definitions/shared_steps.rb * features/step_definitions/transclusion_steps.rb Features: * features/bullets.feature * features/edit_utf8_logged_in.feature * features/general_markup.feature * features/headings.feature * features/indent_outdent.feature * features/links.feature * features/references.feature * features/transclusion.feature Parsing feature files took 0m0.111s Build step 'Execute shell' marked build as failure Recording test results
From what I understand, the people would like the e-mail notification to look similar to this page: https://wmf.ci.cloudbees.com/view/r-ve/job/VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome/164/
(In reply to comment #6) > From what I understand, the people would like the e-mail notification to look > similar to this page: > > https://wmf.ci.cloudbees.com/view/r-ve/job/VisualEditor-en.wikipedia.beta. > wmflabs.org-linux-chrome/164/ Yeah, that would be *much* more useful. It would also be nice to still get the recent changes that were checked in like in the current mail, eg: Changes: [roan.kattouw] Share code between conflict resolution and the edit source switch [roan.kattouw] Don't build out save dialog when initializing editor [roan.kattouw] Use InputWidget facilities in byte limit counter [krinklemail] Remove the toolbar tracker And if they could provide links to changesets gerrit, that would be even better :) It would also be good if there was a direct link provided to the build in jenkins.
Hey, I'm GEOFBOT from GCI! I think I'll try out this task, but I have no clue how to start out. Where should I start out?
Hi Geoffrey and welcome! For future reference, if you get stuck you can ask questions here, at #wikimedia-qa freenode IRC channel or at QA mailing list[1]. To answer your question, start with either installing Jenkins[2] on your local machine or create a free account at a hosted Jenkins service, like Cloudbees[3]. When you have a test Jenkins instance, create a few jobs and start sending e-mail notifications to your e-mail. Then install the plugin[4] Jeff has suggested and try changing the e-mail notifications so they look more like this report[5]. If you get stuck, or if you have any questions, let us know. 1: https://lists.wikimedia.org/mailman/listinfo/qa 2: http://jenkins-ci.org/ 3: http://www.cloudbees.com/ 4: https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin 5: https://wmf.ci.cloudbees.com/view/r-ve/job/VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome/164/
Cloudbees URL: https://geofbot.ci.cloudbees.com/job/test/
Geoffrey, this might be the problematic part: "Have your Jenkins administrator place the script inside JENKINS_HOME\email-templates." (from plugin documentation[1]). You should contact cloudbees support and ask how to do it. 1: https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin
Progress! I managed to extract the first line of console output to find who started the job. It took a lot of guesswork (I have never coded in Groovy before, especially not within a HTML :P)
Geoffrey, feel free to attach the e-mail to this bug, or to send it to me, so I could take a look.
Created attachment 13947 [details] Progress on email-ext customization
Created attachment 13967 [details] First version of jenkins template ready for testing 1. To use this template, place it under $JENKINS_HOME/email-templates/ 2. make sure email-ext plugin is installed 3. Under the job config for post-build actions, remove "email notification" and add "editable email notification" 4. Change default content type to html 5. Change body to ${SCRIPT, template="wikimedia-jenkins.template-1"} 6. Optional - change the subject using the token references; here is the one i used: ${PROJECT_NAME} ${BUILD_NUMBER} - ${BUILD_STATUS} 7. Set the recipients, etc. 8. Open Advanced Settings... and there you can change when an email notification is triggered.
Geoffrey, can you attach an example e-mail, so we can see if the template does what we need?
Created attachment 13971 [details] Example html 2 I don't have tests set up on my jenkins, so unfortunately I couldn't test that aspect of the template I replaced all of the LAN image references with a cloudbees link so they would work, but the links to the build will be broken.
Created attachment 13975 [details] Template v1.1 Better version with some bugs ironed out Would be nice if someone tested out the "tests" part of it because I don't have tests set up for my local jenkins build.
Created attachment 13976 [details] Example html 3 The example html for template 1.1
Geoffrey, I did not want to mess up existing job (test), so I have created another one (test2). Short documentation on how to set up a job to run tests is available[1]. I have cloned browsertests[2] repository and the job runs tests from login.feature[3] file. I will attach e-mail that was sent after the job failed. 1: https://github.com/wikimedia/mediawiki-selenium/blob/master/docs/job.md 2: https://github.com/wikimedia/qa-browsertests 3: https://github.com/wikimedia/qa-browsertests/blob/master/features/login.feature
Created attachment 13979 [details] e-mail notification with failed tests
I forgot to say, now you can test e-mail notifications using your jobs. If you have any questions, let me know.
Geoffrey, see attached file on how an e-mail notification after failed test should look like. It is generated by this test: https://geofbot.ci.cloudbees.com/job/test2/lastCompletedBuild/testReport/
Created attachment 13990 [details] how e-mail notification after failed test should look like
(In reply to comment #24) > Created attachment 13990 [details] > how e-mail notification after failed test should look like Do you want the entire table there in the email? Right now I have the notifications working with everything and it looks pretty close to the jenkins build page.
Geoffrey, the last e-mail that I got from your local Jenkins looked good to me! Please upload the template here so I can test it with our Jenkins and if it works I will close the task. If you would like to continue working on the template (making it look more like Jenkins web page) I will create another task. If you would like to work on something else, we have plenty of tasks.
Created attachment 14019 [details] Final email template (for now)
(In reply to comment #26) > If you would like to continue working on the template (making it look more > like > Jenkins web page) I will create another task. Sure, I would definitely like to finish up this template.
(In reply to comment #15) > Created attachment 13967 [details] > First version of jenkins template ready for testing > > 1. To use this template, place it under $JENKINS_HOME/email-templates/ > 2. make sure email-ext plugin is installed > 3. Under the job config for post-build actions, remove "email notification" > and > add "editable email notification" > 4. Change default content type to html > 5. Change body to ${SCRIPT, template="wikimedia-jenkins.template-1"} > 6. Optional - change the subject using the token references; here is the one > i > used: ${PROJECT_NAME} ${BUILD_NUMBER} - ${BUILD_STATUS} > 7. Set the recipients, etc. > 8. Open Advanced Settings... and there you can change when an email > notification is triggered. If you want the subject to be constant for one build, make the subject just ${PROJECT_NAME}
Asked cloudbees to upload the template: https://cloudbees.zendesk.com/requests/14940
As far as I am concerned, this bug is resolved. If you need more/less information in Jenkins e-mail notification, leave a comment. See attached e-mail for "before" and "after".
Created attachment 14047 [details] before
Created attachment 14048 [details] after
Change 103114 had a related patch set uploaded by Sn1per: Add jenkins email template for better email notifications https://gerrit.wikimedia.org/r/103114
Change 103114 merged by Cmcmahon: Add jenkins email template for better email notifications https://gerrit.wikimedia.org/r/103114
Change 106513 had a related patch set uploaded by Zfilipin: Use new e-mail template https://gerrit.wikimedia.org/r/106513
Change 106513 merged by Cmcmahon: Use new e-mail template https://gerrit.wikimedia.org/r/106513
