Last modified: 2014-06-28 20:51:58 UTC
Per a request from James F, we should start to expand the selection of environments (OS and browser combinations) that Visual Editor automated tests run via CloudBees. For example, add a "Internet Explorer 10 on Windows 8" test, and related variations that CloudBees/SauceLabs can accommodate. Per Chris M, a good way to get started is to copy the existing "Selenium" template in CloudBees and figure out what we need to do to add Windows into the mix.
Created a new test template to mess around with: https://wmf.ci.cloudbees.com/template/81889c2c-ef15-4fad-b39b-8256eb9e841a/
In addition to the new template in CloudBees, I think we'll need to expand the config.yml file in the VisualEditor repo: https://github.com/wikimedia/mediawiki-extensions-VisualEditor/blob/master/modules/ve-mw/test/browser/config/config.yml For example, our existing "firefox" profile is hard-coded to Firefox version 23 on Linux, so we'll need to add more entries for Firefox on Windows 8, etc.
#1 I would hesitate to add more Jenkins jobs until existing ones[1] are rock solid (meaning there were no failures in a week or so). #2 The entire config folder will soon(tm) be moved to mediawiki-selenium gem[2]. We should probably rename config.yml to browsers.yml (since it no longer has any other information, only browsers) and expand the number of combinations, like Wikibase has already done[3] 1: https://wmf.ci.cloudbees.com/view/r-ve/ 2: https://gerrit.wikimedia.org/r/#/c/90560 3: https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/tests/browser/config/browsers.yml
(In reply to comment #3) > #1 I would hesitate to add more Jenkins jobs until existing ones[1] are rock > solid (meaning there were no failures in a week or so). > > #2 The entire config folder will soon(tm) be moved to mediawiki-selenium > gem[2]. We should probably rename config.yml to browsers.yml (since it no > longer has any other information, only browsers) and expand the number of > combinations, like Wikibase has already done[3] > > 1: https://wmf.ci.cloudbees.com/view/r-ve/ > 2: https://gerrit.wikimedia.org/r/#/c/90560 > 3: > https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/tests/ > browser/config/browsers.yml Thanks Željko - I agree with both of your points, especially #1, which is an ongoing problem area (which I am working on!)
(In reply to comment #3) > #2 The entire config folder will soon(tm) be moved to mediawiki-selenium > gem[2]. We should probably rename config.yml to browsers.yml (since it no > longer has any other information, only browsers) and expand the number of > combinations, like Wikibase has already done[3] config.yml is now moved to mediawiki-selenium gem. Feel free to expand the number of browser/OS combinations in the gem.
Per a chat with Rummana, the priority test environments to add (via CloudBees and Sauce Labs) are: 1) Chrome on Windows XP 2) Firefox on Windows XP
Change 110980 had a related patch set uploaded by Jhall: Adding new priority Windows browser tests for VisualEditor. https://gerrit.wikimedia.org/r/110980
Setting bug status back to resolved since the 110980 Gerrit patch is a first step only; if that change works out well, we can add additional (browser + OS) combinations to work towards eventually resolving this bug.
Change 110980 abandoned by Jhall: Adding new priority Windows browser tests for VisualEditor. Reason: Abandoning this change per a discussion between Chris, Željko and myself. We have decided to implement new (browser + OS) combinations for use with SauceLabs via environment variable values rather than via hard-coded amendments to the mediawiki-selenium gem itself. https://gerrit.wikimedia.org/r/110980
Tests have been setup in CloudBees at the following locations: 1) Chrome on Windows XP: https://wmf.ci.cloudbees.com/view/ve/job/VisualEditor-en.wikipedia.beta.wmflabs.org-windows_xp-chrome/ 2) Firefox on Windows XP: https://wmf.ci.cloudbees.com/view/ve/job/VisualEditor-en.wikipedia.beta.wmflabs.org-windows_xp-firefox/
Jeff, where are the jobs configured? I do not see them anywhere. This is where I have looked: https://github.com/zeljkofilipin/jenkins-job-builder-config https://gerrit.wikimedia.org/r/#/admin/projects/integration/jenkins-job-builder-config Did you create the jobs manually?
Can this be closed? We have this job now, for example: https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_xp-firefox-sauce/
Yes.
