Last modified: 2014-10-27 12:56:54 UTC
I have wrote a short e-mail[1] about running tests on PhantomJS, but that was a few months ago. Run all tests in the most recent version of PhantomJS, fix failures and tag tests that do not run in PhantomJS by design with @phantomjs-bug. (For example of @phantomjs-bug tag see aftv5.feature[2] file.) 1: http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/066009.html 2: https://github.com/wikimedia/qa-browsertests/blob/master/features/aftv5.feature#L2
Current status: $ export BROWSER_LABEL=phantomjs z@imac:~/project/wmf/browsertests(master)$ bundle exec cucumber --tags @en.wikipedia.beta.wmflabs.org ... Failing Scenarios: cucumber features/aftv5.feature:23 # Scenario: Click yes and leave feedback cucumber features/file.feature:9 # Scenario: Logged-in user goes to file that does not exist cucumber features/math_readonly.feature:5 # Scenario: Display simple math cucumber features/page.feature:7 # Scenario: Go to page that does not exist cucumber features/page.feature:12 # Scenario: Prepare to start a new page using the URL cucumber features/page.feature:22 # Scenario: Start a new page using the URL cucumber features/page.feature:36 # Scenario: Move existing page dialog cucumber features/page.feature:48 # Scenario: Move existing page cucumber features/page_edit.feature:16 # Scenario: Read link cucumber features/page_edit.feature:25 # Scenario: View History link cucumber features/page_triage_readonly.feature:5 # Scenario: Check that NewPagesFeed has correct controls for anonymous user cucumber features/page_triage_readonly.feature:14 # Scenario: Check set filters selection cucumber features/pdf_readonly.feature:10 # Scenario: Click on Download as PDF link cucumber features/print_export_menu_readonly.feature:9 # Scenario: Check for Print/export section Expanded cucumber features/print_export_menu_readonly.feature:14 # Scenario: Print/export section collapses after it is expanded cucumber features/uls_cog_sidebar_anon_user.feature:58 # Scenario: More languages cucumber features/uls_ime.feature:4 # Scenario: User can see input method keyboard icon cucumber features/uls_ime.feature:9 # Scenario: User can see a dropdown menu when they click on the IME icon 55 scenarios (18 failed, 37 passed) 266 steps (18 failed, 58 skipped, 190 passed) 13m53.749s
$ export BROWSER_LABEL=firefox z@imac:~/project/wmf/browsertests(master)$ bundle exec cucumber --tags @en.wikipedia.beta.wmflabs.org ... Failing Scenarios: cucumber features/aftv5.feature:23 # Scenario: Click yes and leave feedback cucumber features/page_triage_readonly.feature:5 # Scenario: Check that NewPagesFeed has correct controls for anonymous user cucumber features/page_triage_readonly.feature:14 # Scenario: Check set filters selection cucumber features/pdf_readonly.feature:10 # Scenario: Click on Download as PDF link cucumber features/print_export_menu_readonly.feature:14 # Scenario: Print/export section collapses after it is expanded cucumber features/uls_cog_sidebar_anon_user.feature:58 # Scenario: More languages cucumber features/uls_ime.feature:4 # Scenario: User can see input method keyboard icon cucumber features/uls_ime.feature:9 # Scenario: User can see a dropdown menu when they click on the IME icon 55 scenarios (8 failed, 47 passed) 266 steps (8 failed, 11 skipped, 247 passed) 17m22.890s
For some reason, more tests fail when they are run in parallel. $ export BROWSER_LABEL=phantomjs $ bundle exec parallel_cucumber features/ --test-options '--tags @en.wikipedia.beta.wmflabs.org' 4 processes for 26 features, ~ 6 features per process ... 55 scenarios (25 failed, 4 skipped, 26 passed) 266 steps (25 failed, 92 skipped, 149 passed) Took 347.455641 seconds Tests Failed
Number of failed tests does not raise when using Firefox and running the tests in parallel. $ export BROWSER_LABEL=firefox $ bundle exec parallel_cucumber features/ --test-options '--tags @en.wikipedia.beta.wmflabs.org' 4 processes for 26 features, ~ 6 features per process ... 55 scenarios (7 failed, 48 passed) 266 steps (7 failed, 11 skipped, 248 passed) Took 396.053233 seconds Tests Failed
I tried to do a non scientific approach and run the tests in firefox and phantomjs and if I got the same result for the same test (be it the same fail or pass) then I would conclude that the test is o.k. in phantomjs ;) I took this approach because I don't have enough experience and i'm not really sure in what condition are the tests. I'm not sure if that's good enough (please comment), but here are my results (#number means i have log for this): STATUS: aftv5.feature - last test doesn't pass on ff & ph #3 (timeout) - not sure but I would remove @phantomjs-bug tag because I get the same test result as on ff? create_account_readonly.feature - ok on both file.feature - ok on both guided_tour.feature - ok on both login.feature - last test doesn't pass on ff & ph #2 (timeout) login_sul.feature - ok on both math_readonly - ff ok, ph error #19 (unable to locate element) page_edit.feature - test2: ok on both, en.beta... ff one error #5 (modal dialog present), ph another #6 (timeout, element not visible) page.feature - ff ok, ph error #20 (expected #, multiple: unable to locate element) page_triage_readonly.feature - test2 ok on both, on en.beta server returns API page error on both #18 (timeout) pdf_readonly.feature - ok preferences_appearance_readonly.feature - ok preferences_datetime_readonly.feature - ok print_export_menu_readonly.feature - same error ff & ph #7 (expected visible) search.feature - ok uls_accept_language.feature - should be tagged @phantomjs-bug (runs only in ff and chrome) uls_cog_sidebar_anon_user.feature - ok uls_cog_sidebar_logged_user.feature - ok uls.feature - same error ff & ph #9 (unable to locate element) uls_ime.feature - same error ff & ph #8 (expected visible) upload_wizard.feature - ff ok on test2, ph error #10 (timeout) visual_editor.feature - same error ff & ph #11 (timeout) visual_editor_links.feature - ff ok on en.beta - test2 error #14 (timeout), phantomjs errors #13 (Net::ReadTimeout) #14 (timeout) visual_editor_references.feature - ff ok on en.beta - test2 error #15 (timeout), ph error #15 on both env visual_editor_transclusion.feature - ff one error #16 (element not attached to DOM), ph another #17 (timeout) wikilove.feature - ok
Tomislav, I have removed keyword "easy" from this bug since I no longer think it is an easy bug, but feel free to continue working on it. :) As you have correctly noticed, at the moment we have tests that are failing for certain browser/MEDIAWIKI_URL combinations. The ideal scenario would be: - a feature/scenario is passing for at least one browser (usually Firefox or Chrome) - run the same feature/scenario using phantomjs - investigate the failures (if any) and then - fix the failures for phantomjs (if possible) or - tag the feature/scenario with @phantomjs-bug The prerequisite is to get a feature/scenario passing for Firefox/Chrome, meaning that failing tests should be investigates and fixed (if possible) or tagged @firefox-bug/@chrome-bug. If you need help with this, let me know. We can discuss it here, at the QA mailing list or we can pair on it.
See related thread at QA mailing list: http://lists.wikimedia.org/pipermail/qa/2013-November/000723.html
Current status with phantomjs 1.9.2 and the latest commit of browsertests repository. $ export BROWSER_LABEL=phantomjs $ export MEDIAWIKI_USER=Selenium_user $ export MEDIAWIKI_PASSWORD=not-the-real-one $ export MEDIAWIKI_URL=http://commons.wikimedia.beta.wmflabs.org/wiki/ $ bundle exec cucumber -t @commons.wikimedia.beta.wmflabs.org ... Failing Scenarios: cucumber features/upload_wizard.feature:28 # Scenario: Navigate to Release rights page cucumber features/upload_wizard.feature:36 # Scenario: Navigate to Describe page cucumber features/upload_wizard.feature:46 # Scenario: Navigate to Use page 5 scenarios (3 failed, 2 passed) 37 steps (3 failed, 17 skipped, 17 passed) 4m29.849s $ export MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ $ bundle exec cucumber -t @en.wikipedia.beta.wmflabs.org ... Failing Scenarios: cucumber features/aftv5.feature:22 # Scenario: Click Yes return and click No cucumber features/preferences_datetime.feature:15 # Scenario: Preferences Date Time 31 scenarios (2 failed, 29 passed) 163 steps (2 failed, 5 skipped, 156 passed) 7m7.580s $ export MEDIAWIKI_URL=http://test2.wikipedia.org/wiki/ $ bundle exec cucumber -t @test2.wikipedia.org ... Failing Scenarios: cucumber features/hotcat.feature:20 # Scenario: Hotcat category link cucumber features/upload_wizard.feature:23 # Scenario: Navigate to Upload page cucumber features/upload_wizard.feature:28 # Scenario: Navigate to Release rights page cucumber features/upload_wizard.feature:36 # Scenario: Navigate to Describe page cucumber features/upload_wizard.feature:46 # Scenario: Navigate to Use page 39 scenarios (5 failed, 34 passed) 214 steps (5 failed, 21 skipped, 188 passed) 10m58.657s It is interesting that commons.wikimedia.beta.wmflabs.org failures were Watir::Wait::TimeoutError and Watir::Exception::UnknownObjectException, but commons.wikimedia.beta.wmflabs.org and test2.wikipedia.org were all Net::ReadTimeout (except for one Watir::Exception::UnknownObjectException at test2.wikipedia.org).
If you need more information (and you probably do), feel free to ask questions here, at #wikimedia-qa freenode IRC channel or at QA mailing list: https://lists.wikimedia.org/mailman/listinfo/qa
Change 101213 had a related patch set uploaded by Zfilipin: Fix Accept-Language feature[1] for PhantomJS https://gerrit.wikimedia.org/r/101213
Change 101213 merged by Cmcmahon: Fix Accept-Language feature[1] for PhantomJS https://gerrit.wikimedia.org/r/101213
Change 101215 had a related patch set uploaded by Zfilipin: Fix Accept-Language feature for PhantomJS https://gerrit.wikimedia.org/r/101215
Change 101215 merged by jenkins-bot: Fix Accept-Language feature for PhantomJS https://gerrit.wikimedia.org/r/101215
Change 102099 had a related patch set uploaded by Mayankmadan: Fixing failing tests on phantomjs https://gerrit.wikimedia.org/r/102099
Change 102396 had a related patch set uploaded by Mayankmadan: Fixing Failing tests on PhantomJS https://gerrit.wikimedia.org/r/102396
Change 102099 abandoned by Mayankmadan: Fixing failing tests on phantomjs Reason: Created a new review request for this bug https://gerrit.wikimedia.org/r/#/c/102396/ https://gerrit.wikimedia.org/r/102099
Change 102396 abandoned by Mayankmadan: Fixing Failing tests on PhantomJS Reason: There is nothing useful in this commit https://gerrit.wikimedia.org/r/102396
Change 102634 had a related patch set uploaded by Mayankmadan: Fix failing tests on phantomjs https://gerrit.wikimedia.org/r/102634
Change 102664 had a related patch set uploaded by Zfilipin: Updating Ruby gems https://gerrit.wikimedia.org/r/102664
Change 102664 merged by jenkins-bot: Updating Ruby gems https://gerrit.wikimedia.org/r/102664
Only seeing a handful of failures now with PhantomJS 1.9.2: $ export BROWSER_LABEL=phantomjs $ export MEDIAWIKI_USER=Selenium_user $ export MEDIAWIKI_PASSWORD=not-the-real-one $ export MEDIAWIKI_URL=http://commons.wikimedia.beta.wmflabs.org/wiki/ $ bundle exec cucumber -t @commons.wikimedia.beta.wmflabs.org Using the default profile... .............F---...F-----...F--------- (::) failed steps (::) Net::ReadTimeout (Net::ReadTimeout) /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:144:in `readline' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `catch' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1376:in `request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1369:in `block in request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:852:in `start' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request' (eval):1:in `process_watir_call' ./features/step_definitions/upload_wizard_steps.rb:63:in `/^upload file (.+)$/' features/upload_wizard.feature:30:in `And upload file image.png' Net::ReadTimeout (Net::ReadTimeout) /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:144:in `readline' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `catch' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1376:in `request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1369:in `block in request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:852:in `start' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request' (eval):1:in `process_watir_call' ./features/step_definitions/upload_wizard_steps.rb:63:in `/^upload file (.+)$/' features/upload_wizard.feature:38:in `And upload file image.png' Net::ReadTimeout (Net::ReadTimeout) /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:144:in `readline' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `catch' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1376:in `request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1369:in `block in request' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:852:in `start' /Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request' (eval):1:in `process_watir_call' ./features/step_definitions/upload_wizard_steps.rb:63:in `/^upload file (.+)$/' features/upload_wizard.feature:48:in `And upload file image.png' Failing Scenarios: cucumber features/upload_wizard.feature:28 # Scenario: Navigate to Release rights page cucumber features/upload_wizard.feature:36 # Scenario: Navigate to Describe page cucumber features/upload_wizard.feature:46 # Scenario: Navigate to Use page 5 scenarios (3 failed, 2 passed) 37 steps (3 failed, 17 skipped, 17 passed) 4m27.465s
Change 102634 abandoned by Mayankmadan: Fix failing tests on phantomjs https://gerrit.wikimedia.org/r/102634
Change 102634 restored by Mayankmadan: Fix failing tests on phantomjs Reason: All the failing tests are tagged with @phantomjs-bug https://gerrit.wikimedia.org/r/102634
Change 102634 abandoned by Zfilipin: Fix failing tests on phantomjs Reason: Tests that do not run on phantomjs are now tagged differently. All tests by default have @phantomjs tag, and if the test does not work with the browser, the tag is removed. https://gerrit.wikimedia.org/r/102634
Resetting status to NEW; no patches left here to review
Željko / Jeff: Do we still have failing tests that a new contributor could fix? Are the instructions provided here still valid? If both yes, please add "gci2014" to the Whiteboard field of this ticket and potentially "easy" to the Keywords field (if the approach is clear and if this task is self-contained and non-controversial). Thanks!
I am not sure if we care about phantomjs any more at all.
So lets just close this bug. The idea was to easily run browser tests in a headless env since it seemed straight forward to run. Timo pointed us that PhantomJS is using an outdated Webkit engine that is no more in use by modern browsers. For QUnit tests he is moving to a node module "karma" which uses chromium and an XVFB local server. That runs smoothly. For our browser tests, we have our bidaily run execute on SauceLabs. If one day we want to trigger tests on patchset submission, I guess we will use chromium/XVFB as well.
