Last modified: 2014-10-19 22:05:17 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T67773, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 65773 - investigate re-trying each failed test within a build
investigate re-trying each failed test within a build
Status: NEW
Product: Wikimedia
Classification: Unclassified
Quality Assurance (Other open bugs)
wmf-deployment
All All
: Normal enhancement (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-05-26 15:03 UTC by Chris McMahon
Modified: 2014-10-19 22:05 UTC (History)
3 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Chris McMahon 2014-05-26 15:03:41 UTC
Many browser tests fail for reasons other than an issue with the code or the test environment:  a flaky timeout, a momentary network glitch, etc. etc. 

It would be convenient to build into the command that calls Cucumber to run the tests the ability to: 

* track which test is being run
* evaluate the return status of each test, either per Feature or possibly even per Secenario
* if the return value for the individual test is non-zero, do not record the test result.  Instead run the single test with the single failure.
* if the second run of the single test returns a non-zero result, record that result as usual
* if the test passes on the second run, record that result and continue running the tests in the build. 

Some years ago I wrote a harness like this in Ruby.  If it is not convenient to do this directly in a shell script, it might be possible to use Ruby and shell out for the appropriate commands.
Comment 1 James Forrester 2014-10-19 22:05:17 UTC
Pushing this to at least "normal", though "high" would be preferred. :-)

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links