Last modified: 2014-08-07 14:30:02 UTC
tox runs all tests (including manual ones). We should make sure that when running tox the tests on the manual directory are not run, behaviour should be the same than running ./scripts/test
tox is just a wrapper around virtualenv, it then execute a given command. In this case 'nosetests'. Good news, nosetests let one flag tests with an attribute which can then be included/excluded at will using --attr=. I discussed about it with Pywikibot folks, they have unit tests and tests that target the production site. We wanted to prevent running tests again a site. What they ended up with is a base class for each kind of tests that adds an attribute and let them skip tests running against a site. Some magic code at: https://gerrit.wikimedia.org/r/#/c/151604/4/tests/utils.py You could go with a ManualTestCase that will have an attribute manual_test . Then exclude those tests by running: nosetests --attr='!manual_test'. That could be added to the commands= stanza in tox.ini Note, you can create a new tox environment to execute the manual test easily and end up with something like: # Default env executed for py26,py27 targets # Excludes manual tests [testenv] ... commands = nosetests --attr='!manual_test' # Only run manual tests, execute with: tox -emanual_test [testenv:manual_test] commands = nosetests --attr='manual_test' # produce coverage report [testenv:cover] commands = nosetests --attr='!manual_test' --with-coverage --cover-html
>You could go with a ManualTestCase that will have an attribute manual_test Right, we already have such an attribute. I just opened the bug to make sure we do not forget to update our tox config.
Great! Let me know if there if you encounter any issue with tox configuration. Will be happy to brainstorm with you. Also adding relevant Jenkins jobs is quite easy to handle. A short tutorial explains how to amend Jenkins configuration definition at https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Test_your_python#Editing_Jenkins_configuration :)
