Last modified: 2014-11-20 21:22:39 UTC
We currently already set up a web-facing instance of MediaWiki (during the qunit build). Let's extend our test suite by checking the error log before and after and consider any new entries a fatal build failure. php notices, warnings, errors etc. The primary motivation here is to catch php errors triggered during load.php requests because those are almost invisible in general, and even more so when all we see is the qunit callback through phantomjs. There've been a few commits lately in master that messed something up in the registration of javascript modules but nothing caught it. They are only reliably detectable at run-time. And of course it helps catching lots of other errors. Since we already have a policy of not tolerating php errors, this should be trivial to enable. Though we do have a few known php errors, none of them trigger on regular usage (open wiki page, request load.php, run qunit tests etc.).
Recommended approach: * Rename mediawiki-core-qunit to mediawiki-core-browsertest * Build steps: - Install MediaWiki - curl /wiki/Special:BlankPage (anything not HTTP 200 is error) - grunt qunit /wiki/Special:JavaScriptTest/qunit - assert apache/php error log is empty or unchanged
Another way would be to override the PHP handler to log all the exceptions, format them in a nice format (such as HTML?) and then use a post build action in Jenkins to publish the report.
HTML is difficult to work with and format from plain bash (or most programming languages for that matter). Plus we're striving towards getting rid of Jenkins formatters to only use plain text reporters (with regards to build reports static caching). I'd say plain text is more than clear enough. We're all familiar with PHP's error_log format. We just need: * php error log to go to a separate file for the qunit.localhost requests * Jenkins needs to have permission to truncate this file * Apache and/or PHP needs to not keep the handle open on this file, so that when each job truncates the file, it really does truncates it.
(In reply to comment #3) > HTML is difficult to work with and format from plain bash (or most > programming > languages for that matter). Plus we're striving towards getting rid of > Jenkins > formatters to only use plain text reporters (with regards to build reports > static caching). > > I'd say plain text is more than clear enough. We're all familiar with PHP's > error_log format. > > We just need: > * php error log to go to a separate file for the qunit.localhost requests > * Jenkins needs to have permission to truncate this file > * Apache and/or PHP needs to not keep the handle open on this file, so that > when each job truncates the file, it really does truncates it. Also, even if we'd make a custom HTML reporter or something, we still have these same problems. So let's deal with that first.
The important part in my reply was using a custom PHP error/exception handler. It seems to be easier to tweak and more robust. How would you handle several jobs running QUnit tests in parallel ?
Parsing the apache log is a smart idea to catch all issues that happens during run time and we can't catch using static analysis. But since all QUnit jobs are hitting the same virtualhost, you will end up with erros from different extensions in the apache log. Instead we could use php auto_prepend_file that would let you inject whatever PHP at the top of any PHP file being included (such as index.php / api.php ... ) http://www.php.net/manual/fr/ini.core.php#ini.auto-prepend-file That let you write a small PHP File that fetch the Jenkins env variable override the error_log and have something a bit more robust and easy to tweak. Will have to make sure the injected code is only executed when PHP require a MediaWiki entry point.
To clarify, we'd change the php init error_log setting to point to a file inside the env.WORKSPACE directory from Jenkins so that we have the errors on a per job base.
PS2: Depending on when we get to implementation of this, if we have jobs running in Vagrant we don't need any of that hackery since the VM would be dedicated to that one job only.
So we need to: 1) Add configuration to the MediaWiki install to enable logging of errors extensively to the log/ directory in the workspace. 2) Find a way to either have Apache send its error log (which includes PHPs stderr) to a local directory. Assuming that's too complicated as long as we're using the same Apache and vhost for all jobs, lets forget about Apache issues and instead find a way to hijack php's error log. Antoine has started on #1 by setting up logging. We're not yet asserting them, but at least we have them available now from the build artifacts. Antoine suggested we implement #2 by prepending php ini_set (or something like that) to all web entry points. For maintenance scripts we don't need to since we already have stderr there, and for any mediawiki internal things, those are logged to log/ which applies to both web and cli runs.
See also PHP ini setting auto_prepend_file I mentioned in comment #6. Could be pointed to some custom error handler under /srv/slave-scripts/mediawiki (which is integration/jenkins.git).
Since I added a MediaWiki log group for php errors (notices and warnings) we no longer need to get them via the apache2 log or from php sterr. We can capture them directly, and assert them to be empty. Let's set up logGroup[error] and logGroup[exception] for -qunit (www, frontend) and -phpunit (cli, installer, backend) jobs and assert them to be empty. The only thing this still won't catch is notices and warnings very early on in MediaWiki run time. Those would still be seen (for CLI scripts, these would show up in the Jenkins job console output, and for index.php/load.php the qunit job output would show these), but not detected automatically. But that's an edge case we can try and catch later. Right now we're not catching any of them.
Change 174810 had a related patch set uploaded by Krinkle: mediawiki: Configure 'error' and 'exception' log groups https://gerrit.wikimedia.org/r/174810
