Last modified: 2014-07-02 20:12:48 UTC
I'm still not clear on the root cause, but if test2wiki is used as the --wiki argument to `mwscript mergeMessageFileList.php` it will generate a different ExtensionMessages-<VERSION>.php file than the same script called with --wiki=mediawikiwiki The php implementation of activeMWVersions returns the the first wiki listed in the wikiversions.json file because of the intrinsic behavior of iterating php arrays. For a typical group0 initial deploy this will return 1.XwmfY=mediawikiwiki as the version=wikidb pair for the new version. The python implementation of the same script is not guaranteed to iterate the wikiversions.json file in any particular order. Cpython seems to iterate in a consistent order based on the interpreter version. On the production cluster this iteration order currently yields 1.XwmfY=test2wiki for the new version.
Change 124500 had a related patch set uploaded by BryanDavis: Maintain file order when iterating wikiversions.json https://gerrit.wikimedia.org/r/124500
Change 124500 merged by jenkins-bot: Maintain file order when iterating wikiversions.json https://gerrit.wikimedia.org/r/124500
The JSON standard specifically states that "an object is an unordered set of name/value pairs". If the order matters, it's a bug. Either the order shouldn't matter, or a JSON object is the wrong way to represent the data. Change I3c5ca78ae is a kludge, in my opinion. Cpython will randomize the value used to seed its hashing algorithm when invoked with -R or when the PYTHONHASHSEED environment var is set to "random". On Python 3.3 and greater, hash randomization is turned on by default.
The patch I just submitted is at most a bandaid, but it's useful for testing the hypothesis.
Change 124508 had a related patch set uploaded by BryanDavis: Revert "Maintain file order when iterating wikiversions.json" https://gerrit.wikimedia.org/r/124508
Apparently testwiki is the only wiki version that creates a proper ExtensionMessages file. Or better stated, running `mwscript mergeMessageFileList.php --wiki=testwiki` generates a different ExtensionMessages file that using test2wiki or mediawikiwiki does and the testwiki variant seems to not cause l10nupdate-1 to break core translations.
Change 124508 merged by jenkins-bot: Revert "Maintain file order when iterating wikiversions.json" https://gerrit.wikimedia.org/r/124508
* mw-update-l10n gets version=wikidb data via mwversionsinuse * mwversionsinuse reads from /a/common/wikiversions.json * mwscript uses MWMultiVersion::initializeForMaintenance() to process the `--wiki=<WIKIDB>` argument and choose a php-1.XwmfY version to run scripts from. * MWMultiVersion::getVersion() reads the compiled /usr/local/apache/common-local/wikiversions.cdb to find the version for a given database * Scap does not compile /usr/local/apache/common-local/wikiversions.json into /usr/local/apache/common-local/wikiversions.cdb until the final step in the scap process. This combination means that when mw-update-l10n runs `mwscript mergeMessageFileList.php` the version seen by mergeMessageFileList will always be the version that database was at on after the *prior* scap (or sync-wikiversions) execution. This is why it looks like ExtensionMessages-1.23wmf21 was built for the prior version; it was. If you run scap twice without changing wikiversions.json, the second time mergeMessageFileList.php will see the expected version of MediaWiki. The fix for this is to add a step early in scap (just after sync-common) to compile /usr/local/apache/common-local/wikiversions.json to wikiversions.cdb. This shouldn't be done before or during sync-common because that would push the updated version out to the apaches during their runs of sync-common and thus before they have compiled the l10n cache locally.
Change 124627 had a related patch set uploaded by BryanDavis: Compile wikiversions.json to cdb for local sync https://gerrit.wikimedia.org/r/124627
Change 124627 merged by jenkins-bot: Compile wikiversions.json to cdb for local sync https://gerrit.wikimedia.org/r/124627
Change 124762 had a related patch set uploaded by BryanDavis: Compile wikiversions.json to cdb on deploy server https://gerrit.wikimedia.org/r/124762
Change 124763 had a related patch set uploaded by BryanDavis: Create symlink for compile-wikiversions in /usr/local/bin https://gerrit.wikimedia.org/r/124763
Change 124763 merged by Alexandros Kosiaris: Create symlink for compile-wikiversions in /usr/local/bin https://gerrit.wikimedia.org/r/124763
Change 124762 merged by jenkins-bot: Compile wikiversions.json to cdb on deploy server https://gerrit.wikimedia.org/r/124762
Looks like it did the right thing: $ ll /a/common/wmf-config/ExtensionMessages-1.23wmf2*.php -rw-rw-r-- 1 bd808 wikidev 24179 Apr 10 16:16 /a/common/wmf-config/ExtensionMessages-1.23wmf20.php -rw-rw-r-- 1 reedy wikidev 30070 Apr 10 16:16 /a/common/wmf-config/ExtensionMessages-1.23wmf21.php -rw-rw-r-- 1 reedy wikidev 30288 Apr 10 16:16 /a/common/wmf-config/ExtensionMessages-1.23wmf22.php
