Last modified: 2014-08-20 19:35:42 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T69870, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 67870 - wfShellExec() cuts off stdout at multiples of 8192 bytes
wfShellExec() cuts off stdout at multiples of 8192 bytes
Status: RESOLVED FIXED
Product: MediaWiki
Classification: Unclassified
General/Unknown (Other open bugs)
1.24rc
All All
: Normal normal (vote)
: ---
Assigned To: Nobody - You can work on this!
: hhvm
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-07-11 18:04 UTC by Niklas Laxström
Modified: 2014-08-20 19:35 UTC (History)
8 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
Output returned by wfShellExec (8.06 KB, text/plain)
2014-07-12 16:38 UTC, Niklas Laxström
Details

Description Niklas Laxström 2014-07-11 18:04:46 UTC
It doesn't seem to be related to any memory / time limits. The output of the script is just truncated. It works when run under php. (php below is aliased to hhvm)

php extensions/Translate/scripts/createMessageIndex.php --memory-limit=max
[e18f17bf] [no req]   Exception from line 113 of /www/dev.translatewiki.net/w/extensions/Translate/ffs/PythonSingleFFS.php: Failed to decode python file /resources/projects/pywikibot/add_text.py
Backtrace:
#0 /www/dev.translatewiki.net/w/extensions/Translate/ffs/PythonSingleFFS.php(86): PythonSingleFFS->parseFile()
#1 /www/dev.translatewiki.net/w/extensions/Translate/messagegroups/FileBasedMessageGroup.php(52): PythonSingleFFS->read(string)
#2 /www/dev.translatewiki.net/w/extensions/Translate/messagegroups/MessageGroupBase.php(81): FileBasedMessageGroup->load(string)
#3 /www/dev.translatewiki.net/w/extensions/Translate/messagegroups/MessageGroupBase.php(183): MessageGroupBase->getDefinitions()
#4 /www/dev.translatewiki.net/w/extensions/Translate/utils/MessageIndex.php(186): MessageGroupBase->getKeys()
#5 /www/dev.translatewiki.net/w/extensions/Translate/utils/MessageIndex.php(119): MessageIndex->checkAndAdd(array, FileBasedMessageGroup)
#6 /www/dev.translatewiki.net/w/extensions/Translate/scripts/createMessageIndex.php(31): MessageIndex->rebuild()
#7 /www/dev.translatewiki.net/w/maintenance/doMaintenance.php(109): CreateMessageIndex->execute()
#8 /www/dev.translatewiki.net/w/extensions/Translate/scripts/createMessageIndex.php(36): include(string)
#9 {main}

The relevant code:

		$command = wfEscapeShellArg( "import json; execfile(\"$filename\"); " .
			"print json.dumps(msg)" );
		$json = wfShellExec( "python -c $command" );

		$parsed = FormatJson::decode( $json, true );
		if ( !is_array( $parsed ) ) {
			throw new MWException( "Failed to decode python file $filename" );
		}
Comment 1 Ori Livneh 2014-07-11 21:52:25 UTC
Thanks for the report. Could you var_dump() the value of $json after the wfShellExec() invocation and the value of $parsed after the FormatJson::decode() invocation?
Comment 2 Niklas Laxström 2014-07-12 16:38:52 UTC
Created attachment 15922 [details]
Output returned by wfShellExec

I added the following line after the decode call:
var_dump( 'JSON', $json, 'PARSED', $parsed ); die();

The result is in the attachment. Interesting thing is that the length of the truncated output is 8192 bytes.
Comment 3 Max Semenik 2014-07-18 02:34:50 UTC
It's a general wfShellExec() problem.
Comment 4 Gerrit Notification Bot 2014-07-18 20:10:32 UTC
Change 147635 had a related patch set uploaded by MaxSem:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/147635
Comment 5 Gerrit Notification Bot 2014-07-22 16:58:10 UTC
Change 147635 merged by jenkins-bot:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/147635
Comment 6 Gerrit Notification Bot 2014-07-22 17:55:14 UTC
Change 148419 had a related patch set uploaded by MaxSem:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/148419
Comment 7 Gerrit Notification Bot 2014-08-20 19:35:42 UTC
Change 148419 merged by jenkins-bot:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/148419

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links