Last modified: 2014-04-16 20:18:26 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T65222, the corresponding Phabricator task for complete and up-to-date bug report information.
First Last Prev Next    This bug is not in your last search results.
Bug 63222 - Hive queries can bring load on cluster slaves > #CPUs
Hive queries can bring load on cluster slaves > #CPUs
Status: NEW
Product: Analytics
Classification: Unclassified
General/Unknown (Other open bugs)
unspecified
All All
: Normal normal
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency tree / graph
 
Reported: 2014-03-28 11:09 UTC by christian
Modified: 2014-04-16 20:18 UTC (History)
3 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---

Attachments
Load graph for analytics 1018 while running the queries (2014-03-28) (21.66 KB, application/x-msmetafile)
2014-03-28 11:09 UTC, christian 		Details
Load graph for analytics 1019 while running the queries (2014-03-28) (21.42 KB, application/x-msmetafile)
2014-03-28 11:11 UTC, christian 		Details
SQL for first query (1.27 KB, application/octet-stream)
2014-03-28 11:12 UTC, christian 		Details
SQL for second query (1.02 KB, application/octet-stream)
2014-03-28 11:12 UTC, christian 		Details
SQL for third query (1.35 KB, application/octet-stream)
2014-03-28 11:13 UTC, christian 		Details
Add an attachment (proposed patch, testcase, etc.)

Description christian 2014-03-28 11:09:54 UTC 
Created attachment 14954 [details]
Load graph for analytics 1018 while running the queries (2014-03-28)

When running three hive queries on a week's worth of mobile request data,
load on the Hadoop cluster nodes rises above the number of CPUs.

Should we limit the resources that Hive/Hadoop can take on those machines?
Comment 1 Bingle 2014-03-28 11:10:22 UTC 
Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/cards/1503
Comment 2 christian 2014-03-28 11:11:07 UTC 
Created attachment 14955 [details]
Load graph for analytics 1019 while running the queries (2014-03-28)
Comment 3 christian 2014-03-28 11:12:34 UTC 
Created attachment 14956 [details]
SQL for first query
Comment 4 christian 2014-03-28 11:12:53 UTC 
Created attachment 14957 [details]
SQL for second query
Comment 5 christian 2014-03-28 11:13:11 UTC 
Created attachment 14958 [details]
SQL for third query
Comment 6 Toby Negrin 2014-03-28 18:26:54 UTC 
Agree -- perhaps we need to tune the number of slots for mappers/reducers?
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.

Navigation
Links