Last modified: 2014-02-15 00:20:39 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T63384, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 61384 - Elasticsearch ganglia monitor broken for Logstash due to lack of stats groups
Elasticsearch ganglia monitor broken for Logstash due to lack of stats groups
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
Logstash (Other open bugs)
wmf-deployment
All All
: High normal (vote)
: ---
Assigned To: Bryan Davis
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-02-14 18:23 UTC by Bryan Davis
Modified: 2014-02-15 00:20 UTC (History)
5 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Bryan Davis 2014-02-14 18:23:58 UTC
The kibana interface doesn't tag searches with stats groups [0] and the ganglia monitoring script assumes otherwise:

[PYTHON] Can't call the metric_init function in the python module [elasticsearch
_monitoring].

Traceback (most recent call last):
  File "/usr/lib/ganglia/python_modules/elasticsearch_monitoring.py", line 511,
in metric_init
    for group in load(url)['_all']['total']['search']['groups']:
KeyError: 'groups'


When queried on the logstash elasticsearch nodes the relevant section of the _stats/search request looks like this:

    "_all" : {
      "primaries" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 10317,
          "query_time" : "1.6h",
          "query_time_in_millis" : 5915635,
          "query_current" : 0,
          "fetch_total" : 7731,
          "fetch_time" : "23.5s",
          "fetch_time_in_millis" : 23525,
          "fetch_current" : 0
        }
      },
      "total" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 19499,
          "query_time" : "3.1h",
          "query_time_in_millis" : 11201005,
          "query_current" : 0,
          "fetch_total" : 14517,
          "fetch_time" : "42.8s",
          "fetch_time_in_millis" : 42819,
          "fetch_current" : 0
        }
      }
    }


[0]: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search.html#stats-groups
Comment 1 Bryan Davis 2014-02-14 18:33:41 UTC
Nik says that it should be fine to just skip these metrics if not present in the data. Should be a pretty simple "if 'groups' in ..." check to add.
Comment 2 Gerrit Notification Bot 2014-02-14 22:05:03 UTC
Change 113471 had a related patch set uploaded by BryanDavis:
Make elasticsearch ganglia monitor compatible with logstash

https://gerrit.wikimedia.org/r/113471
Comment 3 Gerrit Notification Bot 2014-02-15 00:05:29 UTC
Change 113471 merged by Ori.livneh:
Make elasticsearch ganglia monitor compatible with logstash

https://gerrit.wikimedia.org/r/113471

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links