Welcome, Guest Login

Support Center

Elasticsearch Plugin

Last Updated: Aug 17, 2015 01:06PM EDT

Elasticsearch

 

Elasticsearch is an open-source search server, based on the Lucene search library. It runs in a Java virtual machine on top of a number of operating systems. For more information, visit http://elasticsearch.org

Discovery

As you install the Elasticsearch plugin on your instances, Stackdriver will populate the Elasticsearch service page with inventory and metrics. 

 

 

 

Configuring Stackdriver to monitor Elasticsearch

 

Use the following instructions to configure the Elasticsearch plugin for the Stackdriver agent.

 

Prerequisites

 

The Elasticsearch plugin requires the latest version of the Stackdriver Agent, as well as the yajl library (Yet Another JSON Library http://lloyd.github.io/yajl/).

 

To update the agent, follow these instructions.

 

To get yajl on Red Hat Enterprise Linux, CentOS, Amazon Linux and derivative systems run

$ yum install yajl

To get yajl on Ubuntu (before 14.04) and Debian systems run:

$ apt-get install libyajl1

To get yajl on Ubuntu 14.04 systems, run

$ apt-get install libyajl2
 

Enabling the Elasticsearch monitoring plugin

 

Place the following code in a config file at “/opt/stackdriver/collectd/etc/collectd.d/elasticsearch.conf”:

NOTE: For elasticsearch versions earlier than 1.0, replace the stats URL in the config below with http://localhost:9200/_cluster/nodes/_local/stats/.
 

LoadPlugin match_regex
LoadPlugin target_set
PreCacheChain "RenameES"
<Chain "RenameES">
  <Rule "rename_es">
    <Match regex>
      Plugin "^curl_json$"
      PluginInstance "elasticsearch"
    </Match>
    <Target "set">
      Plugin "elasticsearch"
      PluginInstance "localhost"
    </Target>
  </Rule>
</Chain>

LoadPlugin "curl_json"
<Plugin curl_json>
<URL "http://localhost:9200/_nodes/_local/stats?all=true">
    Instance "elasticsearch"
    <Key "nodes/*/indices/docs/count">
      Type "gauge"
    </Key>
    <Key "nodes/*/indices/docs/deleted">
      Type "gauge"
    </Key>
    <Key "nodes/*/indices/store/size_in_bytes">
      Type "gauge"
    </Key>
    <Key "nodes/*/indices/indexing/index_total">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/indexing/index_time_in_millis">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/indexing/delete_total">
          Type "counter"
      </Key>
      <Key "nodes/*/indices/indexing/delete_time_in_millis">
          Type "counter"
      </Key>
      <Key "nodes/*/indices/get/total">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/get/time_in_millis">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/get/exists_total">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/get/exists_time_in_millis">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/get/missing_total">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/get/missing_time_in_millis">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/search/query_total">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/search/query_time_in_millis">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/search/fetch_total">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/search/fetch_time_in_millis">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/filter_cache/memory_size_in_bytes">
      Type "gauge"
    </Key>
    <Key "nodes/*/indices/filter_cache/evictions">
      Type "counter"
    </Key>
    <Key "nodes/*/indices/fielddata/memory_size_in_bytes">
      Type "gauge"
    </Key>
    <Key "nodes/*/indices/fielddata/evictions">
      Type "counter"
    </Key>
    <Key "nodes/*/process/open_file_descriptors">
      Type "gauge"
    </Key>
    <Key "nodes/*/jvm/mem/heap_used_in_bytes">
      Type "gauge"
    </Key>
    <Key "nodes/*/jvm/mem/heap_committed_in_bytes">
      Type "gauge"
    </Key>
    <Key "nodes/*/jvm/mem/non_heap_used_in_bytes">
      Type "gauge"
    </Key>
    <Key "nodes/*/jvm/mem/non_heap_committed_in_bytes">
      Type "gauge"
    </Key>
    <Key "nodes/*/jvm/threads/count">
      Type "gauge"
    </Key>
    <Key "nodes/*/jvm/threads/peak_count">
      Type "gauge"
    </Key>
    <Key "nodes/*/jvm/gc/collection_count">
      Type "counter"
    </Key>
    <Key "nodes/*/transport/rx_size_in_bytes">
      Type "counter"
    </Key>
    <Key "nodes/*/transport/tx_size_in_bytes">
      Type "counter"
    </Key>
    <Key "nodes/*/transport/server_open">
      Type "gauge"
    </Key>
    <Key "nodes/*/http/current_open">
      Type "gauge"
    </Key>
  </URL>
   <URL "http://localhost:9200/_cluster/health">
      Instance "elasticsearch"
      <Key "number_of_nodes">
        Type "gauge"
      </Key>
      <Key "number_of_data_nodes">
        Type "gauge"
      </Key>
        <Key "active_primary_shards">
                Type "gauge"
        </Key>
        <Key "active_shards">
                Type "gauge"
        </Key>
        <Key "relocating_shards">
                Type "gauge"
        </Key>
        <Key "initializing_shards">
                Type "gauge"
        </Key>
        <Key "unassigned_shards">
                Type "gauge"
        </Key>
        </URL>
</Plugin>

 

After adding the file, you will need to restart the monitoring agent by running one of the following commands:

  • Debian and Ubuntu:  sudo /etc/init.d/stackdriver-agent restart
  • Other Linux distributions:  sudo /sbin/service stackdriver-agent restart

 

What we monitor

 

Once the plugin is enabled, you will be able to monitor metrics and create alerting policies related to Elasticsearch within the Stackdriver interface.

 

For Elasticsearch, we monitor these metrics:

  • Search Requests (count/s)

    • How many Fetch and Query requests came in per second in the last interval

  • Search Time (ms/s)

    • How many milliseconds were spent on Fetch and Query searches per second of the last intervale

  • Get Requests (count/s)

    • How many Get/Exists/Missing requests came in per second in the last interval

  • Get Time (ms/s)

    • How many milliseconds were spent on Get/Exists/Missing requests per second of the last interval

  • Indexing Requests (count/s)

    • How many Index and Delete requests came in per second in the last interval

  • Indexing Time (ms/s)

    • How many milliseconds were spent on Index and Delete requests per second of the last intervale

  • Cache Sizes (bytes)
    • How big are the filter and field caches

  • Cache Evictions (count/s)

    • How many field and filter cache evictions took place per second

  • Store Size (bytes)
    • How big is the elasticsearch index
  • Network Usage (bytes/s)

    • Inbound and outbound network usage in bytes per second

  • Network Connections (count)

    • How many http and transport-level network connections are open when sampled

  • JVM Metrics

    • Information on threads, memory usage, and garbage collections within the ES JVM

  • Open Files (count)

    • How many file descriptors are opened by the Elasticsearch process

  • Document Count (count)
    • How many documents and deleted documents are in the ES
 
https://cdn.desk.com/
false
desk
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete