Welcome, Guest Login

Support Center

Sending custom application metrics to the Stackdriver system

Last Updated: Nov 17, 2015 10:12AM EST

If you are a Google Cloud Monitoring beta user, please use the Cloud Monitoring custom metrics API. The API described below will not work for Google Cloud Monitoring beta users.

Stackdriver supports ingesting, displaying and alerting on custom metrics sent from your application. To view your metrics after you have sent them, select Custom Metrics under Services in the top navigation. (Available for Trial, Pro and Elite Customers only)

To send custom metrics, take the following steps: 

  1. Create a new API key in your Stackdriver Account Settings
  2. Submit Custom Metrics with the API Key
(Code snippets in this section are written in Python.)
 
To push custom metrics to Stackdriver, you will POST a message to the gateway and include the API key in the HTTP header.
 
Each data point must contain three fields: name, value, and collected_at.  
  • ‘name’ is the name of your metric; you can define this using any taxonomy that helps you organize your data. 
  • ‘value’ is the measurement to record for the data point.  
  • ‘collected_at’ is the time that the data point was collected, in Unix epoch time.  If this value is over two hours before the current time, the measurement will be rejected.


By default, custom metrics are not associated with any instance and can be found under the Custom resource type when creating charts or alerting policies.  


If your custom metric is instead related to a particular instance, you can add an optional 'instance' field to your message with a value of the instance id for your instance (eg, i-1234b3f3). These metrics then show up under the Instance resource. This makes it easier to have one metric name shared across a number of instances to include on a single graph or for alerting. See the example below
 

Example data point

data_point = {
   ‘name’: ‘my_custom_metric’,
   ‘value’: 32,
   #Note: Stackdriver rejects when the collected_at value is older than 2 hours.
   ‘collected_at’: 1325394000
   #Note: Include an instance value if your metric is tied to an instance.
   #'instance': 'i-123a567b',
}

 

The data point is submitted to the gateway as part of a gateway message.  The gateway message includes the timestamp the message is created or submitted, your customer identifier, and the protocol version for the message.  The timestamp is used to disambiguate messages.  Messages with the same timestamp are assumed to be the same (see below).  The ‘timestamp’ field must be an integer.
 
The protocol version defines the schema of the message. The current protocol version (and that which is described in this document) is version 1.

Example gateway message

    sd_post = {
        'timestamp': int(time.time()),
        'proto_version': 1,
        'data': data_point,
        }

 

The final message preparation is defining the HTTP headers.  The gateway protocol requires that messages be JSON-encoded.  The API key is also included in the header.

Example headers

 headers = {
        'content-type': 'application/json',
        'x-stackdriver-apikey': args.apikey
        }
Submit the measurement to Stackdriver by POSTing it to the gateway at https://custom-gateway.stackdriver.com/v1/custom.  In the code fragment below, we use the Python “requests” module for HTTP messaging; it can be installed via ‘pip install requests’ or ‘easy_install requests’.

Example http request

    # Note: this code sample works for Stackdriver customers only. Google Cloud
    # Monitoring beta users should use the Cloud Monitoring custom metrics API
    # here: http://goo.gl/W7ORHb
    import requests
    resp = requests.post('https://custom-gateway.stackdriver.com/v1/custom',
                       data=json.dumps(sd_post),
                       headers=headers)

Putting it all together

 
A complete working example that combines all the pieces we have discussed in shown below.

Note: We will not accept metrics sent more frequently than once per minute. If you need to send metrics at a higher frequency, please read our article on sending aggregated metrics with statsd.

 

    # Note: this code sample works for Stackdriver customers only. Google Cloud 
    # Monitoring beta users should use the Cloud Monitoring custom metrics API
    # here: http://goo.gl/W7ORHb
    import requests
    import json
    import time
    def submit_custom_metric():
        data_point = {
            'name': 'my_custom_metric',
            'value': 3.14159,
            #Note: Stackdriver rejects when the collected_at value is older than 2 hours.             
            'collected_at': 1325394000,
            #Note: Include an instance value if your metric is tied to an instance.
            #'instance': 'i-123a567b',
            }
        gateway_msg = {
            'timestamp': int(time.time()),
            'proto_version': 1,
            'data': data_point,
            }
        headers = {
            'content-type': 'application/json',
            'x-stackdriver-apikey': '4VP31M3RJ5JWN29E1LMSE0'
            }
        resp = requests.post(
            'https://custom-gateway.stackdriver.com/v1/custom',
            data=json.dumps(gateway_msg),
            headers=headers)
        
        if not resp.ok:
            print resp.content
            assert resp.ok, 'Failed to submit custom metric.'
        else:
            print "Published."
    if __name__ == '__main__':
        submit_custom_metric()

Multiple Data Points in a Message

The custom metrics API also supports sending a list of data points in the data portion of the message payload.  Otherwise, everything else from the above is exactly the same.  An example of such a message would be to replace gateway_msg in the last example with the following
gateway_msg = {
        'timestamp': int(time.time()),
        'proto_version': 1,
        'data': [
          {
             'name': 'my_custom_metric1',
             'value': 32,
             #Note: Stackdriver rejects when the collected_at value is older than 2 hours.
             'collected_at': 1325394000,
             #Note: Include an instance value if your metric is tied to an instance.
             #'instance': 'i-123a567b',
          },
          {
             'name': 'my_custom_metric2',
             'value': 37,
             #Note: Stackdriver rejects when the collected_at value is older than 2 hours.
             'collected_at': 1325394000,
             #Note: Include an instance value if your metric is tied to an instance.
             #'instance': 'i-123a567b',
          }
         ]
         }

Note that each point within the data element must have its own (not necessarily unique) name, value, and collected_at. Keep in mind that the policy regarding the age of collected_at still applies.

Libraries

A few libraries exist for sending custom metrics to Stackdriver.  Note that these are contributed by other users and not by Stackdriver
These libraries only work for Stackdriver customers. If you a Google Cloud Monitoring beta user, please use the Cloud Monitoring custom metrics API.

Removing Custom Metrics


If you accidentally send incorrect metric keys, you can remove them from being seen in Stackdriver by following the instructions here.  It's worth noting that this removes the key and not the underlying data so if you begin sending a metric with the same name again, the old data will still be visible. Custom metric data points automatically expire from our system after six weeks. 
https://cdn.desk.com/
false
desk
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete