• Introduction
  • Docker Image
  • License and Copyright
• Building
• Running
  • Running Inside OpenShift
  • Running Outside OpenShift
• Configuring OpenShift
  • Pod Volumes
  • Project Config Map
  • Config Map Entry Schema
  • Example
  • Try It!
• Configuring External Endpoints To Monitor
• Prometheus Endpoints
• Jolokia Endpoints
• Generic JSON Endpoints
• Environment Variables
• Metric Tags
  • Pod Label Tags

Hawkular OpenShift Agent is a Hawkular feed implemented in the Go Programming Language. Its main purpose is to monitor a node within an OpenShift environment, collecting metrics from Prometheus and/or Jolokia endpoints deployed in one or more pods within the node. It can also be used to collect metrics from endpoints outside of OpenShift. The agent can be deployed inside the OpenShift node it is monitoring, or outside of OpenShift completely.

Watch this quick 10-minute demo to see the agent in action.

Note that the agent does not collect or store inventory at this time - this is strictly a metric collection and storage agent that integrates with Hawkular Metrics.

Docker Image

License and Copyright

   Copyright 2016-2017 Red Hat, Inc. and/or its affiliates
   and other contributors.

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

To build Hawkular OpenShift Agent:

• Clone this repository inside a GOPATH. These instructions will use the example GOPATH of "/source/go/hawkular-openshift-agent" but you can use whatever you want. Just change the first line of the below instructions to use your GOPATH.

• Install Glide - a Go dependency management tool that Hawkular OpenShift Agent uses to build itself

• Tell Glide to install the Hawkular OpenShift Agent dependencies

• Build Hawkular OpenShift Agent

• At this point you can run the Hawkular OpenShift Agent tests

Running Inside OpenShift

oc cluster up --metrics

cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent
make docker

make openshift-deploy

oc create -f deploy/openshift/hawkular-openshift-agent-configmap.yaml -n default
oc process -f deploy/openshift/hawkular-openshift-agent.yaml | oc create -n default -f -
oc adm policy add-cluster-role-to-user hawkular-openshift-agent system:serviceaccount:default:hawkular-openshift-agent

oc delete all,secrets,sa,templates,configmaps,daemonsets,clusterroles --selector=metrics-infra=agent -n default
oc delete clusterroles hawkular-openshift-agent # this is only needed until this bug is fixed: https://github.com/openshift/origin/issues/12450

make openshift-undeploy

Running Outside OpenShift

cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent
make install
make run

cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent
make install
${GOPATH}/bin/hawkular-openshift-agent -config <your-config-file>

K8S_TOKEN=`oc whoami -t` ${GOPATH}/bin/hawkular-openshift-agent -config config.yaml

When Hawkular OpenShift Agent is monitoring resources running on an OpenShift node, it looks at volumes and config maps to know what to monitor. In effect, the pods tell Hawkular OpenShift Agent what to monitor, and Hawkular OpenShift Agent does it. (Note that where "OpenShift" is mentioned, it is normally synonymous with "Kubernetes" because Hawkular OpenShift Agent is really interfacing with the underlying Kubernetes software that is running in OpenShift)

One caveat must be mentioned up front. Hawkular OpenShift Agent will only monitor a single OpenShift node. If you want to monitor multiple OpenShift nodes, you must run one Hawkular OpenShift Agent process per node. The agent can be deployed as a daemonset to make this easier.

There are two features in OpenShift that Hawkular OpenShift Agent takes advantage of when it comes to configuring what Hawkular OpenShift Agent should be monitoring - one is pod volumes and the second is project config maps.

Pod Volumes

Project Config Map

Config Map Entry Schema

Example

kind: ConfigMap
apiVersion: v1
metadata:
  name: my-web-pod-config
  namespace: my-project
data:
  hawkular-openshift-agent: |
    endpoints:
    - type: prometheus
      collection_interval: 60s
      protocol: http
      port: 8080
      path: /metrics
      metrics:
      - name: the_metric_to_collect

oc create -f my-web-pod-config-map.yaml

oc replace -f my-web-pod-config-map.yaml

...
spec:
  volumes:
    - name: hawkular-openshift-agent
      configMap:
        name: my-web-pod-config
...

Try It!

cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent/examples/jolokia-wildfly-example
make openshift-deploy

Hawkular OpenShift Agent is being developed primarily for running within an OpenShift environment. However, strictly speaking, it does not need to run in or monitor OpenShift. You can run Hawkular OpenShift Agent within your own VM, container, or bare metal and configure it to collect metrics from external endpoints you define in the main config.yaml configuration file.

As an example, suppose you want Hawkular OpenShift Agent to scrape metrics from your Prometheus endpoint running at "http://yourcorp.com:9090/metrics" and store those metrics in Hawkular Metrics. You can add an endpoints section to your Hawkular OpenShift Agent’s configuration file pointing to that endpoint which enables Hawkular OpenShift Agent to begin monitoring that endpoint as soon as Hawkular OpenShift Agent starts. The endpoints section of your YAML configuration file could look like this:

A full Prometheus endpoint configuration can look like this:

Some things to note about configuring your Prometheus endpoints:

• Prometheus endpoints can serve metric data in either text or binary form. The agent automatically supports both - there is no special configuration needed. The agent will detect what form the data is in when the endpoint returns it and parses the data accordingly.

• If this is an endpoint running in an OpenShift pod (and thus this endpoint configuration is found in a config map), you do not specify a full URL; instead you specify the protocol, port, and path and the pod’s IP will be used for the hostname. URLs are only specified for those endpoints running outside of OpenShift.

• The agent supports either http or https endpoints. If the Prometheus endpoint is over the https protocol, you must configure the agent with a certificate and private key. This is done by either starting the agent with the two environment variables HAWKULAR_OPENSHIFT_AGENT_CERT_FILE and HAWKULAR_OPENSHIFT_AGENT_PRIVATE_KEY_FILE or via the Identity section of the agent’s configuration file:

• The credentials are optional. If the Prometheus endpoint does require authorization, you can specify the credentials as either a bearer token or a basic username/password. To avoid putting this information in plaintext you can specify an OpenShift secret name that the agent will use to obtain the credentials (e.g. a password value can be "secret:my-secret/password" which tells the agent to look up the password in the "password" entry found within the OpenShift secret named "my-secret").

• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default. This metric ID will be prefixed with the "metric_id_prefix" if one is defined in the collector section of the agent’s global configuration file.

Prometheus supports the ability to label metrics such as the below:

To Prometheus, each metric with a unique combination of labels is separate time series data. To define separate time series data in Hawkular-Metrics, the agent will create a separate metric definition per label combination. By default, if the agent sees Prometheus data with labels, it will create metric definitions in the format:

You can customize these metric definitions that are created by using ${label-key} tokens in a custom metric ID per the below:

This would create the following metrics in Hawkular:

A full Jolokia endpoint configuration can look like this:

Some things to note about configuring your Jolokia endpoints:

• If this is an endpoint running in an OpenShift pod (and thus this endpoint configuration is found in a config map), you do not specify a full URL; instead you specify the protocol, port, and path and the pod’s IP will be used for the hostname. URLs are only specified for those endpoints running outside of OpenShift.

• The agent supports either http or https endpoints. If the Jolokia endpoint is over the https protocol, you must configure the agent with a certificate and private key. This is done by either starting the agent with the two environment variables HAWKULAR_OPENSHIFT_AGENT_CERT_FILE and HAWKULAR_OPENSHIFT_AGENT_PRIVATE_KEY_FILE or via the Identity section of the agent’s configuration file:

• The credentials are optional. If the Jolokia endpoint does require authorization, you can specify the credentials as either a bearer token or a basic username/password. To avoid putting this information in plaintext you can specify an OpenShift secret name that the agent will use to obtain the credentials (e.g. a password value can be "secret:my-secret/password" which tells the agent to look up the password in the "password" entry found within the OpenShift secret named "my-secret").

• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default.

• You must specify a metric’s "type" as either "counter" or "gauge".

• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default. This metric ID will be prefixed with the "metric_id_prefix" if one is defined in the collector section of the agent’s global configuration file.

• A metric "name" follows a strict format. First is the full MBean name (e.g. java.lang:type=Threading) followed by a hash (#) followed by the attribute that contains the metric data (e.g. ThreadCount). If the attribute is a composite attribute, then you must append a second hash followed by the composite attribute’s subpath name which contains the actual metric value. For example, java.lang:type=Memory#HeapMemoryUsage#used will collect the used value of the composite attribute HeapMemoryUsage from the MBean java.lang:type=Memory.

• An attribute must be a numeric value convertible to a floating point number. However, if the value is such a numeric in the form of a string, the agent will parse the attribute value string and convert it to a floating point number for storage. Also, if the value is a boolean, it will be converted to a 1.0 if the boolean value is true and a 0.0 if the boolean value is false.

Jolokia allows for querying of multiple MBeans and multiple attributes using JMX ObjectName patterns. The Hawkular OpenShift Agent can take advantage of this to allow for easier configuration of the metrics you want to collect from your Jolokia endpoints. Here are some of the scenarios that the agent supports.

• Collect an attribute from multiple MBeans. This will collect the "Count" attribute from all MBeans that match the query in "name":

Note that in your metric ID and description you can use ${x} tokens where the "x" is the MBean name key that was set to the "*" wildcard in the name. You can have multiple wildcards in your metric name such as:

• Collect multiple attributes from a single MBean. This will collect all attributes from the named MBean:

Note that when the attribute is a wildcard, you can use the ${1} token in the ID where you want the name of the attribute in the metric ID. The same with description.

• Collect all data from a composite attribute from a single MBean. This will collect all data for the named composite attribute from the named MBean:

Note that when the inner path of a composite attribute is a wildcard, you can use the ${2} token in the ID where you want the name of the inner path of the composite attribute in the metric ID. The same with description.

• Collect multiple attributes from multiple MBeans:

Hawkular OpenShift Agent has the ability to read any endpoint that exposes metrics in JSON format. So long as the endpoint serves a valid JSON document, the agent can scrape the metrics from that JSON data. One common use-case for this is GoLang’s expvar feature. A GoLang program can expose its metric data over HTTP in JSON format via expvar (see the GoLang expvar documentation for more details) - the agent can read this GoLang expvar endpoint to obtain that metric data. A full JSON endpoint configuration can look like this:

Some things to note about configuring your JSON endpoints:

• If this is an endpoint running in an OpenShift pod (and thus this endpoint configuration is found in a config map), you do not specify a full URL; instead you specify the protocol, port, and path and the pod’s IP will be used for the hostname. URLs are only specified for those endpoints running outside of OpenShift.

• The agent supports either http or https endpoints. If the JSON endpoint is over the https protocol, you must configure the agent with a certificate and private key. This is done by either starting the agent with the two environment variables HAWKULAR_OPENSHIFT_AGENT_CERT_FILE and HAWKULAR_OPENSHIFT_AGENT_PRIVATE_KEY_FILE or via the Identity section of the agent’s configuration file:

• The credentials are optional. If the JSON endpoint does require authorization, you can specify the credentials as either a bearer token or a basic username/password. To avoid putting this information in plaintext you can specify an OpenShift secret name that the agent will use to obtain the credentials (e.g. a password value can be "secret:my-secret/password" which tells the agent to look up the password in the "password" entry found within the OpenShift secret named "my-secret").

• If no metrics are specified, all valid metrics in the JSON data will be collected.

• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default with labels appended to it (see more below).

• You must specify a metric’s "type" as either "counter" or "gauge".

• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default. This metric ID will be prefixed with the "metric_id_prefix" if one is defined in the collector section of the agent’s global configuration file.

• A metric "name" is the name of the top-level JSON element.

The JSON data can include sub-elements under the named top-level elements. In this case, the sub-element names will be used as tags and appended to the metric name enclosed in curly braces. The agent can support maps nested at any level. An example is the best way to illustrate this. Suppose the JSON metric data representing a web application’s average response times looks like this:

The metric names are always found at the top-level of the JSON data. So in this example, the metric being collected has the base metric name "response.times".

But notice this data has a map sub-element under the top element indicating this "metric" is really a collection of related metrics (for those familiar with Prometheus, we can call this the "metric family name"). This map has two entries with key names "GET" and "POST". Under each of these are more maps, each keyed with a web application request path (e.g. "/index.html" or "/admin/query-db") whose values are the actual numeric metric data - the average response time for each requested endpoint.

The Hawkular OpenShift Agent will be able to collect and store this family of metrics named "response.times". It reads the child map entries and considers each map key a label value which will be appended to the metric name to build the metric ID and will be used as a tag on the metric when storing in Hawkular Metrics. The agent recursively decends the JSON tree building new labels until it reaches the numeric metric data.

For this example, the agent will end up storing in Hawkular Metrics the following four individual metrics:

Many of the agent’s configuration settings can optionally be set via environment variables. If one of the environment variables below are set, they serve as the default value for its associated YAML configuration setting. The following are currently supported:

Metric data can be tagged with additional metadata called tags. A metric tag is a simple name/value pair. Tagging metrics allows you to further describe the metric and allows you to query for metric data based on tag queries. For more information on tags and querying tagged metric data, see the Hawkular-Metrics documentation.

Hawkular OpenShift Agent can be configured to attach custom tags to the metrics it collects. There are three places where you can define custom tags in Hawkular OpenShift Agent:

• In the agent’s global configuration (all tags defined here will be attached to all metrics stored by the agent)

• In an endpoint configuration (all tags defined here will be attached to all metrics collected from that endpoint)

• In a metric configuration (all tags defined here will only be attached to the metric)

To define global tags, you would add a tags section under collector in the agent’s global configuration file. The following configuration snippet will tell the agent to attach the tags "my-tag" (with value "my-tag-value") and "another-tag" (with value "another-tag-value") to each and every metric the agent collects.

To define endpoint tags (that is, tags that will be attached to every metric collected from the endpoint), you would add a tags section within the endpoint configuration. The following configuration snippet will tell the agent to attach the tags "my-endpoint-tag" and "my-other-endpoint-tag" to every metric that is collected from this specific Jolokia endpoint:

To define tags on individual metrics, you would add a tags section within a metric configuration. The following configuration snippet will tell the agent to attach the tags "my-metric-tag" and "my-other-metric-tag" to the metric named "java.lang.type=Threading#ThreadCount" that is collected from this specific Jolokia endpoint:

Tag values can be defined with token expressions in the form of ${var} or $var where var is either an agent environment variable name (only supported in global tags) or, if the tag definition is found in an OpenShift config map entry, one of the following:

For example:

Pod Label Tags

collector:
  pod_label_tags_prefix: labels.