Skip to main content

cAdvisor w/ Prometheus API

Connector dependencies​

The following monitoring connectors will be installed when you install the cAdvisor w/ Prometheus API connector through the Configuration > Connectors > Monitoring Connectors menu:

Pack assets​

Templates​

The Monitoring Connector cAdvisor w/ Prometheus API brings 4 host templates:

  • Cloud-Prometheus-cAdvisor-Api-custom: monitors, from a single host, all containers that match each service's filters.
  • Cloud-Prometheus-cAdvisor-Api-Container-custom: monitors one container running outside Kubernetes (Docker, Podman).
  • Cloud-Prometheus-cAdvisor-Api-Kubernetes-Container-custom: monitors one container of a Kubernetes pod.
  • Cloud-Prometheus-cAdvisor-Api-Pod-custom: monitors one Kubernetes pod as a whole.

The last three host templates check that their pod or container is present in the cAdvisor metrics, and are designed to be applied by host discovery.

The connector brings the following service templates (sorted by the host template they are attached to):

Service AliasService TemplateService Description
Container-CpuCloud-Prometheus-cAdvisor-Cpu-Api-customCheck container CPU usage
Container-LoadCloud-Prometheus-cAdvisor-Load-Api-customCheck container load
Container-MemoryCloud-Prometheus-cAdvisor-Memory-Api-customCheck container memory usage
Container-StorageCloud-Prometheus-cAdvisor-Storage-Api-customCheck container storage usage
Container-Task-StateCloud-Prometheus-cAdvisor-Task-State-Api-customCheck the state of container tasks

The services listed above are created automatically when the Cloud-Prometheus-cAdvisor-Api-custom host template is used.

Discovery rules​

Host discovery​

Rule nameDescription
Prometheus cAdvisorDiscover pods and containers (Kubernetes or standalone) from cAdvisor metrics stored in Prometheus

The discovery lists the pods and containers from the container_last_seen metric and applies a host template according to the type of each discovered resource:

Discovered resourceHost templateHost name
Kubernetes podCloud-Prometheus-cAdvisor-Api-Pod-customnamespace_pod
Container of a Kubernetes podCloud-Prometheus-cAdvisor-Api-Kubernetes-Container-customnamespace_pod_container
Container running outside KubernetesCloud-Prometheus-cAdvisor-Api-Container-customcontainer

The discovery also fills in the macros of the hosts: the connection to Prometheus, and the CONTAINER_FILTER, POD_FILTER and NAMESPACE_FILTER filters that select the pod or container of each host.

More information about discovering hosts automatically is available on the dedicated page, in particular about scheduling the discovery job to add the hosts of new pods as your Kubernetes cluster evolves.

Collected metrics & status​

Here is the list of services for this connector, detailing all metrics and statuses linked to each service.

NameUnit
pod_container#container.cpu.utilization.percentage%
pod_container#container.cpu.throttled.percentage%

Applies to the following service templates: Cloud-Prometheus-cAdvisor-Cpu-Api-custom, Cloud-Prometheus-cAdvisor-Container-Cpu-Api-custom

To obtain this new metric format, include --use-new-perfdata in the EXTRAOPTIONS service macro.

The metric instance depends on the resource type:

  • Container in a pod: pod name + container name (web-1_nginx or web-1~nginx, depending on the service).
  • Pod: pod name.
  • Container outside Kubernetes: container name (haproxy). The storage metrics also contain the device (for example web-1~nginx~/dev/vda1).

Prerequisites​

Prometheus and cAdvisor​

The connector queries the API of a Prometheus server that collects the cAdvisor metrics, from one of the following sources:

  • the cAdvisor built into the kubelet (Kubernetes): the /metrics/cadvisor endpoint of each node, usually collected by the kubelet job of the Prometheus Operator. It exports the CPU and memory metrics of the pods and containers, but not the load (container_cpu_load_average_10s) and the task state (container_tasks_state) metrics, nor the disk usage of the containerd containers.
  • A standalone cAdvisor: on a Docker or Podman host, or deployed in Kubernetes as a DaemonSet. It is required to use the load, storage and task state services.

The default filters use the container, pod and namespace labels set by the kubelet since Kubernetes 1.16 (OpenShift 4.3). On older platforms, these labels are named container_name and pod_name: set the filters accordingly, for example --container='container_name!~"|.*POD.*"' --pod='pod_name=~".*"'.

A standalone cAdvisor deployed in Kubernetes does not set these labels. It only sets the container runtime labels (for example container_label_io_kubernetes_pod_name). In that case, either you can either:

  • Use these label names in your filters.
  • Configure Prometheus to copy them to container, pod and namespace (for example with metric_relabel_configs).

Best practice: naming your containers​

By default, Docker gives a container a new random name each time it is recreated. Centreon then creates new metrics and a new RRD file in /var/lib/centreon/metrics, building up stale data.

Always assign a fixed name to each monitored container:

# Command line
docker run -d --name my-nginx nginx

# Docker Compose
services:
  web:
    image: nginx
    container_name: my-nginx

If you are using Kubernetes, the pods of a StatefulSet keep a stable name (my-service-0, my-service-1, etc.), as do the pods created directly (kind: Pod). Pods of other controllers (…) get a new name each time they are recreated. You can configure host discovery to add a host for each new name and delete the old one. However, metrics collected under two different names are not linked or displayed together.

Installing the monitoring connector​

Pack​

The installation procedures for monitoring connectors are slightly different depending on whether your license is offline or online.

  1. If the platform uses an online license, you can skip the package installation instruction below as it is not required to have the connector displayed within the Configuration > Connectors > Monitoring Connectors menu. If the platform uses an offline license, install the package on the central server with the command corresponding to the operating system's package manager:
dnf install centreon-pack-cloud-prometheus-cadvisor-api
  1. Whatever the license type (online or offline), install the cAdvisor w/ Prometheus API connector through the Configuration > Connectors > Monitoring Connectors menu.

Plugin​

Since Centreon 22.04, you can benefit from the 'Automatic plugin installation' feature. When this feature is enabled, you can skip the installation part below.

You still have to manually install the plugin on the poller(s) when:

  • Automatic plugin installation is turned off
  • You want to run a discovery job from a poller that doesn't monitor any resource of this kind yet

More information in the Installing the plugin section.

Use the commands below according to your operating system's package manager:

dnf install centreon-plugin-Cloud-Prometheus-cAdvisor-Api

Using the monitoring connector​

Using a host template provided by the connector​

  1. Log into Centreon and add a new host through Configuration > Hosts.
  2. Fill in the Name, Alias & IP Address/DNS fields according to your resource's settings.
  3. Apply the Cloud-Prometheus-cAdvisor-Api-custom template to the host. A list of macros appears. Macros allow you to define how the connector will connect to the resource, and to customize the connector's behavior.
  4. Fill in the macros you want. Some macros are mandatory.
MacroDescriptionDefault valueMandatory
PROMETHEUSAPIHOSTNAMEPrometheus hostnameX
PROMETHEUSAPIPROTOSpecify https if neededhttp
PROMETHEUSAPIPORTAPI port9090
PROMETHEUSAPIURLAPI URL path/api/v1
EXTRAOPTIONSAny extra option you may want to add to every command (a --verbose flag for example). All options are listed here.
  1. Deploy the configuration. The host appears in the list of hosts, and on the Resources Status page. The command that is sent by the connector is displayed in the details panel of the host: it shows the values of the macros.

Using a service template provided by the connector​

  1. If you have used a host template and checked Create Services linked to the Template too, the services linked to the template have been created automatically, using the corresponding service templates. Otherwise, manually create the services you want and apply a service template to them.
  2. Fill in the macros you want (e.g. to change the thresholds for the alerts). Some macros are mandatory (see the table below).

The service templates of the Cloud-Prometheus-cAdvisor-Api-custom host template select the containers with their own CONTAINER and POD macros. The default values of these macros use the label names of Kubernetes 1.15 and older: with Kubernetes 1.16 and newer, use for example container!~"|.*POD.*" and pod=~".*" (see Prerequisites).

The service templates of the Cloud-Prometheus-cAdvisor-Api-Container-custom, Cloud-Prometheus-cAdvisor-Api-Kubernetes-Container-custom and Cloud-Prometheus-cAdvisor-Api-Pod-custom host templates select their pod or container with the CONTAINER_FILTER, POD_FILTER and NAMESPACE_FILTER macros of their host.

Cloud-Prometheus-cAdvisor-Cpu-Api-custom

MacroDescriptionDefault valueMandatory
CONTAINERFilter on the container (must be a PromQL filter)container_name!~".POD."
PODFilter on the pod (must be a PromQL filter)pod_name=~".*"
WARNINGUSAGEWarning threshold on the CPU usage (%)80
CRITICALUSAGECritical threshold on the CPU usage (%)90
WARNINGTHROTTLEDWarning threshold on the CPU throttling (%)10
CRITICALTHROTTLEDCritical threshold on the CPU throttling (%)20
EXTRAOPTIONSAny extra option you may want to add to the command (a --verbose flag for example). All options are listed here.--verbose

Cloud-Prometheus-cAdvisor-Container-Cpu-Api-custom

MacroDescriptionDefault valueMandatory
WARNING_USAGEWarning threshold on the CPU usage (%)
CRITICAL_USAGECritical threshold on the CPU usage (%)
WARNING_THROTTLEDWarning threshold on the CPU throttling (%)
CRITICAL_THROTTLEDCritical threshold on the CPU throttling (%)
EXTRAOPTIONSAny extra option you may want to add to the command (a --verbose flag for example). All options are listed here.
  1. Deploy the configuration. The service appears in the list of services, and on the Resources Status page. The command that is sent by the connector is displayed in the details panel of the service: it shows the values of the macros.

How to check in the CLI that the configuration is OK and what are the main options for?​

Once the plugin is installed, log into your Centreon poller's CLI using the centreon-engine user account (su - centreon-engine). Test that the connector is able to monitor a resource using a command like this one (replace the sample values with yours):

/usr/lib/centreon/plugins/centreon_prometheus_cadvisor_api.pl \
	--plugin=cloud::prometheus::exporters::cadvisor::plugin \
	--mode=memory \
	--hostname=10.0.0.1 \
	--url-path='/api/v1' \
	--port='9090' \
	--proto='http' \
	--container='container="nginx"' \
	--pod='pod="web-1"' \
	--extra-filter='namespace="default"' \
	--warning-usage='' \
	--critical-usage='' \
	--warning-working='' \
	--critical-working='' \
	--warning-cache='' \
	--critical-cache='' \
	--warning-rss='' \
	--critical-rss='' \
	--warning-swap='' \
	--critical-swap='' \
	--verbose

The expected command output is shown below:

OK: Container 'nginx' [pod: web-1] Memory Usage: 50.00 MB (50.00% of 100.00 MB), Working: 40.00 MB (40.00% of 100.00 MB), Cache: 10.00 MB, Rss: 38.00 MB, Swap: 0.00 B | 'web-1~nginx#memory.used.bytes'=52428800B;;;0;104857600 'web-1~nginx#memory.working.bytes'=41943040B;;;0;104857600 'web-1~nginx#cache.usage.bytes'=10485760B;;;0; 'web-1~nginx#rss.usage.bytes'=39845888B;;;0; 'web-1~nginx#swap.usage.bytes'=0B;;;0;
Container 'nginx' [pod: web-1] Memory Usage: 50.00 MB (50.00% of 100.00 MB), Working: 40.00 MB (40.00% of 100.00 MB), Cache: 10.00 MB, Rss: 38.00 MB, Swap: 0.00 B

Troubleshooting​

Please find the troubleshooting documentation for the API-based plugins in this chapter.

Available modes​

In most cases, a mode corresponds to a service template. The mode appears in the execution command for the connector. In the Centreon interface, you don't need to specify a mode explicitly: its use is implied when you apply a service template. However, you will need to specify the correct mode for the template if you want to test the execution command for the connector in your terminal.

All available modes can be displayed by adding the --list-mode parameter to the command:

/usr/lib/centreon/plugins/centreon_prometheus_cadvisor_api.pl \
	--plugin=cloud::prometheus::exporters::cadvisor::plugin \
	--list-mode

The plugin brings the following modes:

ModeLinked service template
cpu [code]Cloud-Prometheus-cAdvisor-Cpu-Api-custom
Cloud-Prometheus-cAdvisor-Container-Cpu-Api-custom
Cloud-Prometheus-cAdvisor-Pod-Cpu-Api-custom
discovery [code]Used for host discovery
list-containers [code]Not used in this Monitoring Connector
load [code]Cloud-Prometheus-cAdvisor-Load-Api-custom
Cloud-Prometheus-cAdvisor-Container-Load-Api-custom
Cloud-Prometheus-cAdvisor-Pod-Load-Api-custom
memory [code]Cloud-Prometheus-cAdvisor-Memory-Api-custom
Cloud-Prometheus-cAdvisor-Container-Memory-Api-custom
Cloud-Prometheus-cAdvisor-Pod-Memory-Api-custom
presence [code]Used for the host check of Cloud-Prometheus-cAdvisor-Api-Container-custom
Cloud-Prometheus-cAdvisor-Api-Kubernetes-Container-custom
Cloud-Prometheus-cAdvisor-Api-Pod-custom
storage [code]Cloud-Prometheus-cAdvisor-Storage-Api-custom
Cloud-Prometheus-cAdvisor-Container-Storage-Api-custom
Cloud-Prometheus-cAdvisor-Pod-Storage-Api-custom
task-state [code]Cloud-Prometheus-cAdvisor-Task-State-Api-custom
Cloud-Prometheus-cAdvisor-Container-Task-State-Api-custom
Cloud-Prometheus-cAdvisor-Pod-Task-State-Api-custom

Available options​

Generic options​

All generic options are listed here:

OptionDescription
--modeDefine the mode in which you want the plugin to be executed (see --list-mode).
--dyn-modeSpecify a mode with the module's path (advanced).
--list-modeList all available modes.
--mode-versionCheck minimal version of mode. If not, unknown error.
--versionReturn the version of the plugin.
--custommodeWhen a plugin offers several ways (CLI, library, etc.) to get information the desired one must be defined with this option.
--list-custommodeList all available custom modes.
--multipleMultiple custom mode objects. This may be required by some specific modes (advanced).
--pass-managerDefine the password manager you want to use. Supported managers are: environment, file, keepass, hashicorpvault and teampass.
--verboseDisplay extended status information (long output).
--debugDisplay debug messages.
--show-passwordBy default, sensitive information in command lines is hidden in debug output and replaced with *** (however, debug logs may still display sensitive information). Using the --show-password option will display the passwords in plain text.
--filter-perfdataFilter perfdata that match the regexp. Example: adding --filter-perfdata='avg' will remove all metrics that do not contain 'avg' from performance data.
--filter-perfdata-advFilter perfdata based on a "if" condition using the following variables: label, value, unit, warning, critical, min, max. Variables must be written either %{variable} or %(variable). Example: adding --filter-perfdata-adv='not (%(value) == 0 and %(max) eq "")' will remove all metrics whose value equals 0 and that don't have a maximum value.
--explode-perfdata-maxCreate a new metric for each metric that comes with a maximum limit. The new metric will be named identically with a '_max' suffix. Example: it will split 'used_prct'=26.93%;0:80;0:90;0;100 into 'used_prct'=26.93%;0:80;0:90;0;100 'used_prct_max'=100%;;;;
--change-perfdata --extend-perfdataChange or extend perfdata. Syntax: --extend-perfdata=searchlabel,newlabel,target[,[<new-unit-of-measure>],[min],[max]] Common examples: Convert storage free perfdata into used: --change-perfdata='free,used,invert()' Convert storage used perfdata into free: --change-perfdata='used,free,invert()' Scale traffic values automatically: --change-perfdata='traffic,,scale(auto)' Scale traffic values in Mbps: --change-perfdata='traffic_in,,scale(Mbps),mbps' Change traffic values in percent: --change-perfdata='traffic_in,,percent()'
--change-perfdataChange or extend perfdata. Syntax: --extend-perfdata=searchlabel,newlabel,target[,[<new-unit-of-measure>],[min],[max]] Common examples: Convert storage free perfdata into used: --change-perfdata='free,used,invert()' Convert storage used perfdata into free: --change-perfdata='used,free,invert()' Scale traffic values automatically: --change-perfdata='traffic,,scale(auto)' Scale traffic values in Mbps: --change-perfdata='traffic_in,,scale(Mbps),mbps' Change traffic values in percent: --change-perfdata='traffic_in,,percent()'
--extend-perfdataChange or extend perfdata. Syntax: --extend-perfdata=searchlabel,newlabel,target[,[<new-unit-of-measure>],[min],[max]] Common examples: Convert storage free perfdata into used: --change-perfdata='free,used,invert()' Convert storage used perfdata into free: --change-perfdata='used,free,invert()' Scale traffic values automatically: --change-perfdata='traffic,,scale(auto)' Scale traffic values in Mbps: --change-perfdata='traffic_in,,scale(Mbps),mbps' Change traffic values in percent: --change-perfdata='traffic_in,,percent()'
--extend-perfdata-groupAdd new aggregated metrics (min, max, average or sum) for groups of metrics defined by a regex match on the metrics' names. Syntax: --extend-perfdata-group=regex,<names-of-new-metrics>,calculation[,[<new-unit-of-measure>],[min],[max]] regex: regular expression <names-of-new-metrics>: how the new metrics' names are composed (can use $1, $2... for groups defined by () in regex). calculation: how the values of the new metrics should be calculated <new-unit-of-measure> (optional): unit of measure for the new metrics min (optional): lowest value the metrics can reach max (optional): highest value the metrics can reach Common examples: Sum wrong packets from all interfaces (with interface need --units-errors=absolute): --extend-perfdata-group=',packets_wrong,sum(packets_(discard|error)_(in|out))' Sum traffic by interface: --extend-perfdata-group='traffic_in_(.*),traffic_$1,sum(traffic_(in|out)_$1)'
--change-short-output --change-long-outputModify the short/long output that is returned by the plugin. Syntax: --change-short-output=pattern~replacement~modifier Most commonly used modifiers are i (case insensitive) and g (replace all occurrences). Example: adding --change-short-output='OK~Up~gi' will replace all occurrences of 'OK', 'ok', 'Ok' or 'oK' with 'Up'
--change-short-outputModify the short/long output that is returned by the plugin. Syntax: --change-short-output=pattern~replacement~modifier Most commonly used modifiers are i (case insensitive) and g (replace all occurrences). Example: adding --change-short-output='OK~Up~gi' will replace all occurrences of 'OK', 'ok', 'Ok' or 'oK' with 'Up'
--change-long-outputModify the short/long output that is returned by the plugin. Syntax: --change-short-output=pattern~replacement~modifier Most commonly used modifiers are i (case insensitive) and g (replace all occurrences). Example: adding --change-short-output='OK~Up~gi' will replace all occurrences of 'OK', 'ok', 'Ok' or 'oK' with 'Up'
--change-exitReplace an exit code with one of your choice. Example: adding --change-exit=unknown=critical will result in a CRITICAL state instead of an UNKNOWN state.
--change-output-advReplace short output and exit code based on a "if" condition using the following variables: short_output, exit_code. Variables must be written either %{variable} or %(variable). Example: adding --change-output-adv='%(short_output) =~ /UNKNOWN: No daemon/,OK: No daemon,OK' will change the following specific UNKNOWN result to an OK result.
--range-perfdataRewrite the ranges displayed in the perfdata. Accepted values: 0: nothing is changed. 1: if the lower value of the range is equal to 0, it is removed. 2: remove the thresholds from the perfdata.
--filter-uomMask the units when they don't match the given regular expression.
--opt-exitReplace the exit code in case of an execution error (i.e. wrong option provided, SSH connection refused, timeout, etc). Default: unknown.
--output-ignore-perfdataRemove all the metrics from the service. The service will still have a status and an output.
--output-ignore-labelRemove the status label ("OK:", "WARNING:", "UNKNOWN:", CRITICAL:") from the beginning of the output. Example: 'OK: Ram Total:...' will become 'Ram Total:...'
--output-xmlReturn the output in XML format (to send to an XML API).
--output-jsonReturn the output in JSON format (to send to a JSON API).
--output-openmetricsReturn the output in OpenMetrics format (to send to a tool expecting this format).
--output-fileWrite output in file (can be combined with JSON, XML and OpenMetrics options). Example: --output-file=/tmp/output.txt will write the output in /tmp/output.txt.
--disco-formatApplies only to modes beginning with 'list-'. Returns the list of available macros to configure a service discovery rule (formatted in XML).
--disco-showApplies only to modes beginning with 'list-'. Returns the list of discovered objects (formatted in XML) for service discovery.
--float-precisionDefine the float precision for thresholds (default: 8).
--source-encodingDefine the character encoding of the response sent by the monitored resource. Default: 'UTF-8'.
--http-peer-addrSet the address you want to connect to. Useful if hostname is only a vhost, to avoid IP resolution.
--proxyurlProxy URL. Example: http://my.proxy:3128
--proxypacProxy PAC file (can be a URL or a local file).
--insecureAccept insecure SSL connections.
--http-backendPerl library to use for HTTP transactions. Possible values are: lwp (default) and curl.
--timeframeSet timeframe in seconds (i.e. 3600 to check last hour).
--stepSet the step of the metric query (examples: '30s', '1m', '15m', '1h').
--hostnamePrometheus hostname.
--url-pathAPI URL path (default: '/api/v1')
--portAPI port (default: 9090)
--protoSpecify https if needed (default: 'http')
--credentialsSpecify this option if you access the API with authentication
--usernameSpecify the username for authentication (mandatory if --credentials is specified)
--passwordSpecify the password for authentication (mandatory if --credentials is specified)
--basicSpecify this option if you access the API over basic authentication and don't want a '401 UNAUTHORIZED' error to be logged on your web server. Specify this option if you access the API over hidden basic authentication or you'll get a '404 NOT FOUND' error. (use with --credentials)
--timeoutSet HTTP timeout
--headerSet HTTP header (can be multiple, example: --header='Authorization:Bearer ABCD') Useful to access Prometheus API hosted in a specific environment.

Modes options​

All available options for each service template are listed below:

OptionDescription
--cpu-attributeSet the CPU attribute to match element (must be a PromQL filter, default: 'cpu="total"').
--containerFilter on a specific container (must be a PromQL filter, default: 'container!~"|.POD."').
--podFilter on a specific pod (must be a PromQL filter, default: 'pod=~"."'). The label filtered is the first word of the --container and --pod filters. The kubelet cAdvisor metrics label names depend on the platform version: Kubernetes < 1.14: container_name and pod_name only. Kubernetes 1.14 and 1.15: both container_name/pod_name and container/pod. Kubernetes >= 1.16: container and pod only. OpenShift: container_name and pod_name only up to 4.1 (Kubernetes 1.13), both label sets in 4.2 (Kubernetes 1.14), container and pod only from 4.3 (Kubernetes 1.16). Rancher: depends on the Kubernetes version of the managed cluster. Kubernetes 1.16 is available from Rancher 2.3.0 (experimental), generally available in 2.3.2 and the default in 2.3.3. On older platforms, use for example: --container='container_name!~"|.POD."' --pod='pod_name=~"."'
--namespaceFilter on a specific namespace (must be a PromQL filter, default: 'namespace=~".*"'). Two pods with the same name in different namespaces are reported separately, but with the same output and performance data names: filter on the namespace to tell them apart.
--warning-usageWarning threshold.
--warning-throttledWarning threshold.
--critical-usageCritical threshold.
--critical-throttledCritical threshold.
--extra-filterAdd a PromQL filter (can be defined multiple times). Example : --extra-filter='name=~".pretty."'
--metric-overloadOverload default metrics name (can be defined multiple times). Example : --metric-overload='metric,^my_metric_name$' Default : - throttled: ^container_cpu_cfs_throttled_seconds_total.* - usage: ^container_cpu_usage_seconds_total.*
--filter-countersOnly display some counters (regexp can be used). Example: --filter-counters='throttled'

All available options for a given mode can be displayed by adding the --help parameter to the command:

/usr/lib/centreon/plugins/centreon_prometheus_cadvisor_api.pl \
	--plugin=cloud::prometheus::exporters::cadvisor::plugin \
	--mode=memory \
	--help