Kubernetes does not come with a full-featured monitoring solution out-of-the-box, however it does expose metrics for the health and state of the cluster and its workloads. These metrics are exposed by APIs, which are in turn collected by monitoring solutions such Prometheus, Metric Server and many other third-party tools. You can read more about the design proposal for the Kubernetes monitoring architecture.
We can organize Kubernetes metrics into several high-level categories:
- Container & pod metrics – such as number of pods and their health. CPU, memory utilization
- Node metrics — number of nodes and their health; metrics for CPU, memory, network and disk utilization
- Control Plane metrics — health of the individual Kubernetes components themselves
- Workload & cluster metrics – metrics about the state of the objects in the cluster (Deployments, Pods, StatefulSets)
Each type of metric is generated from a different source and retrieved in different ways. In this article, we will look at how some of these metrics are exposed, using a local minikube cluster to demonstrate.
Kubernetes Metrics API
Before looking at the different types of metrics exposed by Kubernetes, let’s talk about the Kubernetes Metrics API. The Metrics API is the part of the Kubernetes project and defines standard metrics types and code as well as client code which can retrieve metrics from Kubernetes. Software such as metrics-server and prometheus-adapter implement the metrics API client for retrieving pod CPU and memory metrics, for example.
Applications which wish to expose custom metrics in the Kubernetes Metrics API would implement the Custom Metrics API Server.
The custom metrics API contain two categories of metrics:
- Custom metrics (k8s.io/metrics/pkg/apis/custom_metrics) — for exposing metrics for application or workloads running inside the Kubernetes cluster.
- External metrics (k8s.io/metrics/pkg/apis/external_metrics) — for exposing metrics for things outside of Kubernetes , such as a message queue.
The Kubernetes Horizontal Pod Autoscaler (HPA) can enable scaling based on custom and external metrics, for example.
Kubelet (cAdvisor) Metrics
Recall that the Kubelet runs on each node in the cluster and is responsible for managing the lifecycle of pods. The Kubelet exposes operational metrics, detailed metrics about api calls and some other low-level metrics. It also exposes metrics for individual containers it manages. It does this using a built-in version of cAdvisor (short for container Advisor) — you don’t need to install it separately.
The Kubelet — through cAdvisor — interacts with the container runtime to gather real-time performance metrics, and the Kubelet’s HTTP API, allows external components like monitoring tools or administrators to query the metrics.
The Kubelet (via cAdvisor) exposes metrics in the popular prometheus format. In fact, scraping the Kubelet metrics endpoint is how Prometheus collects pod resource metrics.
Let’s retrieve some of these metrics from a Kubelet using curl using a local minikube cluster.
❯ minikube start
😄 minikube v1.32.0 on Darwin 14.3.1
...
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
# our pod will need a service account token with RBAC policies
❯ kubectl create serviceaccount cadvisor-test
# for simplicity, we will bind our serviceaccount the "cluster-admin"
❯ kubectl create clusterrolebinding --clusterrole cluster-admin --serviceaccount default:cadvisor-test cadvisor-cluster-admin-binding
clusterrolebinding.rbac.authorization.k8s.io/cadvisor-cluster-admin-binding created
# start a pod with an interactive terminal
❯ kubectl run --rm -it --image=alpine --overrides='{ "spec": { "serviceAccount": "cadvisor-test" } }' cadvisor-test -- /bin/sh
# install curl
/ # apk add curlNow we are ready to do some api calls with curl. Note: 192.168.49.2 is minikube’s node IP on my computer, yours may be different
Querying the /metrics endpoint will retrieve operational, api, and node metrics:
/ # curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://192.168.49.2:10250/metrics
...
# TYPE kubernetes_feature_enabled gauge
kubernetes_feature_enabled{name="APIListChunking",stage="BETA"} 1
kubernetes_feature_enabled{name="APIPriorityAndFairness",stage="BETA"} 1
kubernetes_feature_enabled{name="APIResponseCompression",stage="BETA"} 1
kubernetes_feature_enabled{name="APISelfSubjectReview",stage="BETA"} 1
kubernetes_feature_enabled{name="APIServerIdentity",stage="BETA"} 1
kubernetes_feature_enabled{name="APIServerTracing",stage="BETA"} 1
...
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 15.57
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 26
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.0104832e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.71062708371e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.594912768e+09
...Querying the /metrics/cadvisor endpoint will retrieve container metrics:
/ # curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://192.168.49.2:10250/metrics/cadvisor
# HELP container_cpu_cfs_throttled_seconds_total Total time duration the container has been throttled.
# TYPE container_cpu_cfs_throttled_seconds_total counter
container_cpu_cfs_throttled_seconds_total{container="",id="/",image="",name="",namespace="",pod=""} 0 1710629088892
container_cpu_cfs_throttled_seconds_total{container="",id="/kubepods/pod2136dbe8-ff5e-400d-9911-05eb6f519c88",image="",name="",namespace="kube-system",pod="kindnet-56z9c"} 1.706189 1710629084131c2d8a63a8eb8fcc20e8ebd44",image="gcr.io/k8s-minikube/storage-provisioner:v5",major="252",minor="0",name="k8s_storage-provisioner_storage-provisioner_kube-system_3933d53a-9742-4b93-b94e-513f792
...
# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge
container_cpu_load_average_10s{container="",id="/",image="",name="",namespace="",pod=""} 0 1710629088892
container_cpu_load_average_10s{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1710629088911
container_cpu_load_average_10s{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1710629079040
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/pod1c822e3e-6bd9-4cde-a8b7-586ea7e3d5c4",image="",name="",namespace="default",pod="cadvisor-test"} 0 1710629079824
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/pod1c822e3e-6bd9-4cde-a8b7-586ea7e3d5c4/crio-conmon-898a9ad22ade6e221c45e3e874faa985a0c083313bcd25242f765aab98b8bf0f",image="
...
# HELP container_memory_max_usage_bytes Maximum memory usage recorded in bytes
# TYPE container_memory_max_usage_bytes gauge
container_memory_max_usage_bytes{container="",id="/",image="",name="",namespace="",pod=""} 0 1710629088892
container_memory_max_usage_bytes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1710629088911
container_memory_max_usage_bytes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1710629079040The kubelet’s metrics endpoint contains many low-level metrics – only a subset of which are interesting for cluster operators. More interesting are the container metrics exposed by the cAdvisor endpoint.
This was an example to demonstrate retrieval of metrics from the kubelet’s own metrics endpoint. As we shall see, the Metrics Server aggregates useful resource utilization metrics across nodes and pods.
Metrics Server
Node and Pod metrics are exposed by the Kubelet running on each node, which makes sense since the Kubelet runs on each node and is responsible for the lifecycle of pods running on that node. Although these metrics are made available in the Kubernetes API, they are individual point-in-time metrics for each pod or node.
Wouldn’t it be nice if we were able to aggregate pod and node metrics across the entire cluster, and query these aggregated metrics directly? This is where Metrics Server comes in.
Metric Server is an “addon” maintained by the Kubernetes project and needs to be installed separately.
Metrics Server collects and aggregates resource utilization metrics for pods and nodes in the cluster. It gathers data from the Kubelet on each node and makes it accessible through the Kubernetes API server. Metrics Server uses a pull-based model to scrape data from the Kubelet and provides an HTTP endpoint for metric retrieval.
On a side note, even though many open-source and commercial monitoring tools fetch data from the Metrics APIs that Metric Server provides, the project README warns that its metrics are mainly designed for use by Horizontal/Vertical Pod Autoscaler and not for general metrics consumption:
Metrics Server is not meant for non-autoscaling purposes. For example, don’t use it to forward metrics to monitoring solutions, or as a source of monitoring solution metrics. In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.
https://github.com/kubernetes-sigs/metrics-server
Once installed, the Metrics API endpoints for nodes and pods will appear under the following paths, respectively:
- /apis/metrics.k8s.io/v1beta1/nodes
- /apis/metrics.k8s.io/v1beta1/pods
For example, if you wanted to query the Kubernetes API directly with curl, you could do the following:
# create a proxy to the Kubernetes API
kubectl proxy
Starting to serve on 127.0.0.1:8001
# in a different terminal window
curl -o - http://localhost:8001/apis/metrics.k8s.io/v1beta1/pods/
...
{
"kind": "PodMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metadata": {
"name": "argo-events-controller-manager-bf549bcd7-xjxgm",
"namespace": "argo-events",
"creationTimestamp": "2023-11-06T21:00:10Z",
"labels": {...}
},
"timestamp": "2023-11-06T21:00:00Z",
"window": "14.187s",
"containers": [
{
"name": "controller-manager",
"usage": {
"cpu": "1952818n",
"memory": "34128Ki"
}
}
]
},
...
curl -o - http://localhost:8001/apis/metrics.k8s.io/v1beta1/nodes/
...
{
"kind": "NodeMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metadata": {
"name": "ip-172-26-36-189.eu-west-1.compute.internal",
"creationTimestamp": "2023-11-06T20:53:33Z",
"labels": {
},
"timestamp": "2023-11-06T20:53:24Z",
"window": "20.077s",
"usage": {
"cpu": "651042328n",
"memory": "4517192Ki"
}
},
...
Metrics server also makes it possible to query these metrics using kubectl top. If you are following along with minikube, install the metrics-server add-on with the following command:
❯ minikube addons enable metrics-server
💡 metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.6.4
🌟 The 'metrics-server' addon is enabledCheck if the metrics-server pod is running:
❯ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
metrics-server-7746886d4f-m28n9 1/1 Running 2 (60s ago) 3m2sRun the top command:
❯ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 395m 19% 463Mi 23%Note: you may need to wait a few minutes after installation before metrics are visible.
You can further explore the metrics API with kubectl get …
> kubectl get --raw "/apis/metrics.k8s.io/v1beta1/"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]
},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]}
> kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .
{
"kind": "NodeMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metadata": {
"name": "minikube",
"creationTimestamp": "2024-03-24T18:33:06Z",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "minikube",
"kubernetes.io/os": "linux",
"minikube.k8s.io/commit": "fd7ecd9c4599bef9f04c0986c4a0187f98a4396e",
> kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .
{
"kind": "PodMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metadata": {
"name": "coredns-5d78c9869d-xkcfw",
"namespace": "kube-system",
"creationTimestamp": "2024-03-24T18:38:42Z",
"labels": {
"k8s-app": "kube-dns",
"pod-template-hash": "5d78c9869d"
}
},
"timestamp": "2024-03-24T18:38:17Z",
"window": "1m1.202s",
"containers": [
{
"name": "coredns",Workload Metrics
The last type of metrics we will talk about are workload metrics describing the state of various Kubernetes resources in the cluster. The standard software used to provide these metrics is kube-state-metrics, an official Kubernetes project that also must be installed separately in you cluster.
Recall that Kubernetes maintains the state of all running objects in its own internal etcd database as part of the control plane. Objects could be Pods, Ingresses, Nodes, Jobs, ConfigMaps, etc. These object metrics are also exposed by the Kubernetes API Server. Kube-state-metrics retrieves metrics from the Kubernetes API Server, aggregates them, and makes them available via an HTTP endpoint for other monitoring solutions (like prometheus) to scrape.
There are hundreds of metrics made available by kube-state-metrics. For example, if you wanted to query the memory usage of running pods, a promQL query might look like this:
sum(kube_pod_container_resource_requests{resource="memory"}) by (namespace, pod, node) * on (namespace, pod) group_left() (sum(kube_pod_status_phase{phase="Running"}) by (pod, namespace) == 1)The language used to query metrics depends on the monitoring software you use. If you use NewRelic, the language would be “NewRelic Query Language” (NRQL). If you were using DataDog, it would be “Datadog Query Language” (DQL).
Metrics from kube-state-metrics are critical for understanding the health and state of everything running in your cluster.
Conclusion
In this article I explained a bit how Kubernetes exposes metrics at the low-level. In practice however, most people use end-to-end metrics solutions such as open-source Prometheus or commercial solutions like DataDog or NewRelic to collect, store and visualize these metrics.
