Static pods are pods which are not managed by the Kubernetes control plane. Normally, the kubelet running on each node in a cluster watches the kube-api for instructions to launch or delete pods. However, in the case of static pods, the kubelet can manage pods by watching for manifests added to a directory or a web url.
Why would you want to do this? Some of the use-cases for static pods include:
- Node Initialization: Run a pod during node bootstrap to run tasks before a node is added to the cluster.
- System Services: Running system services and daemons (a DaemonSet is normally better for this).
- Custom Controllers: Running custom controllers or agents.
- Custom Networking: Run custom network plugins or proxy services on nodes.
- Batch Processing: Run specific batch workloads that don’t require orchestration.
- Testing and Development: Quickly test workloads without going through the Kubernetes control plane.
Static pods are also referred to as mirror pods, because a copy of the pod is registered in the kube api. This is a read-only representation of the pod allowing static pods to also appear in the kube-api. This means operations like kubectl delete will not work for static pods.
The kubelet automatically tries to create a mirror Pod on the Kubernetes API server for each static Pod. This means that the Pods running on a node are visible on the API server, but cannot be controlled from there. The Pod names will be suffixed with the node hostname with a leading hyphen.
https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/
Configure Kubelet to Detect and Manage Static Pods
We first need to configure the kubelet with either a path to a directory or a web url to watch for manifests. We’ll run through two examples using minikube, which by the way also uses static pods to launch kubernetes components!
Using Directory Path
We will use the commonly used path for static pods, /etc/kubernetes/manifests/ as the path to watch for manifests.
To configure this, we can pass the argument --pod-manifest-path to the kubelet. Alternatively, we can pass staticPodPath to the kubelet configuration file.
In the case of minikube, kubelet args can be passed using a command line argument when starting minikube:
--extra-config=kubelet.<parameter>The complete minikube startup command:
minikube start --extra-config=kubelet.pod-manifest-path=/etc/kubernetes/manifests/ --container-runtime=cri-o --driver=podman
...
😄 minikube v1.31.2 on Darwin 13.6
✨ Using the podman (experimental) driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔄 Restarting existing podman container for "minikube" ...
🎁 Preparing Kubernetes v1.27.4 on CRI-O 1.24.6 ...
E1021 17:18:25.912265 67058 start.go:131] Unable to get host IP: RoutableHostIPFromInside is currently only implemented for linux
▪ kubelet.pod-manifest-path=/etc/kubernetes/manifests/
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
...I am using the podman open-source container management tool, which is an alternative to docker. Also, note that the kubelet.pod-manifest-path parameter is displayed in the output after startup, confirming the parameter is recognized and correctly set.
Let’s confirm there are no pods running before we add our static pod:
❯ kubectl get pods
No resources found in default namespace.Now we can log into our minikube kubernetes node with ssh, to create the static pod manifest:
❯ minikube ssh
docker@minikube:~$Let’s check whether the manifests directory we specified already exists. Spoiler alert – on minikube it does exist and already contains pod manifests for minikube’s control plane components!
❯ minikube ssh
docker@minikube:~$ cd /etc/kubernetes/manifests/
docker@minikube:/etc/kubernetes/manifests$ ls -1
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yamlLet’s now add our own pod manifest:
docker@minikube:/etc/kubernetes/manifests$ sudo -i
root@minikube:~# cat <<EOF >/etc/kubernetes/manifests/static-web.yaml
apiVersion: v1
kind: Pod
metadata:
name: static-web
spec:
containers:
- name: web
image: docker.io/nginx
ports:
- name: web
containerPort: 80
protocol: TCP
EOF
root@minikube:~# ls -1 /etc/kubernetes/manifests/
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml
static-web.yaml
Note: Since only the root user is permitted to write to this directory, I switched to root user before creating the pod manifest:
ls -ld /etc/kubernetes/manifests/
drwxr-xr-x. 1 root root 136 Oct 21 16:38 /etc/kubernetes/manifests/Let’s jump out of the node, and check running pods with kubectl:
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
static-web-minikube 1/1 Running 0 6m17sSince we launched a pod running the nginx web server, we can connect to the pod and check for the default web page:
❯ k port-forward pod/static-web-minikube 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
And what happens when we delete our pod manifest file?
❯ minikube ssh
docker@minikube:~$ sudo rm /etc/kubernetes/manifests/static-web.yamlThe kubelet will delete the pod …
❯ kubectl get pods
No resources found in default namespace.Using Web Url
It is also possible to configure a web url of a pod manifest for kubelet to watch. Personally, I’ve never used this feature but it’s interesting to know it exists. Let’s try it out.
I’ve created a pod manifest under https://www.kubeblog.com/static-web.yaml, same as the manifest I provided in the previous example.
❯ minikube start --driver=podman --container-runtime=cri-o --extra-config=kubelet.manifest-url=https://www.kubeblog.com/static-web.yaml
...
😄 minikube v1.31.2 on Darwin 13.6
✨ Using the podman (experimental) driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔄 Restarting existing podman container for "minikube" ...
🎁 Preparing Kubernetes v1.27.4 on CRI-O 1.24.6 ...
E1021 20:07:50.372395 71118 start.go:131] Unable to get host IP: RoutableHostIPFromInside is currently only implemented for linux
▪ kubelet.manifest-url=https://www.kubeblog.com/static-web.yaml
🔗 Configuring CNI (Container Networking Interface) ...
...Shortly after starting minikube, the static pod appears as running:
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
static-web-minikube 1/1 Running 0 16mYou can also view running containers when logged into the node using crictl:
❯ minikube ssh
docker@minikube:~$ sudo crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
...
2031829b28d10 docker.io/library/nginx@sha256:3a12fc354e3c4dd62196a809e52a5d2f8f385b52fcc62145b0efec5954bb8fa1 20 minutes ago Running web 0 2dba3ab8cd243 static-web-minikube
...These were some quick examples of running static pods. Knowing that this feature exists can allow for some creative ways to run tasks via pods while bootstrapping nodes. As we saw, tools such as minikube rely on static pods to bootstrap most of the kubernetes components.
Limitations and Considerations of Static Pods
There are a few things to keep in mind when using static pods:
- kubelet will try to restart pods if they crash.
- Static Pods are always bound to one Kubelet on a specific node.
- If manifest is deleted, the pod will be deleted automatically.
- You can only create pods with this method, not deployments, replicasets, or other kubernetes objects.
- The spec of a static Pod cannot refer to other API objects (e.g., ServiceAccount, ConfigMap, Secret, etc).
- Static pods do not support ephemeral containers.
- If you use cluster-autoscaler, be sure to read the FAQ for how mirror (static) pods are handled.
If you found this article useful, please give a “like” or leave a comment.
