Evaluating AWS EKS Pod Identity

EKS Pod Identity was announced at AWS ReInvent:2023. This feature aims to simplify granting pods access to AWS services running in an EKS cluster. AWS EKS Pod Identity allows you to associate an IAM role with a Kubernetes service account and configure your Pods to use that service account.

Introduction

Kubernetes workloads that need access to AWS resources, i.e., using AWS SDKs, must have IAM credentials that permit those specific actions, like accessing S3 buckets. And we all know that using long-lived IAM User access keys is a really bad idea (right?).

If you are an AWS EKS (Elastic Kubernetes Service) user, you are no doubt familiar with IAM Roles for ServiceAccounts (IRSA). EKS Pod Identity is similar to IRSA, but the mechanism for mapping namespaces and serviceaccounts to IAM roles is different.

These are my notes from trying out EKS Pod Identity. I’ll give my opinion on this tool, and explain why we haven’t switched to Pod Identity with the EKS clusters in my company.

The Basics

The official AWS blog post announcing EKS Pod Identity from December 2023 does a great job of explaining the architecture and a provides a walk-through example. Also, in the EKS User guide there is a section on EKS Pod Identity.

As you will notice, Pod Identity is documented along side IAM Roles for ServiceAccounts (IRSA). According to my AWS contacts, at the time of this writing, there are no plans to deprecate IRSA.

How is Pod Identity different from IRSA?

• You map IAM Roles to Kubernetes namespaces and serviceaccounts using an “Pod Identity Association”.
• You do not need to use any annotations on the serviceaccount (as with IRSA).
• Since Pod Identity does not rely on OIDC, the trust policy of IAM Roles used with Pods is greatly simplified.
• Pod Identity supports session tagging, which can be useful for implementing attribute-based access control (ABAC).

Why do we need this if IRSA works?

The main advantages is that the mapping between IAM roles and serviceaccounts/namespaces can be configured independently of the EKS cluster, you don’t need the information about the OIDC url and issuer — the EKS cluster does not even need to be installed yet.

This might be useful if the team managing IAM roles is a different team that manages EKS clusters. Also, since you do not need to provide serviceaccount annotations, Pods can be provided with IAM credentials without even knowing the IAM role ARN. This is useful since you don’t need to synchronize your terraform (OpenTofu) pipelines with your Kubernetes workload deployment pipeline.

# This annotation is NOT required with Pod Identity!

apiVersion: v1
kind: ServiceAccount
metadata:
  name:  myapp-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::9999999999:role/some-app-role

Mapping IAM roles to namespaces/serviceaccounts

As mentioned, a new AWS resource called “Pod Identity Association” is used to connect IAM roles with serviceaccounts. This can be done several ways:

• The AWS cli: aws eks create-pod-identity-association
• The AWS web console
• Terraform

To use the AWS web console, navigate to an EKS cluster –> “Access” –> “Pod Identity associations”. This section will display existing Pod Identity associations and allow you to create new ones.

However, using the AWS console means the EKS cluster must be already running, since the management of Pod Identity associations is part of the cluster’s configuration dashboard.

There is also a terraform resource for Pod Identity associations:

resource "aws_eks_pod_identity_association" "this" {
  cluster_name    = "some-cluster-name"
  namespace       = "kube-system"
  service_account = "karpenter"
  role_arn        = aws_iam_role.karpenter_role.arn
}

The IAM role itself requires a trust policy allowing the service pods.eks.amazonaws.com to assume the role and tag the session. You can also lock-down the trust policy to specific clusters and accounts:

resource "aws_iam_role" "this" {
  name               = "my-app-role"
  assume_role_policy = data.aws_iam_policy_document.this.json
}

data "aws_iam_policy_document" "this" {
  statement {
    actions = [
      "sts:AssumeRole",
      "sts:TagSession",
    ]

    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [data.aws_caller_identity.this.account_id]
    }

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [data.aws_eks_cluster.this.arn]
    }

  }
}

Installation

To use Pod Identity in an EKS cluster, a pod-identity-agent needs to be installed as a DaemonSet. The pod-identity-agent will intercept calls from pods for the IAM credential provider chain, then call the AWS API to retrieve the role credentials from an IAM endpoint.

The other part of the architecture, the mutating webhook which injects credentials into pods, is built into the EKS control plane:

AWS provides the Pod Identity Agent as an add-on. However, since we prefer not to use AWS add-ons, we use the helm chart that lives in the AWS pod-identity-agent public git repo.

Unfortunately, at the time of this writing, this chart is not available in the EKS-charts registry (👎), but I can imagine they will publish it eventually. Also, the git repo does not use version tagging and the release lifecycle is non-existent! (👎)

This means if you are using a helm chart directly from the git repo, you will need to point to the “main” branch. To avoid unexpected surprises, the safer thing to do is to use a fork of the repo and sync changes by hand.

The EKS node also requires an IAM policy, but thankfully AWS added to the AWS Managed AmazonEKSWorkerNodePolicy, which most EKS clusters already include in their node instance role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "WorkerNodePermissions",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumesModifications",
                "ec2:DescribeVpcs",
                "eks:DescribeCluster",
                "eks-auth:AssumeRoleForPodIdentity"  # pod identity permission
            ],
            "Resource": "*"
        }
    ]
}

Why we are not switching to EKS Pod Identity

For us, the fact that a Daemonset is required was a big reason why we hesitated to switch to EKS Pod Identity. Nodes that are starved of resource due to improperly configured workloads or under heavy load spikes could negatively impact the agent pod running on the same node. Since many critical infrastructure software like Karpenter, Cluster Autoscaler and aws-loadbalancer-controller rely on IAM credentials, the pod identity agent is a critical service which could have a big impact on a highly-active production cluster if it does not run as expected.

I personally remember similar hard-to-debug problems with kube2iam, the solution we were using before AWS introduced IRSA. Some pods were not receiving their role credentials because of node resource starvation cause by “noisy neighbors”. Such misconfigured pods are a fact of life and need to be fixed, but they should not cause operational issues for the entire cluster. There were also some isolated cases where pods needed to use aws credentials before the kube2iam daemonset was ready to respond.

Another downside with the Pod Identity agent Daemonset is that it binds to port 80 on the host network — the init pod needs to operate in privileged mode, the agent needs to have the net capabilities enabled and the pod needs to run in host network mode (😥)

Another potential issue is that port 80 on the host might be already used by other processes, as some users have already run into.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: {{ include "eks-pod-identity-agent.fullname" . }}
  namespace: {{ .Release.Namespace }}
...
spec:
  template:
    spec:
      hostNetwork: true
...
      initContainers:
...
          securityContext:
            privileged: true

Yet another issue I ran into when installing the Pod Identity agent, was an error that comes up when IPv6 is disabled on the nodes, which was the case with us. The solution was to set the --bind-hosts agent argument to explicitly configure IPv4 only.

Other Limitations of Pod Identity

There are some other limitations to Pod Identity, including:

• Only runs on EKS clusters, and must be running in AWS (no outpost clusters)
• Worker nodes in the cluster must be Linux Amazon EC2 instances
• EKS add-ons can only use IAM roles for service accounts, probably due to the need to access the agent running as DaemonSet

Conclusion

Although it’s a nice feature to be able to manage IAM roles for pods independently of the cluster and without touching any Kubernetes manifests, it seems at this point the architecture and software could be improved. If Pod Identity did not require a DaemonSet — and all the risks introduced — it would be something I would seriously consider moving to.

I’ll be keeping an eye on EKS Pod Identity to see if they address the problems.