Centralized Secrets with ESO and AWS Secrets Manager

If you manage several Kubernetes clusters across different AWS accounts, you know it can be challenging to manage Kubernetes secrets across all of those clusters. This article describes a method to store secrets values in AWS Secrets Manager in a single account and use those secrets in Kubernetes clusters running in different AWS accounts.

Furthermore, if you have secret values that are common to all Kubernetes clusters, such as license keys or API tokens, storing these values in a single account makes it easier to rotate the secrets and to control access to them.

The approach uses the following concepts:

• AWS Secrets Manager
• AWS IAM Role in the account storing the secret
• AWS IAM Role associated with the ESO SecretStore / ClusterSecretStore
• ESO [Cluster]SecretStore and [Cluster]ExternalSecret

If you are not familiar with the External Secrets Operator (ESO), I’ve written an introductory article. To keep this article short, I won’t go into too many details of how ESO works.

Article Overview

AWS Secrets Manager

As mentioned, we will use AWS Secrets Manager as our secret store. This strategy can also be used for secrets stored in AWS SSM Parameter Store.

Choose an AWS account that will act as the “central” secret store and create the secret. A single AWS Secrets Manager secret can also be used to store multiple secrets in key/value form. If you have multiple secret values you want to share, using the key/value structured data option could reduce coding and configuration overhead.

Here is a terraform code snippet

resource "aws_secretsmanager_secret" "eso_shared_secret" {
  name        = "my-shared-secret"
  description = "Shared secret"
}

resource "aws_secretsmanager_secret_policy" "eso_shared_secret" {
  secret_arn = aws_secretsmanager_secret.eso_shared_secret.arn
  policy     = data.aws_iam_policy_document.eso.json
}

data "aws_iam_policy_document" "eso" {
  statement {
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [module.eso_shared_secret_role.iam_role_arn]
    }
    actions = [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
    ]
    resources = ["*"]
  }
}

Notice the aws_iam_policy_document attached to the secret contains an IAM role identifier. This role is the one that will be assumed by the ESO ClusterSecretStore and we will discuss this in the next section. The policy ensures that the role is only able to read the secret.

AWS IAM Role to be Assumed

In the same AWS account where the secret was created, we will need a role that can access that secret in a read-only way. This will be configured later as the role to assume by ClusterSecretStore for retrieving the secret.

In this terraform snippet, the community iam-assumable-role module is used to create the role.

module "eso_shared_secret_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role"
  trusted_role_arns = [<list of cross-account IAM roles that can assume this role>]
  create_role       = true
  role_name         = "eso-shared-secret-assumable-role"
  custom_role_policy_arns = [
    aws_iam_policy.eso_shared.arn
  ]
  number_of_custom_role_policy_arns = 1
  role_requires_mfa                 = false
}

resource "aws_iam_policy" "eso_shared" {
  name        = "eso-shared-secret-policy"
  description = "ESO ClusterSecretStore policy"
  policy      = data.aws_iam_policy_document.eso_shared_secret.json
}

data "aws_iam_policy_document" "eso_shared_secret" {
  statement {
    actions = [
      "secretsmanager:GetResourcePolicy",
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
      "secretsmanager:ListSecrets",
      "secretsmanager:ListSecretVersionIds",
    ]
    resources = [aws_secretsmanager_secret.eso_shared_secret.arn]
  }

Note that later we will need to pass to trusted_role_arns a list of IAM role arns from other accounts that are allowed to assume this role. Also note that we have an identity-based policy on the role to restrict what actions can be performed by the role. This is important since this role will be assumed by other accounts, and we want to strictly limit the actions that role is allowed to perform.

IAM “IRSA” Role Used By ESO

With the ESO Objects ClusterSecretStore and SecretStore, it is possible to pass an IAM role by using a serviceaccount annotated with an IAM role that is configured as an “IRSA” role. We won’t cover the details of creating an IRSA role, however the terraform community module iam-role-for-service-accounts-eks can help create such a role.

Once you create the IRSA role, you will need to attach an IAM policy to allow the role to assume the IAM role in our “central” secrets account. How this is done will depend on how the role was created:

data "aws_iam_policy_document" "eso_external_secrets_assume_role" {
  statement {
    effect = "Allow"
    actions = [
      "sts:AssumeRole",
    ]
    resources = ["arn:aws:iam::<aws-secrets-account-id>:role/<secrets-account-role-name>"]
  }
}

Next, we associate the IRSA role with the serviceaccount used by the ClusterSecretStore (or SecretStore) using an annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/<my-role-name>
  name: eso-default
  namespace: external-secrets

Then we specify this role in the serviceAccountRef of the ClusterSecretStore (or SecretStore):

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
spec:
...
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            name: eso-default  # serviceaccount with the IRSA annotation
            namespace: external-secrets
...

ClusterSecretStore / SecretStore Assume Role

Returning back to our example ClusterSecretStore definition, we need to pass the IAM role we want to be assumed (the role in the “central” secrets account):

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: shared-secrets
spec:
  conditions:
  - namespaces:
    - myapp
    - kube-system
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            name: eso-default
            namespace: external-secrets
      region: eu-west-1
      role: "arn:aws:iam::<aws-secrets-account-id>:role/<secrets-account-role-name>"
      service: SecretsManager

I know there is a lot going on here, so let’s summarize before we continue:

1. We created an AWS Secrets Manager secret in account “A”
2. We created an IAM role that can read this secret, also in account “A”
3. For our Kubernetes cluster running in account “B”, we created an IRSA enabled IAM role allowed to assume the role in account “A”
4. We associated the IRSA role with the serviceaccount used by the ClusterSecretStore in account “B”
5. We specified that the ClusterSecretStore should use the role in account “A” to retrieve the secret

ClusterExternalSecret

This is the final step, which will create the actual Kubernetes secret that can be used by apps. In this example, I am using a ClusterExternalSecret in order to have the secret created in multiple namespaces, but an ExternalSecret can also be used for a single namespace.

The syntax is a bit tricky, so please refer to the ClusterExternalSecret documentation for an explanation of each field.

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  labels:
  name: my-shared-secret
spec:
  externalSecretName: my-shared-secret
  externalSecretSpec:
    data:
      - remoteRef:
          conversionStrategy: Default
          decodingStrategy: None
          key: <secrets-manager-secret-name>  # the name of the AWS Secrets Manager secret
          property: MY_SECRET
        secretKey: MY_SECRET
    refreshInterval: 1h
    secretStoreRef:
      kind: ClusterSecretStore
      name: shared-secrets  # use to the ClusterSecretStore we created
    target:
      creationPolicy: Owner
      deletionPolicy: Retain
      name: shared-secrets  # the kubernetes secret name
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - some-other-namespace   # list of namespaces to create the secret
          - kube-system
  refreshTime: 10m

If everything goes well, after a short while you should see the Kubernetes secret in the namespace(s):

❯ kubectl get secret
NAME                         TYPE     DATA   AGE
shared-secrets   Opaque   1      14d

❯ kubectl get secret shared-secrets -o yaml
apiVersion: v1
data:
  MY_SECRET: <base64 hash encoded secret>
kind: Secret
metadata:
  name: shared-secrets
  namespace: kube-system
type: Opaque

Summary

This took a few steps to get running, but once you create the AWS Secrets Manager secret, the IAM roles and the ClusterSecretStore, you can reuse those components to create other secrets.

With this scalable approach we are able to centrally manage secret values (e.g., rotate the secret) across many different Kubernetes clusters in different AWS accounts.

Of course, please evaluate whether this approach meets the security standards for your organization, and be sure to control the IAM policies for identities and resources.