Secure EKS Clusters with Falco

Falco is an open-source security tool for linux-based systems. Thanks to its plugin architecture, it can also be used to scan and monitor Kubernetes clusters. This article will cover some of the basics of Falco and show how it can be used to monitor AWS EKS clusters.

Security Scan Basics

There are two basic types of security scans for identifying vulnerabilities in software, networks and systems: “dynamic” and “static”.

Static scans are done before software and systems are deployed, in the coding or build phase. Source code or bytecode is analyzed to search for specific patterns. For example, layers of a docker container can be scanned against public bug or CVE databases. Static scans can also catch passwords or sensitive data hardcoded in source code or containers to help prevent security leaks.

Dynamic scans interact with running systems, simulating attacks, monitoring logs and network traffic and detecting suspicious activity.

Falco is primarily used for dynamic security scanning.

Do I Really Need To Worry About Cloud Security?

In case you believe your cloud provider will detect and protect you from all security breaches, think again!

There are countless examples of exploited cloud infrastructures — we’ve all heard about S3 bucket leaks. A 2002 survey of almost 5000 IT professionals have shared their real world experience with cloud security incidents. The survey revealed a more than 50% increase in security breaches over the previous year.

Cloud security should be seen as a shared effort between the provider and the customer. AWS, as an example, has a complete Security Maturity Model describing the security model it has in place, as well as recommended practices for customers to follow.

If you need more convincing, read about the SCARLETEEL case, in which an attacker was able to use a compromised container to obtain cloud credentials and use these credentials to perform crypto-mining and access sensitive data stored in the cloud.

How Falco Monitors AWS EKS Clusters

The kube-apiserver has the ability to generate fine-grained audit events for every request and every stage of its execution. If you install a Kubernetes cluster and manage the control plane yourself, you currently have two options available for capturing these events:

• Log backend, which writes events into the filesystem
• Webhook backend, which sends events to an external HTTP API

However, if you’re using a managed Kubernetes cluster in the cloud, such as AWS EKS, the webhook option is typically unavailable because it requires setting additional kube-apiserver flags, which requires access to control plane components. For cloud-managed Kubernetes clusters, this means only the audit log option is available.

Thankfully, Falco can also process events from audit logs. For EKS clusters, this is handled by the Falco k8saudit-eks plugin.

Falco Basics

Before diving into how to use the k8saudit-eks plugin, it’s important to understand some basic Falco principles.

Falco has three main configuration concepts:

• Rules: Conditions under which an alert should be generated. A rule is accompanied by a descriptive output string that is sent with the alert.
• Macros: Rule condition snippets that can be re-used inside rules and even other macros. You can think of “macros” as a fancy name for variables. They help to reduce duplication in rules.
• Lists: Collections of items that can be included in rules, macros, or other lists. Unlike rules and macros, lists cannot be parsed as filtering expressions.

A rules contains the actual definitions of the security scan . Let’s have a look at a rule from the k8s-auth plugin:

- rule: Create Privileged Pod
  desc: >
    Detect an attempt to start a pod with a privileged container
  condition: kevt and pod and kcreate and ka.req.pod.containers.privileged intersects (true) and not ka.req.pod.containers.image.repository in (k8s_audit_privileged_images)
  output: Pod started with privileged container (user=%ka.user.name pod=%ka.resp.name resource=%ka.target.resource ns=%ka.target.namespace images=%ka.req.pod.containers.image)
  priority: WARNING
  source: k8s_audit
  tags: [k8s]

Within a rule, the condition field contains the logic of the scan.

Notice the macros kevt and kcreate. The values of these macros can usually be found in the rules file. For example, the value of kcreate is:

- macro: kcreate
  condition: ka.verb=create

This macro uses another variable ka.verb, checking if the value is “create” — meaning a kubernetes object is created.

ka.verb and other variables used in the rule condition, such as ka.req.pod.containers.privileged, are built-in fields for the event source.

You will need to look at the documentation for each event source or plugin to see which fields are exposed. For example, the k8saudit README has a table with a list of fields and their definitions.

Lists are useful for rule conditions that need to allow or restrict multiple items. You can also append or replace other lists, such as those that defined in default rules. This allows you to customize default rules without needing to redefine them.

Let’s take an example of using lists to extend a list found in a default rule. In the rule for detecting privileged pods described earlier, the condition is comparing the container’s image repository with a pre-defined list of images (k8s_audit_privileged_images) which are known to run in privileged mode:

...
and not ka.req.pod.containers.image.repository in (k8s_audit_privileged_images)
...

The default list (partial):

- list: k8s_audit_privileged_images
  items: [
    falcosecurity/falco, docker.io/falcosecurity/falco, public.ecr.aws/falcosecurity/falco,
    docker.io/calico/node, calico/node,
    docker.io/cloudnativelabs/kube-router,
    docker.io/docker/ucp-agent,
    docker.io/mesosphere/mesos-slave,
    docker.io/rook/toolbox,
    docker.io/sysdig/sysdig,
    gcr.io/google_containers/kube-proxy,
...

In your Kubernetes cluster, you may have additional container images that you need to run in privileged mode. We can define a new list containing our own container images, then append to the k8s_audit_privileged_images list:

    - list: k8s_audit_privileged_images
      override:
        items: append
      items: [
        gcr.io/kaniko-project/executor,
        registry.myorganization.com/library/kaniko-aws,
        public.ecr.aws/gitlab/gitlab-runner-helper
      ]

This was a short overview of Falco rules syntax — be sure to check the official documentation to learn more about Falco rules syntax and other features.

The Falco “k8saudit-eks” Plugin

Now that we have an understanding of Falco basics, let’s move onto configuring Falco to monitor an EKS cluster.

Enabling EKS Cluster Audit Logs

Since the k8saudit-eks plugin reads from the audit log stored in Cloudwatch, we will need to enable audit logs in EKS first. How exactly to do this will depend on how you manage you cluster — i.e., terraform, cloud formation, CDK, manually.

To enable audit logs manually in the AWS web console, navigate to your cluster’s EKS dashboard then go to “manage logging”:

You can confirm audit logs are on by looking under the “Observability” tab of the EKS cluster dashboard:

Falco Base Installation

If you do not already have a base installation of Falco running on your EKS cluster, you need to install and configure Falco first. This can be easily done using the Falco helm chart.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

A good starting point is the values-k8saudit.yaml file in the Falco charts git repo.

The example values file for kubernetes installs the k8saudit plugin by default:

falco:
  rules_files:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit
      library_path: libk8saudit.so
      ...
    - name: json
      library_path: libjson.so
      init_config: ""
  # Plugins that Falco will load. Note: the same plugins are installed by the falcoctl-artifact-install init container.
  load_plugins: [k8saudit, json]

Enable the EKS Audit Plugin

Since we are going to monitor an EKS cluster, we will need to change the plugin config to look something like this:

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml #rules to use
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks    # replace k8saudit pluging
      library_path: libk8saudit-eks.so   # new library path for eks plugin
      init_config:
        region: ${REGION} #replace with your region
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: ${CLUSTER_NAME} #replace with your cluster name
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json] #plugins to load

What we are doing here is replacing the k8saudit plugin with the k8saudit-eks plugin. But don’t worry — the k8saudit-eks plugin extracts the same fields as the k8saudit plugin. This means you can use the same rules from the k8saudit plugin!

You do not need to copy the rules however, since Falco ships with a set of default rules, including k8saudit rules. Simply add it to the rules_file list in your helm values:

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d

The last thing you will need is to grant Falco an IAM role with policies to be able to read the eks audit logs stored in Cloudwatch. It is best practice to use IAM Roles for Serviceaccounts (IRSA) to associate the IAM role with the Falco serviceaccount.

The IAM role policy should be similar to this:

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"ReadAccessToCloudWatchLogs",
      "Effect":"Allow",
      "Action":[
        "logs:Describe*",
        "logs:FilterLogEvents",
        "logs:Get*",
        "logs:List*"
      ],
      "Resource":[
        "arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:*"
      ]
    }
  ]
}

You will also need to add a “trust policy” to the IAM role to allow it to be associated with the serviceaccount. Assuming the serviceaccount and namespace are named falco:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.eks.${REGION}.amazonaws.com/id/${OIDC_PROVIDER_ID}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.${REGION}.amazonaws.com/id/${OIDC_PROVIDER_ID}:sub": "system:serviceaccount:falco:falco"
                }
            }
        }
    ]
}

The OIDC_PROVIDER_ID can be found under IAM -> Identity Providers.

Finally, in the falco helm file values.yaml file, pass the IAM role arn:

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: {ROLE_ARN}

Check the official k8saudit-eks docs to make sure you get the latest IAM policies required.

Conclusion

Once you setup Falco to run inside your cluster, it will start processing the AWS EKS audit logs and send the output to whatever destination you have configured. For more information about routing Falco events, check the falcosidekick project, which is a tool that has built-in support for many popular target systems such as Prometheus, Slack, S3, DataDog, Elasticsearch and many many more.

Don’t forget to use lists and macros to override or extend the default rules as needed. This will help reduce false alarms and allow you to customize Falco for your needs.

Falco is quite easy to use and setup once you understand the basics. At my job, we are also using it as a general Kubernetes audit system — to alert us whenever someone tries to start a manual pod, do port-forwarding or exec into a pod. Although this information is in the cloudwatch EKS audit logs, using Falco to with its pre-built rules to extract these events is MUCH easier than trying to create something from scratch.

Stay safe out there!