Skip to content

Instantly share code, notes, and snippets.

@chhetripradeep

chhetripradeep/kube2iam_manual.md

Forked from snoby/kube2iam_manual.md
Created April 21, 2018 17:39
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save chhetripradeep/edd662d08b4d89ae84ea9c8546a78535 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save chhetripradeep/edd662d08b4d89ae84ea9c8546a78535 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @snoby snoby created this gist Jan 18, 2018.
    311 changes: 311 additions & 0 deletions kube2iam_manual.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,311 @@
    # kube2iam your security partner
    <!-- START doctoc generated TOC please keep comment here to allow auto update -->
    <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->


    - [Introduction](#introduction)
    - [Setup kube2iam in your cluster](#setup-kube2iam-in-your-cluster)
    - [Setting up the Role in AWS](#setting-up-the-role-in-aws)
    - [The Role](#the-role)
    - [Trust Relationship](#trust-relationship)
    - [How to define your deployments](#how-to-define-your-deployments)
    - [Success!](#success)
    - [Troubleshooting](#troubleshooting)
    - [Basic Debugging](#basic-debugging)
    - [Error Msg's...](#error-msgs)
    - [Another example](#another-example)
    - [Conclusion](#conclusion)
    - [More Resources:](#more-resources)

    <!-- END doctoc generated TOC please keep comment here to allow auto update -->


    ## Introduction
    The open source project kube2iam is Amazon aws specific. It allows your pod to have access to an aws role, without other pods on the same node having the same privledges. For example if your pod needs to use the aws cli command to copy / sync files with an s3 bucket you can use kube2iam in your cluster so that ONLY your pod has those rights.

    Amazon has stated at Kubecon 2017 that their EKS solution ( Amazon hosted Kubernetes cluster), that they intend to use kube2iam to handle roles and permissions for pods.

    ## Setup kube2iam in your cluster
    To have a seamless deployment of kube2iam in your cluster there are some steps to follow that aren't *exactly* in the readme files. The person responsible for deploying the kube2iam software should deploy it as a daemonset in the cluster so that every node has it. Also until the development team or ops team is comfortable with kube2iam you should probably deploy it with the `--verbose` and `--debug` flags.

    To insure that all the requests for credentials get caught, I personally would recommend using the `iptables=true` option. Althought from personal experience I have seen where removing this option or setting it to false doesn't remove the iptable entry. You actually have to restart the node. Finally if you are using a network overlay with Kubernetes ( if not you should be ), you need to tell it what host interface to use. Personally we use [Calico](https://www.projectcalico.org).

    My deployment file for kube2iam looks like this:

    ```
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
    name: kube2iam
    labels:
    app: kube2iam
    namespace: kube-system
    spec:
    updateStrategy:
    type: RollingUpdate
    template:
    metadata:
    labels:
    name: kube2iam
    spec:
    hostNetwork: true
    serviceAccount: kube2iam
    containers:
    - image: jtblin/kube2iam:latest
    name: kube2iam
    args:
    - "--auto-discover-base-arn"
    - "--iptables=true"
    - "--host-ip=$(HOST_IP)"
    - "--host-interface=cali+"
    - "--verbose"
    - "--debug"
    env:
    - name: HOST_IP
    valueFrom:
    fieldRef:
    fieldPath: status.podIP
    ports:
    - containerPort: 8181
    hostPort: 8181
    name: http
    securityContext:
    privileged: true
    ```

    One final note to mention. If you are using RBAC with your Kubernetes cluster you are going to need to setup some Kubernete Roles and Service Accounts for kube2iam. I didn't find a good reference online so I created the following:

    ```YAML
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: kube2iam
    namespace: kube-system
    ---
    apiVersion: v1
    items:
    - apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
    name: kube2iam
    rules:
    - apiGroups: [""]
    resources: ["namespaces","pods"]
    verbs: ["get","watch","list"]
    - apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
    name: kube2iam
    subjects:
    - kind: ServiceAccount
    name: kube2iam
    namespace: kube-system
    roleRef:
    kind: ClusterRole
    name: kube2iam
    apiGroup: rbac.authorization.k8s.io
    kind: List
    ```
    ## Setting up the Role in AWS
    ### The Role
    When you setup a role you are saying "I give permission for my software to have access to do X on Resource Y". A role is made up of a number of Amazon policies. For example here is the only policy attached to the role 'backup-2-s3':
    ```YAML
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": [
    "s3:ListBucket"
    ],
    "Resource": [
    "arn:aws:s3:::backup"
    ]
    },
    {
    "Effect": "Allow",
    "Action": [
    "s3:*"
    ],
    "Resource": [
    "arn:aws:s3:::backup/*"
    ]
    }
    ]
    ```

    I'm giving the user of this role the right to ListBuckets and to do whatever they want inside the s3 bucket named "backup".
    Let's assume that you've done this and you've deployed kube2iam with the above role and you can't get access to the s3 bucket. Well that's because you actually don't have permission to use this new role. You might have seen an error in the kube2iam logs like this:

    ```
    time="2018-01-17T20:12:01Z" level=info msg="GET /latest/meta-data/iam/security-credentials/ (200) took 9281 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials/ req.remote=100.106.175.8 res.duration=9281 res.status=200
    time="2018-01-17T20:12:01Z" level=error msg="Error assuming role AccessDenied: User: arn:aws:sts::221044447717:assumed-role/nodes.test.c.fobar.com/i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3\n\tstatus code: 403, request id: adf9683b-fbc2-11e7-8d79-93262b130f27" ns.name=test pod.iam.role="arn:aws:iam::221044447717:role/backup-2-s3" req.method=GET req.path=/latest/meta-data/iam/security-credentials/backup-2-s3 req.remote=100.106.175.8
    ```

    Why? keep reading...


    ### Trust Relationship
    This is the topic that confused me for a long time. It wasn't until i sat down and worked through some examples did I really understand what was happening with the trust relationship. The online AWS documentation describes configuring and using a role as:
    > When you run commands using the role profile, the AWS CLI uses the source profile's credentials to call AWS Security Token Service and assume the specified role. The source profile must have permission to call sts:assume-role against the role, and the role must have a trust relationship with the source profile to allow itself to be assumed.
    What does this mean??? We have 2 roles:
    1. The first role `Role 1` is the kubernetes worker node role. This is the role that every worker node has associated to it in your Kubernetes cluster. This role has numerous policies attached to it that give the role the right to DescribeInstances and get certain directories in certain s3 buckets that kops uses to setup your nodes.
    2. `Role 2` is the role that does the special work that you want, in this case to have access to a specific s3 bucket. **This is the role that will be assumed**.

    You must allow `Role 2` to be assumed by `Role 1`. So two additional steps must be done here:

    1. You have to modify `Role 2` so that it can be *Assumed* by `Role 1`. You do this by modifying the trust relationship. You trust `Role 2` to give it the action to assume `Role 1`.

    ```
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Principal": {
    "Service": "ec2.amazonaws.com"
    },
    "Action": "sts:AssumeRole"
    },
    {
    "Sid": "",
    "Effect": "Allow",
    "Principal": {
    "AWS": [
    "arn:aws:iam::221044447717:role/nodes.k8s.baz.foobar.io"
    ]
    },
    "Action": "sts:AssumeRole"
    }
    ]
    }
    ```

    2. You now need to add the policy to allow your Kubernete worker nodes to assume roles that are not in their default role. In kops you can do this as an additional policy. Or you can just attach a new policy to your existing node role:

    ```
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "",
    "Effect": "Allow",
    "Action": [
    "sts:AssumeRole"
    ],
    "Resource": [
    "arn:aws:iam::221044447717:instance-profile/backup-2-s3"
    ]
    }
    ]
    }
    ```

    One thing that you could do in this additional role, since you don't want to have to add additional fields every time you add a resource you want this node to assume, you could change the `arn:aws:iam::221044447717:instance-profile/backup-2-s3` to a `*`. That way you wouldn't have to edit it every time you want to add a new role for that node to assume.

    ### How to define your deployments
    Using kube2iam in your cluster is actually pretty easy. In your deployment file have an annotation in the metadata that specifies the name of the role that you want to use.
    For example to use my backup s3 role I would add the line `iam.amazonaws.com/role: backup-2-s3` under annotations:

    ```YAML
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: remote
    namespace: default
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: remote
    minReadySeconds: 5
    template:
    metadata:
    labels:
    app: remote
    annotations:
    iam.amazonaws.com/role: backup-2-s3
    spec:
    containers:
    - name: remote
    image: "iotapi322/worker:v4"
    resources:
    limits:
    cpu: "200m"
    memory: "256Mi"
    requests:
    cpu: "100m"
    memory: "128Mi"
    ```
    ### Success!
    To show if you truly hvae access to the s3 bucket you can do two things.
    1. Check if you get the correct role value back
    2. Try to use the resource
    Well after the above steps, I verified that I was getting the correct role back:
    ```
    root@remote-79cdc8dd46-mg2g5:/# curl 169.254.169.254/latest/meta-data/iam/security-credentials/
    backup-2-S3
    ```
    and I verfied that I could list the contents of my s3 bucket:
    ```
    root@remote-79cdc8dd46-mg2g5:/# aws s3 ls s3://backup/EFSshares/
    PRE api-adventurebot-01/
    PRE api-adventurebot-02/
    PRE hello-world-nginx/
    PRE lockbot-db2/
    PRE lockbot/
    PRE test-permissions/
    PRE test/
    ```



    ## Troubleshooting
    Troubleshooting permissions and Roles in AWS can be frustrating, however I've found that the logs from kube2iam to be really quite good. When in doubt find the node that your pod is running on. Next find the kube2iam pod that is also running on that node and start tailing the log.

    ### Basic Debugging

    You can curl inside your pod to the meta-data ip address to find out what your role is. If you have specified an annotation and you don't get that annotation your deployment is probably setup wrong.

    Send your curl request set to:
    `curl 169.254.169.254/latest/meta-data/iam/security-credentials/` (you have to have the trailing slash), you should get returned the name of the role that you placed in the annotation. If not something has gone awry.

    I would suggest going into the kubernetes dashboard and finding which node your container is running on, then find the kube2iam pod that is running on that same node and looking at the kube2iam logs.

    ### Error Msg's...
    If you don't have the trust relationship in place you might see the following log lines from the kube2iam pod

    ```
    time="2018-01-18T16:12:39Z" level=info msg="GET /latest/meta-data/iam/security-credentials (301) took 802140 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials req.remote=100.96.7.199 res.duration=802140 res.status=301
    time="2018-01-17T20:12:01Z" level=error msg="Error assuming role AccessDenied: User: arn:aws:sts::221044447717:assumed-role/nodes.test.c.tropo.com/i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3\n\tstatus code: 403, request id: adf9683b-fbc2-11e7-8d79-93262b130f27" ns.name=test pod.iam.role="arn:aws:iam::221044447717:role/backup-2-s3" req.method=GET req.path=/latest/meta-data/iam/security-credentials/backup-2-s3 req.remote=100.106.175.8
    ```
    The highlights from the above error blog are:

    **i-0d803755a438f8069 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::221044447717:role/backup-2-s3**.

    This is showing that the trust relationship has not been set on the worker node.

    ### Another example

    When you remote into your pod and run the aws cli command ` aws s3 ls s3://<s3-bucket>/` you will get back an error like this:

    `Unable to locate credentials. You can configure credentials by running "aws configure".`

    This is because you don't have a trust relationship setup between the **kubernetes node** and your **role**

    ## Conclusion
    I hope this helps clear up some of the confusion with Roles and Trust Relationships. kube2iam is a very powerful and necessary tool in your InfoSec toolbox.

    ## More Resources:
    * [kube2iam](https://github.com/jtblin/kube2iam/)
    * [Example configuration](https://github.com/jtblin/kube2iam/issues/58)
    * [Amazon Docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html#cli-role-prepare)