Last active
May 9, 2023 23:30
-
-
Save hugotkk/29bd55f59d39f268862d19f855eef9bb to your computer and use it in GitHub Desktop.
Revisions
-
hugotkk revised this gist
May 9, 2023 . 1 changed file with 3 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -4,6 +4,8 @@ This is a setup guide for amazon-eks-pod-identity-webhook on self-hosted k3s clu - https://github.com/aws/amazon-eks-pod-identity-webhook/tree/master - https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md The goal in this example is to config the pod to have full S3 access to the AWS account. ## Setup ### AWS @@ -19,7 +21,7 @@ This is a setup guide for amazon-eks-pod-identity-webhook on self-hosted k3s clu 3. Input "Provider Name" as your S3 bucket URL, for example, `s3.us-east-1.amazonaws.com/test2023a`. 4. Set "Audiences" as `k3s`. ## IAM Role (for the pod. this role will be referenced in the sa config) 1. Create a new IAM role. 2. Name it as `myrole`. 3. Set up trusted entities. -
May 9, 2023 . 1 changed file with 4 additions and 5 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -100,7 +100,7 @@ keys.json Details are in `test.sh`. 1. Apply `test.yaml` to create test pod and sa. Make sure the webhook is running. If it's not, re-install the cert manager and webhook by re-running the command. ``` [root@node2 ~]# kubectl get po,sa @@ -120,7 +120,7 @@ serviceaccount/mysa 0 4m54s OK: 143 MiB in 42 packages ``` 3. Test it by listing the contents of the S3 bucket and get caller-identity. ``` [root@node2 ~]# kubectl exec my-pod -- aws sts get-caller-identity @@ -133,10 +133,9 @@ OK: 143 MiB in 42 packages ``` [root@node2 ~]# kubectl exec my-pod -- aws s3 ls 2023-05-09 21:58:22 test2023a ``` 4. If the call to other AWS services, it should fail. ``` [root@node2 ~]# kubectl exec my-pod -- aws ec2 describe-instances --region us-east-1 -
May 9, 2023 . No changes.There are no files selected for viewing
-
May 9, 2023 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,145 @@ This is a setup guide for amazon-eks-pod-identity-webhook on self-hosted k3s cluster - https://github.com/aws/amazon-eks-pod-identity-webhook/tree/master - https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md ## Setup ### AWS ## S3 1. Create an S3 bucket. 2. Navigate to "Permissions" > "Block public access" and disable public ACL block. 3. Enable ACLs by going to "Permissions" > "Object OwnershipInfo". ## Identity Provider 1. Go to IAM > Identity providers > Add provider. 2. Select "OpenID Connect". 3. Input "Provider Name" as your S3 bucket URL, for example, `s3.us-east-1.amazonaws.com/test2023a`. 4. Set "Audiences" as `k3s`. ## IAM Role 1. Create a new IAM role. 2. Name it as `myrole`. 3. Set up trusted entities. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "<use the ARN of created identity provider>" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } ``` 4. Attach `AmazonS3FullAccess` policy to the role. ### k3s Details mentioned in `command.sh`. This script will 1. Install k3s 2. Install cert manager. 3. Install webhook. 4. Create an identity provider with S3. Examples: discovery.json ``` { "issuer": "https://s3.us-east-1.amazonaws.com/test2023a", "jwks_uri": "https://s3.us-east-1.amazonaws.com/test2023a/keys.json", "authorization_endpoint": "urn:kubernetes:programmatic_authorization", "response_types_supported": [ "id_token" ], "subject_types_supported": [ "public" ], "id_token_signing_alg_values_supported": [ "RS256" ], "claims_supported": [ "sub", "iss" ] } ``` keys.json ``` { "keys": [ { "use": "sig", "kty": "RSA", "kid": "dChUxQw2A6HpqZamkglk2Iq_paXdv-CB6j7-9MzbiZE", "alg": "RS256", "n": "uqELZImmnbm-X4IoYOctoW5FbPWE0wBYtTeYxJrOEIkVIXS48Q5dL8Zoiv-5y87dnWVRm1upIVqVV9funsGDTeUfUPMc8LgAp3yp1SvcMttM3s32HMRotd6VhkpryRENQeCn7HxtbFCDOWiQJr0TVV1l1u0neyveyX_XWqMOdNH7GN9S_AHzyERCCpH9oykzgzIES87rnevnY8y243DhbZHFMRW3FG2STYyNQ06KX3hz7624DXpDxvXHtAd8iJeva_xA6oBO7ZpF7zzPvs3lfFie2leubK_bCAjlNXnaa6hLxSUVTclHoRgXx_B0bmqkqImUCQY-Yed1ij6l8XlMKQ", "e": "AQAB" }, { "use": "sig", "kty": "RSA", "kid": "", "alg": "RS256", "n": "uqELZImmnbm-X4IoYOctoW5FbPWE0wBYtTeYxJrOEIkVIXS48Q5dL8Zoiv-5y87dnWVRm1upIVqVV9funsGDTeUfUPMc8LgAp3yp1SvcMttM3s32HMRotd6VhkpryRENQeCn7HxtbFCDOWiQJr0TVV1l1u0neyveyX_XWqMOdNH7GN9S_AHzyERCCpH9oykzgzIES87rnevnY8y243DhbZHFMRW3FG2STYyNQ06KX3hz7624DXpDxvXHtAd8iJeva_xA6oBO7ZpF7zzPvs3lfFie2leubK_bCAjlNXnaa6hLxSUVTclHoRgXx_B0bmqkqImUCQY-Yed1ij6l8XlMKQ", "e": "AQAB" } ] } ``` 4. Configure apiserver to use the identity provider. ## Testing Details are in `test.sh`. 1. Apply test.yaml to create test pod and sa. Make sure the webhook is running. If it's not, re-install the cert manager and webhook by re-running the command. ``` [root@node2 ~]# kubectl get po,sa NAME READY STATUS RESTARTS AGE pod/pod-identity-webhook-c4fb89547-rxdcz 1/1 Running 0 13m pod/my-pod 1/1 Running 0 4m54s NAME SECRETS AGE serviceaccount/default 0 14m serviceaccount/pod-identity-webhook 0 13m serviceaccount/mysa 0 4m54s ``` 2. Install AWS CLI. ``` [root@node2 ~]# kubectl exec my-pod -- apk add aws-cli OK: 143 MiB in 42 packages ``` 3. Test it by listing the contents of the S3 bucket. ``` [root@node2 ~]# kubectl exec my-pod -- aws sts get-caller-identity { "UserId": "AROATKPGOJR2CXJBT2DPN:botocore-session-1683672493", "Account": "<ACCOUNT_ID>", "Arn": "arn:aws:sts::<ACCOUNT_ID>:assumed-role/myrole/botocore-session-1683672493" } ``` ``` [root@node2 ~]# kubectl exec my-pod -- aws s3 ls 2023-05-09 21:58:22 test2023a shouble fail to call other aws service ``` 4. If the call to other AWS servicesm it should fail. ``` [root@node2 ~]# kubectl exec my-pod -- aws ec2 describe-instances --region us-east-1 An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation. command ``` This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,74 @@ #!/bin/env bash # Install k3s curl -sfL https://get.k3s.io | sh - # Install Cert Manager kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml # Install webhook git clone https://github.com/aws/amazon-eks-pod-identity-webhook.git cd amazon-eks-pod-identity-webhook/ yum install -y golang jq make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:latest # Create Identity Provider (s3) PRIV_KEY="sa-signer.key" PUB_KEY="sa-signer.key.pub" PKCS_KEY="sa-signer-pkcs8.pub" ssh-keygen -t rsa -b 2048 -f $PRIV_KEY -m pem -N '' ssh-keygen -e -m PKCS8 -f $PUB_KEY > $PKCS_KEY export S3_BUCKET=test2023a export AWS_REGION=us-east-1 _bucket_name=$(aws s3api list-buckets --query "Buckets[?Name=='$S3_BUCKET'].Name | [0]" --out text) if [ $_bucket_name == "None" ]; then if [ "$AWS_REGION" == "us-east-1" ]; then aws s3api create-bucket --bucket $S3_BUCKET else aws s3api create-bucket --bucket $S3_BUCKET --create-bucket-configuration LocationConstraint=$AWS_REGION fi fi echo "export S3_BUCKET=$S3_BUCKET" export HOSTNAME=s3.$AWS_REGION.amazonaws.com export ISSUER_HOSTPATH=$HOSTNAME/$S3_BUCKET go run ./hack/self-hosted/main.go -key $PKCS_KEY | jq '.keys += [.keys[0]] | .keys[1].kid = ""' > keys.json cat <<EOF > discovery.json { "issuer": "https://$ISSUER_HOSTPATH", "jwks_uri": "https://$ISSUER_HOSTPATH/keys.json", "authorization_endpoint": "urn:kubernetes:programmatic_authorization", "response_types_supported": [ "id_token" ], "subject_types_supported": [ "public" ], "id_token_signing_alg_values_supported": [ "RS256" ], "claims_supported": [ "sub", "iss" ] } EOF aws s3 cp --acl public-read ./keys.json s3://$S3_BUCKET/keys.json aws s3 cp --acl public-read ./discovery.json s3://$S3_BUCKET/.well-known/openid-configuration mkdir -p /etc/rancher/k3s/ cat <<EOF > /etc/rancher/k3s/config.yaml kube-apiserver-arg: - --service-account-key-file=$PWD/sa-signer-pkcs8.pub - --service-account-signing-key-file=$PWD/sa-signer.key - --service-account-issuer=https://$ISSUER_HOSTPATH #- --api-audiences=k3s EOF systemctl restart k3s kubectl apply -f test.yaml This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,7 @@ #!/bin/env bash kubectl apply -f test.yaml kubectl exec my-pod -- apk add aws-cli kubectl exec my-pod -- aws sts get-caller-identity kubectl exec my-pod -- aws s3 ls kubectl exec my-pod -- aws ec2 describe-instances --region us-east-1 This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,25 @@ apiVersion: v1 kind: ServiceAccount metadata: name: mysa namespace: default annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT>:role/myrole" eks.amazonaws.com/audience: "k3s" eks.amazonaws.com/sts-regional-endpoints: "true" eks.amazonaws.com/token-expiration: "86400" --- apiVersion: v1 kind: Pod metadata: name: my-pod spec: serviceAccountName: mysa containers: - name: alpine image: alpine:latest command: ["/bin/sh"] args: ["-c", "while true; do echo 'Hello, World!'; sleep 10; done"]