AWS / Heptio Authenticator from an EC2 instance - amazon-web-services

Question

So I've followed the AWS article Deploying the Heptio Authenticator to kops, and I was able to successfully able to get things running. From my laptop, which has a fully configured AWSCLI setup, I can run any kubectl command since I'm an administrator. The token is obtained by using the following command in my ~/.kube/config file:
users:
- name: mycluster-exec
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- token
- --cluster-id
- mycluster
- --role
- arn:aws:iam::<account-number>:role/KubernetesAdministrator
command: aws-iam-authenticator
env: null
Now what I'd like to do is essentially described in the GitHub issue EKS heptio authentication using IAM without AWSCLI. I'm not using EKS, but the principle is the same. I have EC2 instances that are running my CI system's build agents, and I would like those build agents to NOT have static, hard-coded credentials (i.e. a static AWS ID and Secret Key). I'd prefer those nodes to also use the aws-iam-authenticator binary to temporarily grab credentials as needed to deploy / make changes to my Kubernetes clusters.
I created a Role / Instance profile called KubernetesCIRole, and attached that IAM role to my EC2 build agent nodes. I then added the following to the ConfigMap:
apiVersion: v1
data:
config.yaml: |
clusterID: mycluster
server:
mapRoles:
- roleARN: arn:aws:iam::<account-number>:role/KubernetesAdministrator
username: kubernetes-admin
groups:
- system:masters
- roleARN: arn:aws:iam::<account-number>:role/KubernetesCIRole
username: kubernetes-admin
groups:
- system:masters
kind: ConfigMap
metadata:
labels:
k8s-app: heptio-authenticator-aws
name: heptio-authenticator-aws
namespace: kube-system
However, when I configure the ~/.kube/config on that EC2 build agent machine, and then run something simple like kubectl --v=10 get pods, I receive the following:
I0828 10:16:30.605964 5196 loader.go:359] Config loaded from file /home/ubuntu/.kube/config
I0828 10:16:30.606744 5196 loader.go:359] Config loaded from file /home/ubuntu/.kube/config
I0828 10:16:30.607704 5196 loader.go:359] Config loaded from file /home/ubuntu/.kube/config
...
I0828 10:16:33.092683 5196 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.11.2 (linux/amd64) kubernetes/bb9ffb1" 'https://api.mycluster.k8s.mycompany.com/api?timeout=32s'
I0828 10:16:33.605698 5196 round_trippers.go:405] GET https://api.mycluster.k8s.mycompany.com/api?timeout=32s 401 Unauthorized in 512 milliseconds
I0828 10:16:33.605727 5196 round_trippers.go:411] Response Headers:
I0828 10:16:33.605734 5196 round_trippers.go:414] Content-Type: application/json
I0828 10:16:33.605749 5196 round_trippers.go:414] Www-Authenticate: Basic realm="kubernetes-master"
I0828 10:16:33.605758 5196 round_trippers.go:414] Content-Length: 129
I0828 10:16:33.605767 5196 round_trippers.go:414] Date: Tue, 28 Aug 2018 10:16:33 GMT
I0828 10:16:33.608443 5196 request.go:897] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0828 10:16:33.610858 5196 cached_discovery.go:111] skipped caching discovery info due to Unauthorized
F0828 10:16:33.610901 5196 helpers.go:119] error: the server doesn't have a resource type "pods"
Why doesn't this work? If I've specified in the Heptio / AWS IAM Authenticator ConfigMap that a given role has cluster administrator access (or anything else, for that matter), shouldn't I be able to authenticate?
Thanks for the help in advance!

Answer 1

Finally figured it out. The AWS documentation Managing Users or IAM Roles for your Cluster has the part that's needed:
First, you need to create an IAM role (and corresponding instance profile) which has the following basic permissions, and attach it to your EC2 instance:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Then you need to update the ConfigMap with something like the following:
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: <ARN of instance role (not instance profile)>
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:masters
The key part is the username system:node:{{EC2PrivateDNSName}}. I guess the {{EC2PrivateDNSName}} serves as a placeholder for any EC2 instance with the corresponding rolearn attached to it.