Terraform creating role with missing AccessKeyId - amazon-web-services

Question

Terraform is creating role and attaching it to the EC2 instance successfully.
However, when I try to run commands with aws cli, it is giving error with missing AccessKeyId:
aws ec2 describe-instances --debug
2022-01-12 18:44:25,755 - MainThread - botocore.utils - DEBUG - Retrieved credentials is missing required field: AccessKeyId
2022-01-12 18:44:25,755 - MainThread - botocore.utils - DEBUG - Error response received when retrievingcredentials: {'Code': 'AssumeRoleUnauthorizedAccess', 'Message': 'EC2 cannot assume the role tf_eks_role_bastion. Please see documentation at https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_errors-info-doc.', 'LastUpdated': '2022-01-12T18:42:15Z'}.
My main.tf creates a role, attaches two policies to it, creates an instance-profile for the role and attaches the instance-profile to the newly created ec2-instance.
I got it working while changing main.tf and constanly re-applying the changes. But after executing terraform destroy and then terraform apply again, it stopped working again.
Also, when I create a role in AWS Console manually and attach it to the same ec2-instance, it starts working.
Does anyone understand this missing AccessKeyId error?
My main.tf:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.27"
}
}
required_version = ">= 0.14.9"
}
provider "aws" {
profile = "default"
region = "eu-central-1"
}
resource "aws_security_group" "http_sg" {
name = "tf_bastion_host allow ht_p inbound from anywhere"
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "ssh_sg" {
name = "tf_bastion_host allow ssh from anywhere"
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# create role for bastion host
resource "aws_iam_role" "eks_role" {
name = "tf_eks_role_bastion"
assume_role_policy = jsonencode({
Version = "2012-10-17"
"Statement" : [
{
"Effect" : "Allow",
"Principal" : {
"Service" : "eks.amazonaws.com"
},
"Action" : "sts:AssumeRole"
},
]
})
tags = {
tag-key = "tf_bastion_host"
}
}
# attach policy to role
resource "aws_iam_policy_attachment" "eks_attachment_cluster_policy" {
name = "tf_eks_attachment_cluster_policy"
roles = ["${aws_iam_role.eks_role.name}"]
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}
# attach policy to role
resource "aws_iam_policy_attachment" "eks_attachment_service_policy" {
name = "tf_eks_attachment_service_policy"
roles = ["${aws_iam_role.eks_role.name}"]
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
}
# create an instance profile
resource "aws_iam_instance_profile" "bastion_host_profile" {
name = "tf_bastion_host_profile"
role = aws_iam_role.eks_role.name
}
resource "aws_instance" "bastion_host" {
ami = "ami-05d34d340fb1d89e5"
instance_type = "t2.micro"
count = 1
associate_public_ip_address = true
# use the jenkins key-pair for now
key_name = "jenkins"
# attach the instance profile to the EC2 instance
iam_instance_profile = aws_iam_instance_profile.bastion_host_profile.name
vpc_security_group_ids = [
aws_security_group.http_sg.id,
aws_security_group.ssh_sg.id
]
user_data = file("installs.sh")
tags = {
Name = "tf_bastion_host",
Environment = "production"
}
}
Output of terraform apply -auto-approve:
Terraform used the selected providers to generate the
following execution plan. Resource actions are
indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_iam_instance_profile.bastion_host_profile will be created
+ resource "aws_iam_instance_profile" "bastion_host_profile" {
+ arn = (known after apply)
+ create_date = (known after apply)
+ id = (known after apply)
+ name = "tf_bastion_host_profile"
+ path = "/"
+ role = "tf_eks_role_bastion"
+ tags_all = (known after apply)
+ unique_id = (known after apply)
}
# aws_iam_policy_attachment.eks_attachment_cluster_policy will be created
+ resource "aws_iam_policy_attachment" "eks_attachment_cluster_policy" {
+ id = (known after apply)
+ name = "tf_eks_attachment_cluster_policy"
+ policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
+ roles = [
+ "tf_eks_role_bastion",
]
}
# aws_iam_policy_attachment.eks_attachment_service_policy will be created
+ resource "aws_iam_policy_attachment" "eks_attachment_service_policy" {
+ id = (known after apply)
+ name = "tf_eks_attachment_service_policy"
+ policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
+ roles = [
+ "tf_eks_role_bastion",
]
}
# aws_iam_role.eks_role will be created
+ resource "aws_iam_role" "eks_role" {
+ arn = (known after apply)
+ assume_role_policy = jsonencode(
{
+ Statement = [
+ {
+ Action = "sts:AssumeRole"
+ Effect = "Allow"
+ Principal = {
+ Service = "eks.amazonaws.com"
}
},
]
+ Version = "2012-10-17"
}
)
+ create_date = (known after apply)
+ force_detach_policies = false
+ id = (known after apply)
+ managed_policy_arns = (known after apply)
+ max_session_duration = 3600
+ name = "tf_eks_role_bastion"
+ name_prefix = (known after apply)
+ path = "/"
+ tags = {
+ "tag-key" = "tf_bastion_host"
}
+ tags_all = {
+ "tag-key" = "tf_bastion_host"
}
+ unique_id = (known after apply)
+ inline_policy {
+ name = (known after apply)
+ policy = (known after apply)
}
}
# aws_instance.bastion_host[0] will be created
+ resource "aws_instance" "bastion_host" {
+ ami = "ami-05d34d340fb1d89e5"
+ arn = (known after apply)
+ associate_public_ip_address = true
+ availability_zone = (known after apply)
+ cpu_core_count = (known after apply)
+ cpu_threads_per_core = (known after apply)
+ disable_api_termination = (known after apply)
+ ebs_optimized = (known after apply)
+ get_password_data = false
+ host_id = (known after apply)
+ iam_instance_profile = "tf_bastion_host_profile"
+ id = (known after apply)
+ instance_initiated_shutdown_behavior = (known after apply)
+ instance_state = (known after apply)
+ instance_type = "t2.micro"
+ ipv6_address_count = (known after apply)
+ ipv6_addresses = (known after apply)
+ key_name = "jenkins"
+ monitoring = (known after apply)
+ outpost_arn = (known after apply)
+ password_data = (known after apply)
+ placement_group = (known after apply)
+ placement_partition_number = (known after apply)
+ primary_network_interface_id = (known after apply)
+ private_dns = (known after apply)
+ private_ip = (known after apply)
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ subnet_id = (known after apply)
+ tags = {
+ "Environment" = "production"
+ "Name" = "tf_bastion_host"
}
+ tags_all = {
+ "Environment" = "production"
+ "Name" = "tf_bastion_host"
}
+ tenancy = (known after apply)
+ user_data = "f27e2f754e7658f0f0cdd09facb579d44b20ea5f"
+ user_data_base64 = (known after apply)
+ vpc_security_group_ids = (known after apply)
+ capacity_reservation_specification {
+ capacity_reservation_preference = (known after apply)
+ capacity_reservation_target {
+ capacity_reservation_id = (known after apply)
}
}
+ ebs_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ snapshot_id = (known after apply)
+ tags = (known after apply)
+ throughput = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
+ enclave_options {
+ enabled = (known after apply)
}
+ ephemeral_block_device {
+ device_name = (known after apply)
+ no_device = (known after apply)
+ virtual_name = (known after apply)
}
+ metadata_options {
+ http_endpoint = (known after apply)
+ http_put_response_hop_limit = (known after apply)
+ http_tokens = (known after apply)
}
+ network_interface {
+ delete_on_termination = (known after apply)
+ device_index = (known after apply)
+ network_interface_id = (known after apply)
}
+ root_block_device {
+ delete_on_termination = (known after apply)
+ device_name = (known after apply)
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ tags = (known after apply)
+ throughput = (known after apply)
+ volume_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
}
# aws_security_group.http_sg will be created
+ resource "aws_security_group" "http_sg" {
+ arn = (known after apply)
+ description = "Managed by Terraform"
+ egress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 0
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "-1"
+ security_groups = []
+ self = false
+ to_port = 0
},
]
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 80
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = []
+ self = false
+ to_port = 80
},
]
+ name = "tf_bastion_host allow ht_p inbound from anywhere"
+ name_prefix = (known after apply)
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags_all = (known after apply)
+ vpc_id = (known after apply)
}
# aws_security_group.ssh_sg will be created
+ resource "aws_security_group" "ssh_sg" {
+ arn = (known after apply)
+ description = "Managed by Terraform"
+ egress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 0
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "-1"
+ security_groups = []
+ self = false
+ to_port = 0
},
]
+ id = (known after apply)
+ ingress = [
+ {
+ cidr_blocks = [
+ "0.0.0.0/0",
]
+ description = ""
+ from_port = 22
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_groups = []
+ self = false
+ to_port = 22
},
]
+ name = "tf_bastion_host allow ssh from anywhere"
+ name_prefix = (known after apply)
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags_all = (known after apply)
+ vpc_id = (known after apply)
}
Plan: 7 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ private_instance_ip = (known after apply)
+ public_instance_dns = (known after apply)
aws_iam_role.eks_role: Creating...
aws_security_group.http_sg: Creating...
aws_security_group.ssh_sg: Creating...
aws_security_group.http_sg: Creation complete after 3s [id=sg-0ae7e9865c60ce9c9]
aws_security_group.ssh_sg: Creation complete after 3s [id=sg-016f588fb10a7dbad]
aws_iam_role.eks_role: Creation complete after 3s [id=tf_eks_role_bastion]
aws_iam_policy_attachment.eks_attachment_cluster_policy: Creating...
aws_iam_policy_attachment.eks_attachment_service_policy: Creating...
aws_iam_instance_profile.bastion_host_profile: Creating...
aws_iam_policy_attachment.eks_attachment_service_policy: Creation complete after 2s [id=tf_eks_attachment_service_policy]
aws_iam_instance_profile.bastion_host_profile: Creation complete after 2s [id=tf_bastion_host_profile]
aws_instance.bastion_host[0]: Creating...
aws_iam_policy_attachment.eks_attachment_cluster_policy: Creation complete after 2s [id=tf_eks_attachment_cluster_policy]
aws_instance.bastion_host[0]: Still creating... [10s elapsed]
aws_instance.bastion_host[0]: Still creating... [20s elapsed]
aws_instance.bastion_host[0]: Still creating... [30s elapsed]
aws_instance.bastion_host[0]: Creation complete after 39s [id=i-07926ae9044680939]
Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Answer 1

In the assume_role_policy of your IAM role
"Service" : "eks.amazonaws.com"
should be changed to
"Service" : "ec2.amazonaws.com"
If your role is going to be used by an EC2 instance, the allowed principal needs to be ec2.amazonaws.com. You might also want to review the managed policies you are attaching to the role, they are more suitable for an EKS cluster and not a bastion host.