I am receiving the following errors in the EC2 CloudWatch Agent logs, /var/logs/awslogs.log:
I verified the EC2 has a role:
And the role has the correct policies:
I have set the correct region in /etc/awslogs/awscli.conf:
I noticed that running aws configure list in the EC2 gives this:
Is this incorrect? Should it list the profile (EC2_Cloudwatch_Profile) there?
I was using terraform and reprovisioning by doing:
terraform destroy && terraform apply
Looks like due to IAM being a global service it is "eventually consistent" and not "immediately consistent", when the profile instance was destroyed, the terraform apply began too quickly. Despite the "destroy" being complete, the arn for the previous profile instance was still there, and was re-used. However, the ID changed to a new ID.
Replacing the EC2 would bring it up to speed with the correct ID. However, my solution is to just wait longer between terraform destroy and apply.
Related
Steps that brought this error
Disassociated a instance profile from an ec2 instance (successful)
Associating a new instance profile to the EC2 instance . This fails with an error:
Value (arn:aws:iam::1234556:instance-profile/test-instance-profile) for parameter iamInstanceProfile.arn is invalid. Invalid IAM Instance Profile ARN
code
client.associate_iam_instance_profile(
IamInstanceProfile={
'Arn': 'arn:aws:iam::1234556:instance-profile/test-instance-profile'
},
InstanceId=instance_id
)
I verified the the instance profile exists before associating it
Based on the comments, trust policy in the IAM role was missing for EC2
The delay you are seeing for InstanceProfile is intended; this is to account for and ensure the IAM service has propagated the profile fully. We do apologize for any inconvenience this may cause.
AWS::IAM::InstanceProfile resources always take exactly 2 minutes to create
Even though the above post is a bit old, and AWS mimimise the delay, but it still exists.
Invalid IAM Instance Profile name #15341
This happened to me as well. There seems to be some race condition for newly created profiles. When I wait a short period and rerun the terraform it succeeds.
This is a strange one... If I click on the instance id, and then navigate to security, it tells me the instance has role X. Then I back out to view all instances, mark the checkbox for the instance in question, go to Actions -> Security -> Modify IAM Role, and it shows me a different role, role Y. I then try to set it to No IAM Role (or any various role), and I get this error:
"Multiple roles associated to instance
The selected instance has more than one IAM role associated. This usually occurs when the instance is in the process of replacing an existing instance profile association. "
I have no idea what to do because I didn't think an EC2 instance was supposed to be able to have two roles... nothing can assume two roles at once, anyway. So this feels like a bug... can anyone help me solve this?
I had the same issue and it seems that when replacing the instance profile it somehow stays in a state that's not completely associated.
Using the CLI we can see the status of the profile association:
aws ec2 describe-iam-instance-profile-associations
In my case the problematic profile was showing as "associating", while all the others show "associated".
Get the AssociationID for the problematic association and disassociate it with the command
aws ec2 disassociate-iam-instance-profile --association-id iip-assoc-xxxxxx
After that you should see the previous profile you had originally and everything should be consistent.
Hope it helps solving the problem.
I wanted to expound on Nelson Brito's answer since I found a way to return your instance to a normal state. I ran into this situation yesterday when helping a user, and I observed my instance with two profile associations -- one in a state of associating and one in a state of disassociating. The command to find this was:
aws ec2 describe-iam-instance-profile-associations --filters Name=instance-id,Values=i-xxxxxx
To fix the issue, I first removed the associating profile using the command:
aws ec2 disassociate-iam-instance-profile --association-id iip-assoc-xxxxxx
Next, I went to the console and detached the instance from all profiles (there is probably a CLI invocation, but I didn't figure it out). When done, you should have a clean instance:
aws ec2 describe-iam-instance-profile-associations --filters Name=instance-id,Values=i-xxxxxx
{
"IamInstanceProfileAssociations": []
}
Here's where we get to the root cause. When I re-assigned the role that was previously stuck in associating, well, it remained stuck in associating. The root cause of my problem was that the user had created the role without setting a trust relationship with EC2. The fix was two parts:
Update the instance trust relationship to add "Service": "ec2.amazonaws.com" as an allowed principal
Use the CLI to disassociate the role and add it again
tl;dr - If you ever run into this, ensure that the role you're trying to assign to your ec2 instance can be assumed by your ec2 instance.
We had the issue Multiple roles associated to instance after bringing an existing EC2 into CloudFormation. Nelson Brito's answer helped to resolve the issue.
To avoid the issue altogether when bringing an existing EC2 into CloudFormation, I now recommend
first disassociate the IAM role (for example via the AWS Console)
import the EC2 only, i.e. without InstanceProfile and Role in the import template
add InstanceProfile and Role to the template and update the stack.
While creating AWS EMR cluster, always i get the issue- Service role EMR_DefaultRole has insufficient EC2 permissions
And the cluster terminates automatically, have even done steps as per aws documentation of recreating emr specific roles, but no progress please guide how to resolve the issue- Service role EMR_DefaultRole has insufficient EC2 permissions
EMR needs two roles to start the cluster 1) EC2 Instance profile role 2)EMR Service role. The service role should have enough permissions to provision new resources to start the cluster, EC2 instances, their network etc. There could be many reasons for this common error:
Verify the resources and their actions. Refer https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-role.html.
Check if you are passing the tag that signifies if cluster needs to use emr managed policy.
{
"Key": "for-use-with-amazon-emr-managed-policies",
"Value": "true"
}
At last try to find out the exact reason from cloud trail. Go to aws>cloud trail. From the event history configuration enable the error code so that you can see the exact error. If you find the error code something like 'You are not authorized to perform this operation. Encoded authorization failure message'. Then open the event history details, pick up the encrypted error message and decrypt using aws cli
aws sts decode-authorization-message message. This will show you the complete role details, event, resources, action. Compare it with AWS IAM permissions and you can find out the missing permission or parameter that you need to pass while creating the job flow.
I tried to detach one of my IAM role from my instance (still running) and got a response of successful detachment.
Afterwards I tried to attach a new IAM role to the exactly same instance, however, this message occured: The association <AssociationId> is not the active association.
After using aws ec2 describe-iam-instance-profile-associations to check the IAM instance profile associations, I found that the state is disassociating. And I rechecked the associations the other day, it's still stuck at disassociating.
Then I tried aws ec2 associate-iam-instance-profile to associate my instance with a new role, but all I got is another instance association stuck at associating.
I also tried replace-iam-instance-profile-association and the same showed up: The association <AssociationId> is not the active association.
And rebooting instance did not work either.
Any solutions?
Thanks.
I've fixed this issue by launching a new instance base on the EBS snapshot of the problematic instance, which is the last thing I wanna do.
Anyway, this could be considered as a workaround. :(
It really sucks that you have to pay to create AWS technical support cases.
Found an easy solution for this!
Hope this helps some people finding this.
After getting stuck in the "disassociating" or "associating" state, use the aws CLI to find the associations that causes the problem (They will be stuck at the state ""disassociating" or "associating""):
aws ec2 describe-iam-instance-profile-associations
After finding them use:
aws ec2 disassociate-iam-instance-profile --association-id iip-assoc-xxxxxx
to remove them. Not quite intuitive but you can actually remove the ones in the state "disassociating" after that you can add a new role/instance-profile.
Error : Unable to detach, there are no existing instance profile associations.
While you are trying to add Role to EC2 instance
Debug and Verify:
run > aws iam list-instance-profiles
command output :
{
"InstanceProfiles": []
}
run > aws iam list-instance-profiles-for-role --role-name Your-Role-Name
command output :
{
"InstanceProfiles": []
}
Solution :
run > aws iam create-instance-profile --instance-profile-name profile-name-sameas-role-name
run > aws iam add-role-to-instance-profile --instance-profile-name profile-name-sameas-role-name --role-name role-name
Done !!
Go Back to EC2 dashboard and try to Add the IAM Role again. This time it should work.
I have created a few environments before so I know how the Amazon EBS works however lately I have been having the following issue while creating an environment:
The instance profile aws-elasticbeanstalk-ec2-role associated with the environment has no role. Please attach a role to the instance profile.
I follow the steps to create a new environment. When it gets to the Permissions page, I create a new role as there are no existing ones. Then I follow the rest of the steps and eventually it starts to launch. After a couple minutes, I get the error above. Any help towards this will be helpful.
To solve this issue, I created a new role from the IAM Manager console. I selected Amazon EC2 as my Service Role Type. I attached the AWSElasticBeanstalkFullAccess policy. Then when creating a new environment, I chose the new role I created.
Though I am bit late in answering this issue, posting here if someone faces this error now.
In case your user has all the required permission to create role and BS has already created the "The instance profile aws-elasticbeanstalk-ec2-role".
The reason of this error is due to roles only and when we try to launch EC2 from aws BS, it creates a role naming "aws-elasticbeanstalk-ec2-role" with required permissions.
But, if there is already a role with Trusted entities "AWS service: ec2" so BS will check permission in that already existing older role.
So go to Roles > Search ec2 related roles and select the role which is active.
And just add AWSElasticBeanstalkFullAccess permission to that ec2 role, and this issue will be resolved.