The official AWS documentation states that instance profile credentials "are temporary and would eventually expire", and I was wondering how often they expire.
I'am asking because I have applications using an InstanceProfileCredentialsProvider as credential provider, which by default does not refresh credentials, running for days without problems.
We have noticed from logging that the temporary credentials issued against an attached role last approximately 6 hours.
Does anyone know the mechanism of how they are refreshed, supposedly 15 minutes before they expire? Is the SSM service monitoring the expiration and asking for new credentials?
We are currently chasing down what appears to be an issue with the credentials not being refreshed after the EC2 instance has no activity on the overnight. Trying to determine whether app pool idle timeout or recycle interval is playing a hand.
Related
We have few microservice's which get deployed on Ec2 instance properly and run fine.
But few of the pods inconsistently keep on getting "The security token included in the request is expired" error when connecting to DynamoDB and SNS. surprisingly the DB connections to Aurora don't seem to be a problem from the same microservice.
These pods face this issue for a few minutes and then again start working properly on their own.
Even if we restart the POD, it starts to work fine.
Things we have tried:
RetryPolicy retryPolicy = new RetryPolicy(PredefinedRetryPolicies.DEFAULT_RETRY_CONDITION,
PredefinedRetryPolicies.DEFAULT_BACKOFF_STRATEGY, MAX_RETRY_COUNT_AWS_TOKEN_EXPIRED, true);
ClientConfiguration clientConfiguration = new ClientConfiguration().withRetryPolicy(retryPolicy);
return AmazonSNSClientBuilder.standard().withClientConfiguration(clientConfiguration).build();
Also, we are trying to retry the above block of code from exception catch block, to instantiate snsclient, if snsClient.publish fails with token expiry error, assuming it will create new snsClient where the security token would be new, but that does not work either.
From AWS SDK documentation, the IAM roles in EC2 are used by instance metadata service to get new STS token using temporary credentials just before they expire. But at times this does not seem to be working.
Below are my queries:
What could be the issue?
How to debug if the call of instance meta data service to AWS is failing from my ec2 instance? cloud trail is not showing anything.
We sometimes face DNS resolver issue in our eco system, can this be the cause ? does ec2 instance meta data service also use DNS resolver to connect to AWS to get new STS token?
95% PODS work well, 5% PODS fail with this issue for few mins in a week.
Please suggest.
On expiry of IAM roles:
What is the logic for IAM roles to expire, when you need them for longer. I really cannot see it.
How do you when processing, get the an extension of the IAM role without some serious failure?
I did the IAM course but felt that was not well addressed.
E.g. a long running AWS EMR Spark data pipeline on a massive scale for cross account access?
If the Spark data pipeline has finished the Stage of reading from S3 and that role expires, may be that does not matter if you do not save subsequently to S3.
Instance roles such as those used in EMR are renewed automatically:
The application is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials.
Look in the AWS SDK for com.amazonaws.auth.InstanceProfileCredentialsProvider; this is called by the clients to get the IAM Credentials. It spawns a thread com.amazonaws.auth.EC2CredentialsFetcher which does HTTP requests to the special 169.x.x.x http server which provides these details. Every spark worker creating an s3 client (or s3a, on ASF builds) will instantiate an InstanceProfileCredentialsProvider, after which everything will "just work"
IAM roles used to always expire after 1h; any job lasting 65+ minutes would have triggered a refresh.
try it and see.
This happens when accessing EC2 console for any region.
Global services like S3, IAM and account related pages can be accessed without problems.
No AWS Organization issue, no policy issue.
Just go to your email and click the link AWS management console and open an AWS account.
So, now you can access your EC2.
Check whether you have skipped one or more steps during registration.
You may have received an email stating you need to complete those steps.
File a support case to receive the email again if you missed it.
It takes a while to AWS to verify your card details after you register a new account with AWS. It would be great if AWS would give a more sensible message.
Wait for account activation :
After you choose a Support plan, a confirmation page indicates that your account is being activated. Accounts are usually activated within a few minutes, but the process might take up to 24 hours.
AWS create and activate account
My application hosted on EC2 instance needs to make constant connection to SQS and any loss in connection would lead to data loss. This can be successfully done using IAM user Access key and secret access key. But I want to use IAM roles for the same. IAM roles uses temporary credentials, and rotates credentials periodically. I am not sure if my application would lose connection to SQS at the time when temporary credentials are expired and rotated.
The temporary credentials associated with IAM roles have an expiration, but they are refreshed before the expiration occurs. This should not cause an issue unless your application does not properly check for the updated credentials when the expiration time approaches.
But a more fundamental factor in the question is that you may be unfamiliar with the underpinnings of the SQS API.
SQS does not rely on an authenticated "connection," so there is not a single connection that you can "lose." Technically, anyone can "connect" to SQS because the connection itself isn't what's authenticated. SQS authenticates each action independently -- every long poll request, every delete message action, etc., is authenticated by the service at the time it occurs. (If authentication fails, only the individual request fails.)
As long as your code uses appropriately fresh temporary credentials for each request it makes, using IAM roles will not impact your ability to continuously interact with the service.
any loss in connection would lead to data loss
You need to retry any errors that occur. Errors can happen for any number of reasons, but because SQS is interacted with by your code over HTTPS, each interaction with the service is not reliant on a continuous connection. You can (and should, but only for performance reasons) use HTTP keep-alives, but HTTPS doesn't depend on a single connection being continuously maintained.
Yes, you can roles easily. No need for temporary credentials.
You can use ec2 IAM role. You can create a role where you can attach permissions for SQS and the same role can be attached to ec2 where your server application is hosted.
Reference fo the same - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
I launched EC2 instance with assigned IAM role.
I am able to retrieve the key.
As we can see, it has expiration time.
My application reads it once per 8-10 hours, so if key has been rotated app fails to write to S3.
How Amazon does rotate this temporary credentials?
Is it possible to configure or disable expiration time?
Purpose - we don't want to store credentials in source code.
Thanks.
No you cannot change this.
New credentials are made available no later than 5 minutes prior to the expiration of the previous ones - if you are close to the expiration time then you should start checking for new credentials and start using them.
The answer is no.
Furthermore, when launching an EC2 instance in an IAM role there is no reason, and you should not, attempt to use those generated keys directly. The purpose of IAM role is allow your application to use AWS services for which the role is authorized without you having to deploy API keys to your application. That is the beauty and benefit of using IAM roles for EC2
Please refer to http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html