On expiry of IAM roles:
What is the logic for IAM roles to expire, when you need them for longer. I really cannot see it.
How do you when processing, get the an extension of the IAM role without some serious failure?
I did the IAM course but felt that was not well addressed.
E.g. a long running AWS EMR Spark data pipeline on a massive scale for cross account access?
If the Spark data pipeline has finished the Stage of reading from S3 and that role expires, may be that does not matter if you do not save subsequently to S3.
Instance roles such as those used in EMR are renewed automatically:
The application is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials.
Look in the AWS SDK for com.amazonaws.auth.InstanceProfileCredentialsProvider; this is called by the clients to get the IAM Credentials. It spawns a thread com.amazonaws.auth.EC2CredentialsFetcher which does HTTP requests to the special 169.x.x.x http server which provides these details. Every spark worker creating an s3 client (or s3a, on ASF builds) will instantiate an InstanceProfileCredentialsProvider, after which everything will "just work"
IAM roles used to always expire after 1h; any job lasting 65+ minutes would have triggered a refresh.
try it and see.
Related
I would like to run a batch job on-prem and access AWS resources in our account.
I think the recommendation is to create an IAM user, which will be a machine user. Since I don't have a way to assign a role to the on-prem machine, or federate with AWS identity, I'll create an access key and install it on the on-prem machine. What's the best way to link my machine user to a policy?
I can create an IAM policy which allows the required actions (reading AWS SSM Parameters).
But, how should I link the machine user to the policy? I'm setting up these users/policies with Pulumi. Some options I'm aware of:
I can create a role, but then I think the machine user would have to assume the role. (My understanding is that roles do not have immediate "membership", it's just that users have the ability to assume roles. Or, AWS infrastructure can be set up with a role, like an EC2 or an EKS cluster can act as a role. In the future I do plan to move this job's execution to AWS infrastructure, but for now that's not an option.) Is assuming a role easy, for example a aws sts CLI call that I could put in my batch job's startup script before calling the main binary?
Or I could just attach the policy directly to the machine user. Generally that's not recommended from what I've read: you should have a layer between users and policies so when users change what they're doing you have indirection. But in this case maybe that's fine.
Or finally I could create a user group, attach the policy to the group, and add the machine user as a member of the group. Is that layer of indirection useful / an appropriate use of groups, especially if I'm already managing these policies with IaC? Most documentation recommends roles for the user-to-policy indirection, so I'm hesitant to use groups that way. However, that seems to be the expected approach for human users (glad for feedback on that too).
"Is it better to use AWS IAM User Group, or IAM Role for users to assume?" says a group would help manage permissions for multiple users (but so does Pulumi and I only have 1 or 2 machine users); and a role would help separate access rights from long-lived credentials but it seems like rotating the machine user's access key would have that benefit too without the extra assume-role step.
This page shows how to send an email using SES. The example works by reading the credentials from ~/.aws/credentials, which are the root (yet "shared"??) credentials.
The documentation advises in various places against using the root credentials.
Acquiring temporary credentials
using roles is mentioned as an option, yet assume_role() is not defined for SES client objects.
How do I send an email through SES with temporary SES-specific credentials?
Update
The context for my question is an application running on an EC2 instance.
There are a few pieces to this.
First you need an IAM policy. You can use one of the built-in policies, such as AmazonSESFullAccess or you can create your own. The holder of a particular policy will be able to access the resources and actions defined in the policy. You can create this policy manually, or work through the AWS console and it will walk you through it. IAM --> Policies --> Create Policy
Secondly, you will need a role. Also, easily done in the console. IAM --> Roles --> Create role. Trusted entity is AWS service. Highlight EC2. In the next screen, select the policy you want to associate with this role. This is the policy you created above. If your EC2 already has a role, then you can add the IAM policy to this role. Assigning an IAM policy to a role, is what they refer to as a trust policy.
Now any code that runs on your EC2 instance will be able to send messages to your SES service. The EC2 assumes the role assigned to it. And the SES policy is defined for that role. This will allow EC2 to get temporary credentials (behind the scenes).
The back story is as follows. Any API call to an AWS service needs to have a key and secret. When you make API calls from your local computer, you may use your personal key and secret (or even root ones). When you need to make API calls from another service, you do not have that key and secret. It would not be secure or practical to store the credentials on an EC2. Or even worse, in an S3 bucket. That is why AWS came up with the Role concept. Roles can request temporary credentials from an internal service called Simple Token Service (STS). A role is attached to an EC2 instance for example. And if the right policy is attached to that role, the EC2 instance can request to get temporary credentials to make an API call to another service. All of this happens behind the scenes.
Two options...
You could create IAM User credentials with the appropriate permissions and put them in the ~./aws/credentials file. Then your application will find them and use them to connect with Amazon SES.
Or, your application could use a set of IAM User credentials to call assume_role() (which is an IAM command). This will return a set of temporary credentials that could be used with Amazon SES. However, if you are going to provide a set of credentials that will be used to call assume_role(), then you may as well just use those credentials directly with Amazon SES.
An IAM User can be used for people OR applications.
My application hosted on EC2 instance needs to make constant connection to SQS and any loss in connection would lead to data loss. This can be successfully done using IAM user Access key and secret access key. But I want to use IAM roles for the same. IAM roles uses temporary credentials, and rotates credentials periodically. I am not sure if my application would lose connection to SQS at the time when temporary credentials are expired and rotated.
The temporary credentials associated with IAM roles have an expiration, but they are refreshed before the expiration occurs. This should not cause an issue unless your application does not properly check for the updated credentials when the expiration time approaches.
But a more fundamental factor in the question is that you may be unfamiliar with the underpinnings of the SQS API.
SQS does not rely on an authenticated "connection," so there is not a single connection that you can "lose." Technically, anyone can "connect" to SQS because the connection itself isn't what's authenticated. SQS authenticates each action independently -- every long poll request, every delete message action, etc., is authenticated by the service at the time it occurs. (If authentication fails, only the individual request fails.)
As long as your code uses appropriately fresh temporary credentials for each request it makes, using IAM roles will not impact your ability to continuously interact with the service.
any loss in connection would lead to data loss
You need to retry any errors that occur. Errors can happen for any number of reasons, but because SQS is interacted with by your code over HTTPS, each interaction with the service is not reliant on a continuous connection. You can (and should, but only for performance reasons) use HTTP keep-alives, but HTTPS doesn't depend on a single connection being continuously maintained.
Yes, you can roles easily. No need for temporary credentials.
You can use ec2 IAM role. You can create a role where you can attach permissions for SQS and the same role can be attached to ec2 where your server application is hosted.
Reference fo the same - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
I am creating shortlived users on AWS on the fly and while debugging why these newly created logins tended to fail with an InvalidAccessKeyId realised that just adding a small sleep solved the problem.
xref How long should I wait after applying an AWS IAM policy before it is valid? re: time for consistency throughout AWS
My follow up question to the above: is there a way to synchronously create a consistent IAM policy? Or at least a way to know they are ready to use?
Amazon IAM is not designed for providing short-lived credentials. You should create IAM Users for long-lived requirements, such as logins for humans and logins for persistent applications.
An IAM User should not be used for application login purposes. For example, if you are creating an Instagram-like application, you should maintain your own database of users or utilize Amazon Cognito for user authentication.
So, how do you then grant users access to AWS resources? For example, if you have an Instagram-like application and you wish to grant application users the ability to upload/download their pictures in Amazon S3 but want to restrict access to a certain bucket and directory?...
The answer is to create temporary credentials using the AWS Security Token Service (STS). Credentials can be created with a given policy for a specific period of time. These credentials work immediately. For example, if an Instragram-like user logs into the app, the backend app could generate temporary credentials that allow the user to access a specific directory within a specific Amazon S3 bucket for a set period of time (eg 15 minutes). These credentials are then passed to the mobile app/web browser for direct access to AWS services.
I launched EC2 instance with assigned IAM role.
I am able to retrieve the key.
As we can see, it has expiration time.
My application reads it once per 8-10 hours, so if key has been rotated app fails to write to S3.
How Amazon does rotate this temporary credentials?
Is it possible to configure or disable expiration time?
Purpose - we don't want to store credentials in source code.
Thanks.
No you cannot change this.
New credentials are made available no later than 5 minutes prior to the expiration of the previous ones - if you are close to the expiration time then you should start checking for new credentials and start using them.
The answer is no.
Furthermore, when launching an EC2 instance in an IAM role there is no reason, and you should not, attempt to use those generated keys directly. The purpose of IAM role is allow your application to use AWS services for which the role is authorized without you having to deploy API keys to your application. That is the beauty and benefit of using IAM roles for EC2
Please refer to http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html