My application hosted on EC2 instance needs to make constant connection to SQS and any loss in connection would lead to data loss. This can be successfully done using IAM user Access key and secret access key. But I want to use IAM roles for the same. IAM roles uses temporary credentials, and rotates credentials periodically. I am not sure if my application would lose connection to SQS at the time when temporary credentials are expired and rotated.
The temporary credentials associated with IAM roles have an expiration, but they are refreshed before the expiration occurs. This should not cause an issue unless your application does not properly check for the updated credentials when the expiration time approaches.
But a more fundamental factor in the question is that you may be unfamiliar with the underpinnings of the SQS API.
SQS does not rely on an authenticated "connection," so there is not a single connection that you can "lose." Technically, anyone can "connect" to SQS because the connection itself isn't what's authenticated. SQS authenticates each action independently -- every long poll request, every delete message action, etc., is authenticated by the service at the time it occurs. (If authentication fails, only the individual request fails.)
As long as your code uses appropriately fresh temporary credentials for each request it makes, using IAM roles will not impact your ability to continuously interact with the service.
any loss in connection would lead to data loss
You need to retry any errors that occur. Errors can happen for any number of reasons, but because SQS is interacted with by your code over HTTPS, each interaction with the service is not reliant on a continuous connection. You can (and should, but only for performance reasons) use HTTP keep-alives, but HTTPS doesn't depend on a single connection being continuously maintained.
Yes, you can roles easily. No need for temporary credentials.
You can use ec2 IAM role. You can create a role where you can attach permissions for SQS and the same role can be attached to ec2 where your server application is hosted.
Reference fo the same - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
Related
On expiry of IAM roles:
What is the logic for IAM roles to expire, when you need them for longer. I really cannot see it.
How do you when processing, get the an extension of the IAM role without some serious failure?
I did the IAM course but felt that was not well addressed.
E.g. a long running AWS EMR Spark data pipeline on a massive scale for cross account access?
If the Spark data pipeline has finished the Stage of reading from S3 and that role expires, may be that does not matter if you do not save subsequently to S3.
Instance roles such as those used in EMR are renewed automatically:
The application is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes before the expiration of the old credentials.
Look in the AWS SDK for com.amazonaws.auth.InstanceProfileCredentialsProvider; this is called by the clients to get the IAM Credentials. It spawns a thread com.amazonaws.auth.EC2CredentialsFetcher which does HTTP requests to the special 169.x.x.x http server which provides these details. Every spark worker creating an s3 client (or s3a, on ASF builds) will instantiate an InstanceProfileCredentialsProvider, after which everything will "just work"
IAM roles used to always expire after 1h; any job lasting 65+ minutes would have triggered a refresh.
try it and see.
lets say I have a on-premise application that needs to access various AWS services such as S3, Cloudwatch etc. What is the correct way to handle this authentication? I have read recommendations to create a new iam role and then distribute the AWS keys on the server that the application runs. But wouldn't this be very bad practice in case the keys gets stolen or exposed in some way? It would also be more work to rotate credentials for example. Is it possible to assign roles in some other ways or this is the correct way to do it? Isn't it better to assign roles or that isn't possible when not running the app in AWS?
Creat an IAM user with “Programmatic Access” only, which will provide you with a key and secret pair.
As a general rule, your application can use one set of credentials to get another, more privileged set of credentials. The app must be able to authenticate somehow so it needs some basic form of service account credentials to start with.
One way you can do this is to create an IAM user with minimal privileges. This IAM user is able to assume a specific IAM service role, but nothing else. That service role actually confers permissions to interact with S3, CloudWatch etc. Your application is configured with, or somehow securely retrieves, the credentials associated with the IAM user. Your application then uses these to call STS and assume the IAM service role, getting back short-lived STS credentials (access key, secret key, and session token). You should leverage the additional 'external ID' with the IAM role, as one more security factor.
Your application is also responsible for getting a new set of credentials before the existing set expires. You can do that in a number of ways, for example by using new STS credentials for every single request you make (so they never expire) or simply paying attention to the credentials expiration time and refreshing prior.
Also, read Temporary Credentials for Users in Untrusted Environments.
If your application is running on an Amazon EC2 instance and it is the only application on that instance, then:
Create an IAM Role
Assign the appropriate permissions to the Role
Assign the IAM Role to the EC2 instance
Any software running on the instance will automatically have access to credentials to access AWS. These credentials automatically rotate every 6 hours.
If you are not running on an EC2 instance:
Create an IAM User
Assign the appropriate permissions to the User
Generate credentials for the User (Access Key, Secret Key) and store them in a credentials file on the computer being used by the application
Any software running on the instance will automatically have access to these credentials to access AWS.
I am opening an AWS Service (say: AWS Rekognition) for my app's users.
The problem is: when one user (ex: user1) creates a resource (such as a collection), other users (ex: user2, user3) also see the resource that was created by user1.
I have tried to use Identity Pool, and acquired Token/Identity from my backend server for my users but things are not better (my users still see the resources of each other).
What should I do to let user1 receive user1's resource only?
I have been struggling with this problem for days, but can't seem to figure out.
Regards
There are two approaches to this architecture:
Option 1: Client/Server
In this architecture, client apps (eg on a mobile device or a web-based app) make calls to an API that is hosted by your back-end application. The back-end app then verifies the request and makes calls to AWS on behalf of the user.
The user's app never receives AWS credentials. This is very secure because the back-end app can authenticate all requests and apply business logic.
Option 2: Providing AWS credentials
In this architecture, the client apps receive temporary AWS credentials that enables them to directly call AWS services (which matches the architecture you describe).
The benefit is that the app can directly access AWS services such as Amazon S3. The downside is that they you need to very tightly limit the permissions they are given to ensure they only access the desired resources.
Some services make this easy by allowing Conditions on IAM Permissions that can limit the resources that can be accessed, such as by tag or other identifier.
However, based upon Actions, Resources, and Condition Keys for Amazon Rekognition - AWS Identity and Access Management, there is no such capability for Amazon Rekognition:
Rekognition has no service-specific context keys that can be used in the Condition element of policy statements.
I think you could limit the calls by providing a Resource string in the IAM Policy, which can limit their ability to make certain calls (eg DeleteFaces) so that it is only done against a specific collection.
However, please note that list calls such as ListCollections are either permitted fully or not at all. It is not possible to limit the list of collections returned. (This is the same as most AWS Services, such as listing EC2 instances.)
Thus, when using this method of providing credentials, you should be very careful about the permissions granted to the app.
We're developing an ASP.NET Core web application that will be running on-premises in a customer environment, and that application needs access to certain resources in our AWS account. Because this crosses an organizational boundary, we're creating an IAM Role in our AWS account and granting the customer access to assume that role from their account.
I believe the way this must work is that the ASP.NET Core application has "permanent" credentials to the customer's AWS account, and then obtains temporary credentials for the IAM Role in our account in order to do work. However, the application needs to do most of its work in an IHostedService/BackgroundService, which is a singleton running for the duration of the application lifetime, so if we use the standard dependency injection, we'll have a single long-lived instance of a given AWS service client. Does anyone know if the AWS SDK has any ability to continually re-obtain temporary credentials for an IAM Role? If not, I assume we'll have to stop obtaining the service clients via DI and keep periodically re-creating them ourselves to refresh the credentials.
I believe the way this must work is that the ASP.NET Core application has "permanent" credentials to the customer's AWS account, and then obtains temporary credentials for the IAM Role in our account in order to do work. However, the application needs to do most of its work in an
From the usage document, SDK's AssumeRole function just make a request to get temporary credential as simple strings. You must input these credential yourself to any object/service client that you want to use that specific credential returned from AssumeRoleRequest
Which mean there is no built-in auto-refresh that suits your needs
You may want to look into awsume and the AutoAwsume plugin for it.
This is something that can be used to auto-refresh temporary IAM role credentials on a machine.
A similar solution was used by developers in a former project of mine as a security best practice to avoid giving out long-lived credentials.
I am creating shortlived users on AWS on the fly and while debugging why these newly created logins tended to fail with an InvalidAccessKeyId realised that just adding a small sleep solved the problem.
xref How long should I wait after applying an AWS IAM policy before it is valid? re: time for consistency throughout AWS
My follow up question to the above: is there a way to synchronously create a consistent IAM policy? Or at least a way to know they are ready to use?
Amazon IAM is not designed for providing short-lived credentials. You should create IAM Users for long-lived requirements, such as logins for humans and logins for persistent applications.
An IAM User should not be used for application login purposes. For example, if you are creating an Instagram-like application, you should maintain your own database of users or utilize Amazon Cognito for user authentication.
So, how do you then grant users access to AWS resources? For example, if you have an Instagram-like application and you wish to grant application users the ability to upload/download their pictures in Amazon S3 but want to restrict access to a certain bucket and directory?...
The answer is to create temporary credentials using the AWS Security Token Service (STS). Credentials can be created with a given policy for a specific period of time. These credentials work immediately. For example, if an Instragram-like user logs into the app, the backend app could generate temporary credentials that allow the user to access a specific directory within a specific Amazon S3 bucket for a set period of time (eg 15 minutes). These credentials are then passed to the mobile app/web browser for direct access to AWS services.