When trying to connect to an AWS service via Boto3, I occasionally get the following error:
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV3Handler'] Check your credentials
This is running on an EC2 instance with an IAM Role configured. This error happens rarely.
IAM roles provide credentials via the AWS metadata service. Boto3 will connect to this service to get credentials, but this connection can time out. By default, Boto3 will not retry connections to the metadata service, but this can be changed by setting the environment variable AWS_METADATA_SERVICE_NUM_ATTEMPTS to a number higher than 1.
See the docs:
AWS_METADATA_SERVICE_NUM_ATTEMPTS
When attempting to retrieve credentials on an EC2 instance that has been configured with an IAM role, boto3 will only make one attempt to retrieve credentials from the instance metadata service before giving up. If you know your code will be running on an EC2 instance, you can increase this value to make boto3 retry multiple times before giving up.
Related
We have few microservice's which get deployed on Ec2 instance properly and run fine.
But few of the pods inconsistently keep on getting "The security token included in the request is expired" error when connecting to DynamoDB and SNS. surprisingly the DB connections to Aurora don't seem to be a problem from the same microservice.
These pods face this issue for a few minutes and then again start working properly on their own.
Even if we restart the POD, it starts to work fine.
Things we have tried:
RetryPolicy retryPolicy = new RetryPolicy(PredefinedRetryPolicies.DEFAULT_RETRY_CONDITION,
PredefinedRetryPolicies.DEFAULT_BACKOFF_STRATEGY, MAX_RETRY_COUNT_AWS_TOKEN_EXPIRED, true);
ClientConfiguration clientConfiguration = new ClientConfiguration().withRetryPolicy(retryPolicy);
return AmazonSNSClientBuilder.standard().withClientConfiguration(clientConfiguration).build();
Also, we are trying to retry the above block of code from exception catch block, to instantiate snsclient, if snsClient.publish fails with token expiry error, assuming it will create new snsClient where the security token would be new, but that does not work either.
From AWS SDK documentation, the IAM roles in EC2 are used by instance metadata service to get new STS token using temporary credentials just before they expire. But at times this does not seem to be working.
Below are my queries:
What could be the issue?
How to debug if the call of instance meta data service to AWS is failing from my ec2 instance? cloud trail is not showing anything.
We sometimes face DNS resolver issue in our eco system, can this be the cause ? does ec2 instance meta data service also use DNS resolver to connect to AWS to get new STS token?
95% PODS work well, 5% PODS fail with this issue for few mins in a week.
Please suggest.
I am new to AWS infra and currently doing some POC/Feasibility for new work.
So I have created a S3 bucket in Ireland server, train and publish Sagemaker endpoint in Ireland server and its giving result in Jupyter notebook there. Now I want to use that endpoint in my browser javascript library to show some graphics. When I try to test my endpoint in Postman then its giving region specific error
{
"message": "Credential should be scoped to a valid region, not 'us-east-1'.
Credential should be scoped to correct service: 'sagemaker'. "
}
My AWS account is not yet enterprise managed so I am using as 'root user', Whenever I go to my profile>Security_Credential page and generate any security credential then it always create for 'us-east-1' region, As Sagemaker is region specific service, I am not able to find the way to create region specific security key for root user, can someone please help
You should create an IAM role first that defines what should be permitted (mainly calling the invoke-endpoint API call for SageMaker runtime). Then you should create an IAM user, add the above role to that user, and then generate credentials that you can use in your Postman to call the service. Here you can find some details about the IAM role for SageMaker that you can use in this process: https://docs.aws.amazon.com/sagemaker/latest/dg/using-identity-based-policies.html
A popular option to achieve external access to a SageMaker endpoint, is to create an API Gateway that calls a Lambda Function that is then calling the invoke-endpoint API. This chain is giving you various options such as different authentication options for the users and API keys as part of API-GW, processing the user input and inference output using API-GW and Lambda code, and giving the permission to call the SageMaker endpoint to the Lambda function. This chain removes the need for the credentials creation, update and distribution.
I set up IAM authentication on an RDS instance, and I'm able to use IAM to get database passwords that work for 15-minutes. This is fine to access the database for backups, but this database backs an web application so currently after 15 minutes the password used by the app to connect to the DB becomes invalid and the app crashes as it can no longer access the DB.
However, in the RDS IAM docs there's this line:
For applications running on Amazon EC2, you can use EC2 instance profile credentials to access the database, so you don't need to use database passwords on your EC2 instance.
This implies that on EC2 there's no need to use the IAM temporary DB password, which would mean that my app should be able to connect to the DB as long as it's running on EC2 and I set up the role permissions (which I think I did correctly). However, I can't get my app running on EC2 to be able to connect to the RDS DB except by using the 15-minute temporary password. If I try connecting with a normal MySQL connection with no password I get permission denied. Is there something special that needs to be done to connect to RDS using the EC2 instance profile, or is it not possible without using 15-minute temporary passwords?
According to the documentation you linked (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html), you need to perform the following steps (See under "Authenticating to a DB Instance or DB Cluster Using IAM Database Authentication"):
Use the AWS SDK for Java or AWS CLI to get an authentication token you can use to identify the IAM user or role. To learn how to get an authentication token, see Getting an Authentication Token.
Connect to the database using an SSL connection, specifying the IAM user or role as the database user account and the authentication token as the password. For more information, see Connecting to a DB Instance or DB Cluster Using IAM Database Authentication.
That means for every connection you intend to open, you need to get a valid Token using the AWS SDK. This is where using the correct instance profile with the RDS permission is needed. See also the code examples further down the AWS documentation page.
I think however this requires quite a bit of effort on your side, to always get a valid token before opening a connection. It makes using an off-the-shelf connection pool difficult. Probably once open, the connection will remain open even after the token expires, but you still need to handle the case where more connections need to be opened at a later time.
I would stick with a normal user/password access for the application, using IAM for this case seems to be too much effort.
For applications running on Amazon EC2, you can use EC2 instance profile credentials to access the database, so you don't need to use database passwords on your EC2 instance.
You're misinterpreting what this means. It means you don't have to use static passwords or store them on the instance.
The idea is that you generate a new authentication token each time you establish a connection to the database. The token is generated on your instance, using the instance role credentials. It can only be used to authenticate for 15 minutes, but once connected, you don't lose your database connection after 15 minutes. You remain connected.
If your application doesn't reuse database connections, then you likely have a design flaw there.
I'm deploying my Django application on ec2 on AWS.
I did configuration setting up ~/.boto and finally succeed in 'python manage.py collectstatic'.
If there is an error, then error is caused! (I know because I solved it by setting up ~/.boto configuration file!).
But after configuration , when I query my image file at S3 mapped to my imageField model, it shows the error message below:
No handler was ready to authenticate. 1 handlers were checked.
['HmacAuthV1Handler'] Check your credentials
I think I made it authentication, but why is this message occuring?
Using a role is absolutely the correct way to handle authentication in EC2 to AWS. Putting long term credentials on the machine is a disgusting alternative. Assuming you're using a standard SDK, ( and boto absolutely is), the SDK will automatically use the role's temporary credentials to authenticate, so all you have to do is launch the instance with an "instance profile" specifying a role, and you get secure credentials delivery for free.
You'll have to replace your server to do so but_being able to recreate servers is fundamental to success in aws anyway. The sooner you start thinking that way, the better the cloud will work for you.
Once the role is attached to the instance, the policies defining the role's permission can be modified dynamically. So you don't need to get the permissions sorted out before creating the role.
At the high level, you specify a role at instance creation time. The EC2 console can facilitate the process of creating a role, allowing the EC2 service to access it, and specifying at instance creation time.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html provides detailed instructions.
I'm setting up a Kubernetes cluster on AWS and as part of the configuration for say the API Server, I provide the --cloud-provider=aws setting.
Once it starts up, however, I see in the logs that it complains about not having AWS credentials:
NoCredentialProviders: no valid providers in chain
After some searching, it seems that this issue was resolved for most people by using the "kube-up" script. However, for those who are not using the script to set up their cluster, how do we provide Kubernetes with AWS credentials?
It sounds like you don't have the appropriate IAM instance profile set on your master VM. The kube-up script for AWS creates a role and associated policy that is attached to the master VM when it is created. Having the IAM policy attached should give you the credentials necessary to make API calls into AWS.