I'm using spring-cloud-starter-aws-jdbc to connect to an RDS instance. I initially went the traditional spring.datasource route, but I needed to make use of read-replicas and wanted to configure this without introducing any weird code.
The error I'm getting is:
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper#47acd13b: Failed to connect to service endpoint: , com.amazonaws.auth.profile.ProfileCredentialsProvider#6f8e9d06: profile file cannot be null]
| at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:136)
Initially I tried adding an AmazonRDS bean to my configuration and provided the credentials directly, but that wasn't good enough. I set a breakpoint inside getCredentials() and can see it being called twice: the first time there are 5 credential providers, one of which contains the AWS credentials I'm passing in via environment variables.
The second time, there are only two providers, neither of which contain my credentials, and so the app crashes. Has anyone ever used this library before and been successful? I can't figure out why it's fetching the credentials twice when I've already provided the RDS client and even tried providing the credentials with a bean.
Related
I have a Java Spring Boot Application deployed in GCP Cloud Run that tries to access Cloud KMS, However when the below code is executed it fails,
KeyManagementServiceClient client = KeyManagementServiceClient.create()
The error that occurs is,
java.io.IOException: The Application Default Credentials are not available.
They are available if running in Google Compute Engine.
Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS
must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials
for more information.
at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:134)
at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:125)
at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:97)
at com.google.api.gax.core.GoogleCredentialsProvider.getCredentials(GoogleCredentialsProvider.java:70)
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:168)
at com.google.cloud.kms.v1.stub.GrpcKeyManagementServiceStub.create(GrpcKeyManagementServiceStub.java:458)
at com.google.cloud.kms.v1.stub.KeyManagementServiceStubSettings.createStub(KeyManagementServiceStubSettings.java:668)
at com.google.cloud.kms.v1.KeyManagementServiceClient.<init>(KeyManagementServiceClient.java:191)
at com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:173)
at com.google.cloud.kms.v1.KeyManagementServiceClient.create(KeyManagementServiceClient.java:164)
at ..................................................................................
In my understanding, as long as the Service Account using which the Cloud Run Service runs has the required role(s) to access the other GCP service(s), this scenario should work fine.
Why is that the KeyManagementServiceClient creation process expecting an explicit GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to the Service Account Key file here?
Please advise.
After deploying the cloud run application using gcloud run deploy with proxy settings like below --set-env-vars "_JAVA_OPTIONS=-Dhttps.proxyHost=<<https proxy>> -Dhttp.proxyHost=<<http proxy>> -Dhttps.proxyPort=<<https port>> -Dhttp.proxyPort=<<http port>>, the exception is gone.
It appears like the GoogleCredentials.getApplicationDefault() call makes an authentication call to Google APIs using the Cloud Run's service account and for this call to work, it needs the Internet Proxy as our GCP environment resides inside our VPN.
We have few microservice's which get deployed on Ec2 instance properly and run fine.
But few of the pods inconsistently keep on getting "The security token included in the request is expired" error when connecting to DynamoDB and SNS. surprisingly the DB connections to Aurora don't seem to be a problem from the same microservice.
These pods face this issue for a few minutes and then again start working properly on their own.
Even if we restart the POD, it starts to work fine.
Things we have tried:
RetryPolicy retryPolicy = new RetryPolicy(PredefinedRetryPolicies.DEFAULT_RETRY_CONDITION,
PredefinedRetryPolicies.DEFAULT_BACKOFF_STRATEGY, MAX_RETRY_COUNT_AWS_TOKEN_EXPIRED, true);
ClientConfiguration clientConfiguration = new ClientConfiguration().withRetryPolicy(retryPolicy);
return AmazonSNSClientBuilder.standard().withClientConfiguration(clientConfiguration).build();
Also, we are trying to retry the above block of code from exception catch block, to instantiate snsclient, if snsClient.publish fails with token expiry error, assuming it will create new snsClient where the security token would be new, but that does not work either.
From AWS SDK documentation, the IAM roles in EC2 are used by instance metadata service to get new STS token using temporary credentials just before they expire. But at times this does not seem to be working.
Below are my queries:
What could be the issue?
How to debug if the call of instance meta data service to AWS is failing from my ec2 instance? cloud trail is not showing anything.
We sometimes face DNS resolver issue in our eco system, can this be the cause ? does ec2 instance meta data service also use DNS resolver to connect to AWS to get new STS token?
95% PODS work well, 5% PODS fail with this issue for few mins in a week.
Please suggest.
Does anyone have an idea, how I can access the email address of the service account, which is running my cloud run service, at runtime?
When deploying the service to gcloud, I use a specific service account for running the service.
During runtime I need the email/ID of this service account, in order to do blob signing using IAMCredentialsService.
Is there a possibility to get the service account ID somehow? The ComputeCredential object I have at hand doesn't provide this information. Right now I have to set an environment variable which contains the service account email address, which I can use at runtime within the service.
In your cloud run container, you need to reach this URL (a GET)
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email
With this header
Metadata-Flavor: Google
If you have difficulty getting the value, provide your language and I will see if I can provide a code sample for you.
See more in documentation https://cloud.google.com/run/docs/reference/container-contract#metadata-server
Container instance metadata server
Cloud Run container instances expose a metadata server that you can use to retrieve details about your container instance, such as the project ID, region, instance ID or service accounts. It can also be used to generate tokens for the runtime service account.
You can access this data from the metadata server using simple HTTP requests to the http://metadata.google.internal/ endpoint with the Metadata-Flavor: Google header: no client libraries are required. For more information, see Getting metadata.
I'm beginning to work with Secrets Manager and created my first secret in AWS. During the process, it gave me some sample code to work with. I put that in a small application and ran it. The code:
String region = "us-east-1";
string secret = "";
MemoryStream memoryStream = new MemoryStream();
IAmazonSecretsManager client = new AmazonSecretsManagerClient(
RegionEndpoint.GetBySystemName(region));
GetSecretValueRequest request = new GetSecretValueRequest();
request.SecretId = "MySecretNameExample";
GetSecretValueResponse response = null;
response = client.GetSecretValue(request);
The problem is that:
I was able to successfully retrieve the secret that I created and
nowhere am I creating a Credentials object with any valid AWS credential data
Where is this code getting the credential information from?
If you refer to the documenation for the API for this line of code:
IAmazonSecretsManager client = new AmazonSecretsManagerClient(
RegionEndpoint.GetBySystemName(region));
AmazonSecretsManagerClient
You will find the following description:
Constructs AmazonSecretsManagerClient with the credentials loaded from
the application's default configuration, and if unsuccessful from the
Instance Profile service on an EC2 instance.
This means that you are either running on an EC2 or ECS service (or related service such as Beanstalk, ...) with a role assigned to the instance or your have configured your credentials in the standard method in a credentials file. The AWS SDK is helping you locate credentials.
This document link will explain in more detail how AWS credentials are managed and selected.
Working with AWS Credentials
I have seen a lot of developers get the little details wrong with how credentials work and how they are used within the SDKs. Given that AWS credentials hold the keys to the AWS kingdom, managing and protecting them is vitally important.
The AWS SDK uses the a resolution strategy that looks in a number of locations until it finds credentials it can use. Typically the DefaultProviderChain class is responsible for performing the resolution. More information is here, but the gist is the lookup is performed in the following order (for Java, other languages are similar):
environment variables
Java system properties
credentials file (e.g. in the home directory)
instance profile credentials (only available when running in AWS)
When you run within AWS infrastructure, you can assign a profile or role to the resource that's running your code. Doing that makes credentials automatically available to your code. The idea is that they've made it easy to avoid putting credentials directly into your code.
I'm deploying my Django application on ec2 on AWS.
I did configuration setting up ~/.boto and finally succeed in 'python manage.py collectstatic'.
If there is an error, then error is caused! (I know because I solved it by setting up ~/.boto configuration file!).
But after configuration , when I query my image file at S3 mapped to my imageField model, it shows the error message below:
No handler was ready to authenticate. 1 handlers were checked.
['HmacAuthV1Handler'] Check your credentials
I think I made it authentication, but why is this message occuring?
Using a role is absolutely the correct way to handle authentication in EC2 to AWS. Putting long term credentials on the machine is a disgusting alternative. Assuming you're using a standard SDK, ( and boto absolutely is), the SDK will automatically use the role's temporary credentials to authenticate, so all you have to do is launch the instance with an "instance profile" specifying a role, and you get secure credentials delivery for free.
You'll have to replace your server to do so but_being able to recreate servers is fundamental to success in aws anyway. The sooner you start thinking that way, the better the cloud will work for you.
Once the role is attached to the instance, the policies defining the role's permission can be modified dynamically. So you don't need to get the permissions sorted out before creating the role.
At the high level, you specify a role at instance creation time. The EC2 console can facilitate the process of creating a role, allowing the EC2 service to access it, and specifying at instance creation time.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html provides detailed instructions.