I enjoy using the AWS SDK without having to specify where to find the credentials, it makes it easier to configure on multiple environment where different types of credentials are available.
The AWS SDK for Ruby searches for credentials [...]
Is there some way I can retrieve the code that does this to configure Faraday with AWS ? In order to configure Faraday I need something like
faraday.request(:aws_sigv4,
service: 'es',
credentials: credentials,
region: ENV['AWS_REGION'],
)
Now I would love that this credentials be picked "automatically" as in the aws sdk v3. How Can I do that ?
(ie where is the code in the AWS SDK v3 that does something like
credentials = Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
unless credentials.set?
credentials = Aws::InstanceProfileCredentials.new
end
...
The class Aws::CredentialProviderChain is the one in charge of resolving the credentials, but it is tagged as #api private so there's currently no guarantee it will still be there after updates (I've opened a discussion to make it public).
If you're okay to use it, you can resolve credentials like this. I'm going to test it in CI (ENV creds), development (Aws config creds), and staging/production environments (Instance profile creds).
Aws::CredentialProviderChain.new.resolve
You can use it in middlewares like this (for example configuring Elasticsearch/Faraday)
faraday.request(:aws_sigv4,
service: 'es',
credentials: Aws::CredentialProviderChain.new.resolve,
region: ENV['AWS_REGION'],
)
end
Related
When using a simple AWS credentials file like this on Ubuntu Linux with credentials file: ~/.aws/credentials
[leigh.m] # let's call this account ID: 1000 for simplicity
AWS_ACCESS_KEY_ID=some_access_key_id
AWS_SECRET_ACCESS_KEY=some_secret_access_key
AWS Toolkit in VSCode works perfectly and I can access AWS resources via VSCode.
However I need to assume roles cross-accounts (with a source 'root' profile controlling access to the other accounts via roles) which all have MFA set.
My ~/.aws/config file that facilitates this looks like:
[profile my-assumed-role-1] # let's say this has account ID: 2000
role_arn=arn:aws:iam::2000:role/AdminRole
source_profile=leigh.m
role_session_name=my-assumed-role-1-session-name
mfa_serial=arn:aws:iam::1000:mfa/leigh.m
duration_seconds=3600
This setup works perfectly well now and I can assume the role for example via CLI I can list buckets in account 2000 by passing that profile, like: aws s3api list-buckets --profile my-assumed-role-1
However, when I try to access that named profile via AWS Toolkit in VSCode I get the error message in AWS toolkit logs:
Shared Credentials Profile leigh.m is not valid. It will not be used by the toolkit.
Failed to connect with "profile:my-assumed-role-1": Profile leigh.m is not a valid Credential Profile: Profile leigh.m is not supported by the Toolkit.
It appears to be looking for a Profile 'leigh.m', so I have tried adding to ~/.aws/config this section:
[profile leigh.m]
mfa_serial=arn:aws:iam::1000:mfa/leigh.m
region=eu-west-1
output=json
I also tried to not use the config file and place everything in credentials file (as it appears to suggest here: https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/mfa-credentials.html)
But still the same issue. Anything I have missed?
(I did have a look to see if there were any questions already on S/O on correct setup of AWS credentials for AWS Toolkit and MFA (as my AWS credentials and config files work correctly for all other requirements), this was the only question from ~5 years ago: Visual Studio AWS toolkit with multifactor authentication? but did not answer and no questions/ answers on AWS Toolkit, MFA and assuming roles with names profiles)
This is a VSCode AWS Toolkit bug who are releasing a fix for this: https://github.com/aws/aws-toolkit-vscode/issues/2394
Summary: AWS Toolkit only works with lowercase keys, so this will not work:
AWS_ACCESS_KEY_ID=some_access_key_id
AWS_SECRET_ACCESS_KEY=some_secret_access_key
When the fix is released it will work with caps & lowercase, of course in the short term just change credentials file to:
aws_access_key_id=some_access_key_id
aws_secret_access_key=some_secret_access_key
It will work as expected
I'm trying to use the aws-sdk-go in my application. It's running on EC2 instance. Now in the Configuring Credentials of the doc,https://docs.aws.amazon.com/sdk-for-go/api/, it says it will look in
*Environment Credentials - Set of environment variables that are useful when sub processes are created for specific roles.
* Shared Credentials file (~/.aws/credentials) - This file stores your credentials based on a profile name and is useful for local development.
*EC2 Instance Role Credentials - Use EC2 Instance Role to assign credentials to application running on an EC2 instance. This removes the need to manage credential files in production.`
Wouldn't the best order be the reverse order? But my main question is do I need to ask the instance if it has a role and then use that to set up the credentials if it has a role? This is where I'm not sure of what I need to do and how.
I did try a simple test of creating a empty config with essentially only setting the region and running it on the instance with the role and it seems to have "worked" but in this case, I am not sure if I need to explicitly set the role or not.
awsSDK.Config{
Region: awsSDK.String(a.region),
MaxRetries: awsSDK.Int(maxRetries),
HTTPClient: http.DefaultClient,
}
I just want to confirm is this the proper way of doing it or not. My thinking is I need to do something like the following
role = use sdk call to get role on machine
set awsSDK.Config { Credentials: credentials form of role,
...
}
issue service command with returned client.
Any more docs/pointers would be great!
I have never used the go SDK, but the AWS SDKs I used automatically use the EC2 instance role if credentials are not found from any other source.
Here's an AWS blog post explaining the approach AWS SDKs follow when fetching credentials: https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/. In particular, see this:
If you use code like this, the SDKs look for the credentials in this
order:
In environment variables. (Not the .NET SDK, as noted earlier.)
In the central credentials file (~/.aws/credentials or
%USERPROFILE%.awscredentials).
In an existing default, SDK-specific
configuration file, if one exists. This would be the case if you had
been using the SDK before these changes were made.
For the .NET SDK, in the SDK Store, if it exists.
If the code is running on an EC2
instance, via an IAM role for Amazon EC2. In that case, the code gets
temporary security credentials from the instance metadata service; the
credentials have the permissions derived from the role that is
associated with the instance.
In my apps, when I need to connect to AWS resources, I tend to use an access key and secret key that have specific predefined IAM roles. Assuming I have those two, the code I use to create a session is:
awsCredentials := credentials.NewStaticCredentials(awsAccessKeyID, awsSecretAccessKey, "")
awsSession = session.Must(session.NewSession(&aws.Config{
Credentials: awsCredentials,
Region: aws.String(awsRegion),
}))
When I use this, the two keys are usually specified as either environment variables (if I deploy to a docker container).
A complete example: https://github.com/retgits/flogo-components/blob/master/activity/amazons3/activity.go
I have setup a new ElasticSearch cluster on AWS which is only allowing access to a specific IAM user.
However, I'm trying to connect to this from Ruby and looked at using the AWS SDK but that has no methods for actually making HTTP operations against your ES cluster, only accessing the configuration APIs.
As usual, this requires all the AWS request signing stuff that they require for API access, but I can't find anything that indicates how to do this stuff. I'm using Ruby.
Essentially, what I'm after is being able to make GET and PUT requests to this cluster using the IAM user creds. IP restriction isn't an option for me.
You can make signed, secure requests to Amazon Elasticsearch from Ruby. I did the following with an app on Heroku.
Ensure you have elasticsearch gem >= v1.0.15 as support for this was only implemented there Dec 4th, 2015.
You also need this gem:
gem 'faraday_middleware-aws-signers-v4'
Example from the elasticsearch-ruby/elasticsearch-transport
documentation:
You can use any standard Faraday middleware and plugins in the configuration block, for example sign the requests for the AWS Elasticsearch service:
With the following code:
require 'faraday_middleware/aws_signers_v4'
client = Elasticsearch::Client.new(url: ENV['AWS_ENDPOINT_URL']) do |f|
f.request :aws_signers_v4,
credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
service_name: 'es',
region: 'us-east-1'
f.adapter Faraday.default_adapter
end
This also works with the searchkick gem with Rails. Set Searchkick.client using the above example, in an initializer:
# config/initializers/elasticsearch.rb
require 'faraday_middleware/aws_signers_v4'
Searchkick.client = Elasticsearch::Client.new(url: ENV['AWS_ENDPOINT_URL']) do |f|
f.request :aws_signers_v4,
credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
service_name: 'es',
region: 'us-east-1'
f.adapter Faraday.default_adapter
end
I'm having a problem with my AWS credentials. I used the credentials file that I created on ~/.aws/credentials just as it is written on the AWS doc. However, apache just can't read it.
First, I was getting this error:
Error retrieving credentials from the instance profile metadata server. When you are not running inside of Amazon EC2, you must provide your AWS access key ID and secret access key in the "key" and "secret" options when creating a client or provide an instantiated Aws\Common\Credentials CredentialsInterface object.
Then I tried some solutions that I found on internet. For example, I tried to check my HOME variable. It was /home/ubuntu. I tried also to move my credentials file to the /var/www directory even if it is not my web server directory. Nothing worked. I was still getting the same error.
As a second solution, I saw that we could call directly the CredentialsProvider and indicate the directory on the client.
https://forums.aws.amazon.com/thread.jspa?messageID=583216򎘰
The error changed but I couldn't make it work:
Cannot read credentials from /.aws/credentials
I saw also that we could use the default provider of the CredentialsProvider instead of indicating a path.
http://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/credentials.html#using-credentials-from-environment-variables
I tried and I keep getting the same error:
Cannot read credentials from /.aws/credentials
Just in case you need this information, I'm using aws/aws-sdk-php (3.2.5). The service I'm trying to use is the AWS Elastic Transcoder. My EC2 instance is an Ubuntu 14.04. It runs a Symfony application deployed using Capifony.
Before I try on this production server, I tried it in a development server where it works perfectly only with the ~/.aws/credentials file. This development server is exactly a copy of the production server. However, it doesn't use Capifony for the deployment. It is just a normal git clone of the project. And it has only one EBS volume while the production server has one for the OS and one for the application.
Ah! And I also checked if the permissions/owners of the credentials file were the same on both servers and they are the same. I tried a 777 to see if it could change something but nothing.
Does anybody have an idea?
It sounds like you're doing it wrong. You do not need to deploy credentials to an EC2 instance in order to have that instance interact with other AWS services, and if fact should not ever deploy credentials to an EC2 instance.
Instead, when you create your instance, you associate an IAM role with it. That role has policies that control access to the other AWS services.
You can create an empty role, launch the instance, and then modify the role later. You can't assign a role after the instance has been launched.
You can now add roles to an instance after it has been assigned.
It is still considered a best practice to not deploy actual credentials to an EC2 instance.
If this can help someone, I managed to make my .ini file work, doing this way:
$profile = 'default';
$path = '/mnt/app/www/.aws/credentials/default.ini';
$provider = CredentialProvider::ini($profile, $path);
$provider = CredentialProvider::memoize($provider);
$client = ElasticTranscoderClient::factory(array(
'region' => 'eu-west-1',
'version' => '2012-09-25',
'credentials' => $provider
));
The CredentialProvider is explained on this doc:
http://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/credentials.html#ini-provider
I still don't understand why my application can't read the file on the home directory (~/.aws/credentials/default.ini) on one server but in the other it does.
If someone knows something about it, please let me know.
The SDK reads from a file located at ~/.aws/credentials, but it looks like you're saving a file at ~/.aws/credentials/default.ini. If you move the file, the error you were experiencing should be cleared up.
2 Ways of solving this problem to me Node.js
Its going to get my credentials from /home/{USER}/.aws/credentials usin' the default profile
const aws = require('aws-sdk');
aws.config.credentials = aws.SharedIniFileCredentials({profile: 'default'})
...
The hardcoded way
var lambda = new aws.Lambda({
region: 'us-east-1',
accessKeyId: <KEY>
secretAccessKey: <KEY>
});
For some reason Packer fails to authenticate to AWS, using plain aws client works though, and my environment variables are correctly set:
AWS_ROLE_SESSION_NAME=...
AWS_SESSION_TOKEN=...
AWS_SECRET_ACCESS_KEY=...
AWS_ROLE=...
AWS_ACCESS_KEY_ID=...
AWS_CLI=...
AWS_ACCOUNT=...
AWS_SECURITY_TOKEN=...
I am using authentication using aws saml, and Packer gives me the following:
Error querying AMI: AWS was not able to validate the provided access credentials (AuthFailure)
The problem lies within the way Packer authenticates with AWS.
Packer is written in go and uses goamz for authentication. When creating a config using aws saml, a couple of files are generated in ~/.aws : config and credentials.
Turns out this credentials file takes precedence over the environment variables, so if these credentials are incorrect and you rely on your environment variables, you will get the same error.
Since aws-saml needs aws_access_key_id and aws_secret_access_key to be defined, deleting the credentials file would not suffice in this case.
We had to copy these values into ~/.aws/config and delete the credentials file, then Packer was happy to use our environment variables.
A ticket has been raised in github for goamz so AWS CLI and Packer can have the same authenticating behavior, feel free to vote it up if you have the issue too : https://github.com/mitchellh/goamz/issues/171