I am using aws-go sdk
using static creds I am able to connect using
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(awsAccesstKey, awsSecretKey, "")),
is there anyway using nodeIAM role, by which we can connect to AWS?
The documentation explains how to configure the SDK, including credentials:
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatalf("failed to load configuration, %v", err)
}
From the link above:
When you initialize an aws.Config instance using
config.LoadDefaultConfig, the SDK uses its default credential chain to
find AWS credentials. This default credential chain looks for
credentials in the following order:
Environment variables.
Static Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
Web Identity Token (AWS_WEB_IDENTITY_TOKEN_FILE)
Shared configuration files.
SDK defaults to credentials file under .aws folder that is placed in the home folder on your computer.
SDK defaults to config file under .aws folder that is placed in the home folder on your computer.
If your application uses an ECS task definition or RunTask API operation, IAM role for tasks.
If your application is running on an Amazon EC2 instance, IAM role for Amazon EC2.
The SDK detects and uses the built-in providers automatically, without
requiring manual configurations. For example, if you use IAM roles for
Amazon EC2 instances, your applications automatically use the
instance’s credentials. You don’t need to manually configure
credentials in your application.
Related
I have moved my aws credentials from ~/.aws/credentials to resources folder of maven project . the folder structure looks like this
resources/aws/
->config
->credentials
I am using aws java sdk version 2+ . How can i read the values from resources folder to get region, access keys , create bucket and perform operations.
You should not place credentials files in resources directory. AWS Java SDK supports credential files in ~/.aws out-of-the box:
The following list shows the supported credential retrieval techniques:
Java system properties–aws.accessKeyId and aws.secretAccessKey. The AWS SDK for Java uses the SystemPropertyCredentialsProvider to load these credentials.
Environment variables–AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS SDK for Java uses the EnvironmentVariableCredentialsProvider class to load these credentials.
The default credential profiles file– The specific location of this file can vary per platform, but is typically located at ~/.aws/credentials. This file is shared by many of the AWS SDKs and by the AWS CLI. The AWS SDK for Java uses the ProfileCredentialsProvider to load these credentials.
You can create a credentials file by using the aws configure command provided by the AWS CLI. You can also create it by editing the file with a text editor. For information about the credentials file format, see AWS Credentials File Format.
Amazon ECS container credentials– This is loaded from Amazon ECS if the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set. The AWS SDK for Java uses the ContainerCredentialsProvider to load these credentials.
Instance profile credentials– This is used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service. The AWS SDK for Java uses the InstanceProfileCredentialsProvider to load these credentials.
So, either use ProfileCredentialsProvider or pass the credentials via system properties or environment variables and use SystemPropertyCredentialsProvider / EnvironmentVariableCredentialsProvider.
AWS Java SDK v2 does not support getting credentials from the resource folder (classpath) directly.
As an alternative, you can put AWS credentials in a properties file in the resource folder:
[[project]/src/test/resources/aws-credentials.properties:
aws_access_key_id = xxx
aws_secret_access_key = xxx
Spring config:
<util:properties id="awsCredentialFile"
location="classpath:aws-credentials.properties"/>
and your code:
#Resource(name = "awsCredentialFile")
public void setProperties(Properties properties) {
this.accessKey = properties.getProperty("aws_access_key_id");
this.secretKey = properties.getProperty("aws_secret_access_key");
}
StaticCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));
S3Client s3 = S3Client.builder()
.credentialsProvider(credentialsProvider)
.build();
I have an AWS Batch Job which is a .NET CORE app running as a container which downloads from an SFTP server a CSV parses it and inserts data into AWS RDS.
When the CSV is corrupt the job is failing and is supposed to send a SNS notification, instead I see the following error in CloudWatch logs.
"Message": "User: arn:aws:sts::654001826221:assumed-role/fileimportworker-batch/5f77c736e4e64c2d82df278800ec4f25 is not authorized to perform: SNS:Publish on resource: arn:aws:sns:eu-west-1:accountIdHere:Test-SNS-Batch",
My IAM role attached to the batch Job role has SNS:Published allowed, S3 allowed, also provides read access to 2 secrets in Secret Manager. S3 and SecretManager access work, the task is able to download the file from SFTP and put it to S3 and also to read the RDS password from secret manager.
AWS Batch Job may use credentials from a container instead of your environment variables. You have to look at credential precedence.
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-precedence
The AWS CLI uses credentials and configuration settings located in
multiple places, such as the system or user environment variables,
local AWS configuration files, or explicitly declared on the command
line as a parameter. Certain locations take precedence over others.
The AWS CLI credentials and configuration settings take precedence in
the following order:
Command line options – Overrides settings in any other location. You
can specify --region, --output, and --profile as parameters on the
command line.
Environment variables – You can store values in your system's
environment variables.
CLI credentials file – The credentials and config file are updated
when you run the command aws configure. The credentials file is
located at ~/.aws/credentials on Linux or macOS, or at
C:\Users\USERNAME.aws\credentials on Windows. This file can contain
the credential details for the default profile and any named profiles.
CLI configuration file – The credentials and config file are updated
when you run the command aws configure. The config file is located at
~/.aws/config on Linux or macOS, or at C:\Users\USERNAME.aws\config
on Windows. This file contains the configuration settings for the
default profile and any named profiles.
Container credentials – You can associate an IAM role with each of
your Amazon Elastic Container Service (Amazon ECS) task definitions.
Temporary credentials for that role are then available to that task's
containers. For more information, see IAM Roles for Tasks in the
Amazon Elastic Container Service Developer Guide.
Instance profile credentials – You can associate an IAM role with each
of your Amazon Elastic Compute Cloud (Amazon EC2) instances. Temporary
credentials for that role are then available to code running in the
instance. The credentials are delivered through the Amazon EC2
metadata service. For more information, see IAM Roles for Amazon EC2
in the Amazon EC2 User Guide for Linux Instances and Using Instance
Profiles in the IAM User Guide.
P.S. To intergate AWS Batch with SNS without coding, you can use the Eventbridge rule to listen to event patterns from AWS Batch. You just select the target of the rule to publish the message on the SNS topic you want.
https://docs.aws.amazon.com/batch/latest/userguide/batch_sns_tutorial.html
I want a run a python boto3 scrip from my local machine using Pycharm. I am having config file setup in my machine that contains different roles based on the accounts. Say for example I want to execute the scrip using dev-power user that is included in the config file. But I am getting the below error:
"botocore.exceptions.NoCredentialsError: Unable to locate credentials"
Could you please let me know the process by which I can do this from local IDE?
You cannot use IAM Role to access AWS services from your development environment. An IAM Role can only be attached to an IAM User or an AWS service and cannot be used for programmatic access.
For programmatic access to different AWS services you will need to create an IAM User which will have access to required services or has a role attached to it with specific policies.
Reference :
AWS IAM Roles
AWS IAM Users
IAM User provides "access-key" and "secrete-key" which can be configured on your machine to access the services from your development environment.
Reference :
AWS Access Credentials
Setting up your AWS credentials
I'm trying to use the aws-sdk-go in my application. It's running on EC2 instance. Now in the Configuring Credentials of the doc,https://docs.aws.amazon.com/sdk-for-go/api/, it says it will look in
*Environment Credentials - Set of environment variables that are useful when sub processes are created for specific roles.
* Shared Credentials file (~/.aws/credentials) - This file stores your credentials based on a profile name and is useful for local development.
*EC2 Instance Role Credentials - Use EC2 Instance Role to assign credentials to application running on an EC2 instance. This removes the need to manage credential files in production.`
Wouldn't the best order be the reverse order? But my main question is do I need to ask the instance if it has a role and then use that to set up the credentials if it has a role? This is where I'm not sure of what I need to do and how.
I did try a simple test of creating a empty config with essentially only setting the region and running it on the instance with the role and it seems to have "worked" but in this case, I am not sure if I need to explicitly set the role or not.
awsSDK.Config{
Region: awsSDK.String(a.region),
MaxRetries: awsSDK.Int(maxRetries),
HTTPClient: http.DefaultClient,
}
I just want to confirm is this the proper way of doing it or not. My thinking is I need to do something like the following
role = use sdk call to get role on machine
set awsSDK.Config { Credentials: credentials form of role,
...
}
issue service command with returned client.
Any more docs/pointers would be great!
I have never used the go SDK, but the AWS SDKs I used automatically use the EC2 instance role if credentials are not found from any other source.
Here's an AWS blog post explaining the approach AWS SDKs follow when fetching credentials: https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/. In particular, see this:
If you use code like this, the SDKs look for the credentials in this
order:
In environment variables. (Not the .NET SDK, as noted earlier.)
In the central credentials file (~/.aws/credentials or
%USERPROFILE%.awscredentials).
In an existing default, SDK-specific
configuration file, if one exists. This would be the case if you had
been using the SDK before these changes were made.
For the .NET SDK, in the SDK Store, if it exists.
If the code is running on an EC2
instance, via an IAM role for Amazon EC2. In that case, the code gets
temporary security credentials from the instance metadata service; the
credentials have the permissions derived from the role that is
associated with the instance.
In my apps, when I need to connect to AWS resources, I tend to use an access key and secret key that have specific predefined IAM roles. Assuming I have those two, the code I use to create a session is:
awsCredentials := credentials.NewStaticCredentials(awsAccessKeyID, awsSecretAccessKey, "")
awsSession = session.Must(session.NewSession(&aws.Config{
Credentials: awsCredentials,
Region: aws.String(awsRegion),
}))
When I use this, the two keys are usually specified as either environment variables (if I deploy to a docker container).
A complete example: https://github.com/retgits/flogo-components/blob/master/activity/amazons3/activity.go
I have this snippet to upload a file on S3
s3 = boto3.resource('s3')
s3.Object('bucketname', timestamped_filename).put(Body=open(FILE_SAVE_PATH, 'rb'))
my bucket has a delete/upload permission for everyone, so it does work on my Windows machine.
However, when I try to run the same code on my Mac it throws
botocore.exeptions.NoCredentialsError: Unable to locate credentials
Is this behavior normal?
And what kind of credentials I can possibly provide if I'm accessing a public bucket?
Thank you.
When making an API call to AWS, valid credentials must be provided. These credentials are associated with an IAM User and grant access to AWS services.
When making API calls (or using the AWS Command-Line Interface (CLI)) from an Amazon EC2 instance, these credentials can be granted to the EC2 instance by assigning an IAM Role to the instance at launch time.
When making calls from a non-EC2 computer, credentials must be provided via a configuration file or environment variables.
It appears that your Windows machine is either an EC2 instance with a role, or it has a local configuration file with valid credentials; and it appears that your Mac has neither of these.
See: boto3 Credentials documentation