I used to hide connection credentials in environmental variables (.bash_profile). Recently working with SageMaker, I tried a similar process with the terminal available in SageMaker but I am getting the following error,
NameError: name 'DB_USER' is not defined
Is there any efficient way to hide the credentials in SageMaker?
the recommended way to handle secret storage within AWS is AWS Secrets Manager. Secrets Manager stores secret in a secured fashion as a key-value pair. The key benefit is that it allows you to administer access to those secrets via IAM roles and permission abstractions, and retrieve them with the SDK of your choice, such as boto3 for example. Secrets Manager is actually also used by Amazon SageMaker for git credential storage in the case of third-party git integrations
Extending on Olivier's answers, you could provide your Sagemaker endpoint with the proper roles in the deployment code like so
role = 'arn:aws:iam::xxxxxxxxxx:role/service-role/AmazonSageMaker-ExecutionRole-xxxxxxxxxx:role'
sagemaker_model = MXNetModel(model_data = 's3://' + bucket + '/model/model.tar.gz',
role = role,
entry_point = 'entry_point.py',
py_version='py3',
framework_version='1.4.1',
sagemaker_session = sagemaker_session)
Just remember to provide the necessary permissions in the Role you provided
Related
I must be missing something in how AWS secrets can be accessed through Terraform. Here is the scenario I am struggling with:
I create an IAM user named "infra_user", create ID and secret access key for the user, download the values in plain txt.
"infra_user" will be used to authenticate via terraform to provision resources, lets say an S3 and an EC2 instance.
To protect the ID and secret key of "infra_user", I store them in AWS secrets manager.
In order to authenticate with "infra_user" in my terraform script, I will need to retrieve the secrets via the following block:
data "aws_secretsmanager_secret" "arn" {
arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:example-123456"
}
But, to even use the data block in my script and retrieve the secrets wouldn't I need to authenticate to AWS in some other way in my provider block before I declare any resources?
If I create another user, say "tf_user", to just retrieve the secrets where would I store the access key for "tf_user"? How do I avoid this circular authentication loop?
The Terraform AWS provider documentation has a section on Authentication and Configuration and lists an order of precedence for how the provider discovers which credentials to use. You can choose the method that makes the most sense for your use case.
For example, one (insecure) method would be to set the credentials directly in the provider:
provider "aws" {
region = "us-west-2"
access_key = "my-access-key"
secret_key = "my-secret-key"
}
Or, you could set the environment variables in your shell:
export AWS_ACCESS_KEY_ID="my-access-key"
export AWS_SECRET_ACCESS_KEY="my-secret-key"
export AWS_DEFAULT_REGION="us-west-2"
now your provider block simplifies to:
provider "aws" {}
when you run terraform commands it will automatically use the credentials in your environment.
Or as yet another alternative, you can store the credentials in a profile configuration file and choose which profile to use by setting the AWS_PROFILE environment variable.
Authentication is somewhat more complex and configurable than it seems at first glance, so I'd encourage you to read the documentation.
I would like to use AWS Assume Roles, with Terraform Cloud / Enterprise
In Terraform Open Source, you would typically just do an Assume Role, leveraging the .aws/Credential Profile on the CLI, which is the initial authentication, and performing the Assume Role:
provider "aws" {
assume_role {
role_arn = "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
session_name = "SESSION_NAME"
external_id = "EXTERNAL_ID"
}
}
The issue is, with Terraform Enterprise or Cloud, you cannot reference a profile, as the immutable infrastructure will not have that file in its directory.
Terraform Cloud/Enterprise needs to have an Access Key ID, and Secret Access Key, set as a variable, so its infrastructure can perform the Terraform RUN, via its Pipeline, and authenticate to what ever AWS Account you would like to provision within.
So the question is:
How can I perform an AWS Assume Role, leveraging the Access Key ID, and Secret Access Key, of the AWS account with the "Action": "sts:AssumeRole", Policy?
I would think, the below would work, however Terraform is doing the initial authentication via the AWS Credential Profile creds, for the account which has the sts:AssumeRole policy
Can Terraform look at the access_key, and secret_key, to determine what AWS account to use, when trying to assume the role, rather than use the AWS Credential Profile?
provider "aws" {
region = var.aws_region
access_key = var.access_key_id
secret_key = var.secret_access_key
assume_role {
role_arn = "arn:aws:iam::566264069176:role/RemoteAdmin"
#role_arn = "arn:aws:iam::<awsaccount>:role/<rolename>" # Do a replace in "file_update_automation.ps1"
session_name = "RemoteAdminRole"
}
}
In order to allow Terraform Cloud/Enterprise to get new Assume Role Session Tokens, it would need to use the Access_key and Secret_key, to tell it what AWS Account has the sts:assume role, linking to the member AWS Account to be provisioned, and not an AWS Creds Profile
Thank you
This can be achive if you have a business plan enabled and implement self hosted terraform agents in you infrastructure.See video.
I used the exact same provider configuration minus the explicit adding of the acces keys.
The access keys were added in the Terraform Cloud workspace as environment variables.
This is definitely possible with Terraform Enterprise (TFE) if your TFE infrastructure is also hosted in AWS and the instance profile is trusted by the role you are trying to assume.
For Terraform Cloud (TFC) it is a different story, today there is no way to create a trust between TFC and an IAM role, but we can leverage the AWS SDK's ability to pickup credentials from environment variables. You have 2 options:
Create an AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variable on the workspace and set them (remember to mark the secret access key as sensitive). The provider will pickup these in the env and work the same as on your local.
If all workspaces need to use the same access and secret keys, you can set the env variables on a variable set, which will apply to all workspaces.
I am using AWS Secret Manager Service to retrieve some confidential information like SMTP details or connection strings. However, to get secret value from AWS Secret Manager Service it seems like we need to pass the Access key and secret key apart from which secret we want to retrieve. So I am maintaining those values in config file.
public AwsSecretManagerService(IOptions<AwsAppSettings> settings)
{
awsAppSettings = settings.Value;
amazonSecretsManagerClient = new AmazonSecretsManagerClient
(awsAppSettings.Accesskey, awsAppSettings.SecretKey, RegionEndpoint.GetBySystemName(awsAppSettings.Region));
}
public async Task<SecretValueResponse> GetSecretValueAsync(SecretValueRequest secretValueRequest)
{
return _mapper.Map<SecretValueResponse>(await amazonSecretsManagerClient.GetSecretValueAsync(_mapper.Map<GetSecretValueRequest>(secretValueRequest)));
}
So I am thinking I am kind of defeating the whole purpose of using secret manager by maintaining the AWS credentials in app settings file. I am wondering what is the right way to do this
It is not a good practice to pass or add AWS credentials of an IAM User (access key and secret access key) in the code.
Instead, don't pass it and update your code as follows:
amazonSecretsManagerClient = new AmazonSecretsManagerClient
(RegionEndpoint.GetBySystemName(awsAppSettings.Region));
Question: Then how would it access the AWS services?
Answer: If you are going to execute your code on your local system, install and configure AWS CLI instead of passing AWS credentials via CLI or Terminal, it will use those AWS configured credentials to access the AWS services.
Reference for AWS CLI Installation: Installing the AWS CLI
Reference for AWS CLI Configuration: Configuring the AWS CLI
If you are going to execute your code on an AWS service (e.g., EC2 instance), attach an IAM role with that AWS resource (e.g., EC2 instance) having sufficient permissions, it will use that IAM role to access the AWS services.
For different AWS services, I need different IAM users to secure the access control. Sometimes, I even need to use different IAM user credentials within a single project in a EC2 instance. What's the proper way to manage this and how I can deploy/attach these IAM user credentials to a single EC2 instance?
While I fully agree with accepted answer that using static credentials is one way of solving this problem, I would like to suggest some improvements over it (and proposed Secrets Manager).
What I would advise as architectural step forward to achieve full isolation of credentials, having them dynamic, and not stored in central place (Secrets Manager proposed above) is dockerizing application and running on AWS Elastic Container Service (ECS). This way you can assign different IAM role to different ECS Tasks.
Benefits over Secrets Manager solution
- use case of someone tampering with credentials in Secrets Manager is fully avoided, as credentials are of dynamic nature (temporary, and automatically assumed through SDKs)
Credentials are managed on AWS side for you
Only ECS Service can assume this IAM role, meaning you can't have actual person stealing the credentials, or developer connecting to production environment from his local machine with this credentials.
AWS Official Documentation for Task Roles
The normal way to provide credentials to applications running on an Amazon EC2 instance is to assign an IAM Role to the instance. Temporary credentials associated with the role when then be provided via Instance Metadata. The AWS SDKs will automatically use these credentials.
However, this only works for one set of credentials. If you wish to use more than one credential, you will need to provide the credentials in a credentials file.
The AWS credentials file can contain multiple profiles, eg:
[default]
aws_access_key_id = AKIAaaaaa
aws_secret_access_key = abcdefg
[user2]
aws_access_key_id = AKIAbbbb
aws_secret_access_key = xyzzzy
As a convenience, this can also be configured via the AWS CLI:
$ aws configure --profile user2
AWS Access Key ID [None]: AKIAbbbb
AWS Secret Access Key [None]: xyzzy
Default region name [None]: us-east-1
Default output format [None]: text
The profile to use can be set via an Environment Variable:
Linux: export AWS_PROFILE="user2"
Windows: set AWS_PROFILE="user2"
Alternatively, when calling AWS services via an SDK, simply specify the Profile to use. Here is an example with Python from Credentials — Boto 3 documentation:
session = boto3.Session(profile_name='user2')
# Any clients created from this session will use credentials
# from the [user2] section of ~/.aws/credentials.
dev_s3_client = session.client('s3')
There is an equivalent capability in the SDKs for other languages, too.
We want to use AWS services via API calls, CLI, Etc., from our on-premise infrastructure as well as from AWS cloud infrastructure.
As we know, we can use the AWS access key as follow:
This is a snippet from an example
// Setup AWS SNS
AWS.config.update({
region: 'eu-west-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
});
var sns = new AWS.SNS();
var params = {
Message: "SMS message test",
MessageStructure: 'string',
PhoneNumber: '0045xxxxxxxx',
Subject: 'Alarm',
MessageAttributes :{
'AWS.SNS.SMS.SenderID': {
'DataType': 'String',
'StringValue': 'MySender'
},
'AWS.SNS.SMS.SMSType': 'Transactional'
}
};
sns.publish(params, function(err_publish, data) {
if (err_publish) {}
});
This code uses the access keys through the environment variables. That approach is partially accepted because you're tied to modify those environment variables to update access keys.
So, what are the best practices or different approaches for using the access keys in a good manner?
AWS Documentation
Access keys consist of an access key ID (for example, AKIAIOSFODNN7EXAMPLE) and a secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY). You use access keys to sign programmatic requests that you make to AWS if you use the AWS SDKs, REST, or Query API operations. The AWS SDKs use your access keys to sign requests for you, so that you don't have to handle the signing process. You can also sign requests manually. For more information, see Signing AWS API Requests.
Access keys are also used with command line interfaces (CLIs). When you use a CLI, the commands that you issue are signed by your access keys. You can pass access keys either with the command or store as configuration settings on your computer.
Temporary access keys, known as temporary security credentials
In addition to the access key ID and secret access key, temporary security credentials include a security token that you must send to AWS when you use temporary security credentials
Advantages
They are short term.
After they expire, they're no longer valid.
You can use temporary access keys in less secure environments or distribute them to grant users temporary access to resources in your AWS account.
For example, you can grant entities from other AWS accounts access to resources in your AWS account (cross-account access). You can also grant users who don't have AWS security credentials access to resources in your AWS account (federation). For more information, see Temporary Security Credentials in the IAM User Guide.
Approaches for using Access keys
Access keys in Environment variables
This approach is the most common for development and testing environments because they will test their developments using a close scope, likewise for scenarios where our apps are deployed within an on-premise infrastructure.
Usage of Access keys through Environment variables (NodeJs)
// Setup AWS SNS
AWS.config.update({
region: 'eu-west-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
});
var sns = new AWS.SNS();
var params = {...});
sns.publish(params, function(err_publish, data) {...});
Access keys through instance metadata
This is the most secure way to use the Access keys within an EC2 or container because you don't need to put any Access keys neither in your code nor in Environment variables.
Retrieving Security Credentials from Instance Metadata
The following command retrieves the security credentials for an IAM role named s3access.
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
The following is an example output
{
"Code" : "Success",
"LastUpdated" : "2012-04-26T16:39:16Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAIOSFODNN7EXAMPLE",
"SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"Token" : "token",
"Expiration" : "2017-05-17T15:09:54Z"
}
IAM Roles
The IAM Roles provide a good and secure way to grant permissions to your services.
IAM Role console: This is how looks like
The IAM Role allows you to describe a policy with the specific permissions. That role could be attached to an EC2 instance and automatically the service within it will be granted with those permissions. So, we can execute API calls without the need to put the Access keys:
var s3 = new AWS.S3({params:{Bucket: 'bucketname', Key: 'filename'}});
var body = fs.createReadStream('file_to_upload');
s3.upload({Body:body})
.on('httpUploadProgress',function(evt){
console.log(evt);
})
.send(function(err,data){
console.log(err,data);
});
As you can see, there is any Access keys in that code because the SDK will get the Access keys from /latest/meta-data/iam/security-credentials/s3access.
Access keys within Named Profiles
The Named profiles are used for CLI (Command line interface), so you can create profiles with different usages. For example, you can use the AWS access keys from a specific region.
The AWS CLI supports named profiles stored in the config and credentials files. You can configure additional profiles by using aws configure with the --profile option or by adding entries to the config and credentials files.
The following example shows a credentials file with two profiles:
~/.aws/credentials
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
[user2]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Each profile uses different credentials—perhaps from two different IAM users—and can also use different regions and output formats.
~/.aws/config
[default]
region=us-west-2
output=json
[profile user2]
region=us-east-1
output=text
Using Profiles with the AWS CLI
$ aws ec2 describe-instances --profile user2
The command above will use the credentials within profile user2
Best practices for using Access keys
Remove (or Don't Generate) Account Access Key
An access key is required in order to sign requests that you make using the AWS Command Line Tools, the AWS SDKs, or direct API calls. Anyone who has the access key for your AWS account root user has unrestricted access to all the resources in your account, including billing information. You cannot restrict the permissions for your AWS account root user.
Use Temporary Security Credentials (IAM Roles) Instead of Long-Term Access Keys
In many scenarios, you don't need a long-term access key that never expires (as you have with an IAM user). Instead, you can create IAM roles and generate temporary security credentials. Temporary security credentials consist of an access key ID and a secret access key, but they also include a security token that indicates when the credentials expire.
Manage IAM User Access Keys Properly
If you do need to create access keys for programmatic access to AWS, create an IAM user and grant that user only the permissions he or she needs. Then generate an access key for that user. For details, see Managing Access Keys for IAM Users in the IAM User Guide.
Precautions when using access keys
Don't embed access keys directly into code.
Use different access keys for different applications.
Rotate access keys periodically.
Remove unused access keys.
Configure multi-factor authentication for your most sensitive operations.
Resources
AWS Security Credentials
AWS Account Root User Credentials vs. IAM User Credentials
AWS Security Audit Guidelines
AWS Account Identifiers
Using Instance Profiles