This question already has answers here:
AWS ECS - How to retrieve specific key from secret manager?
(3 answers)
Closed 7 days ago.
I am trying to use a secret stored in AWS Secret Manager in my ECS Task Definition. I am able to correctly retrieve the secret, as it exists if I SSM into my running container.
However, the env variable is not being populated correctly. Here is how I am configuring the secret in the task definition:
"secrets": [
{
"name": "test_secret",
"valueFrom": "arn:aws:secretsmanager:<region>:<acc-id>:secret:<some-secret-id>"
}
]
When I SSM into the container, and try to echo the env var I get this: {"key-of-secret":"value-of-secret"}, instead of the env variable having the value of the secret.
I have followed the documentation from AWS, but without any success.
Has anyone encountered this before? I am trying to set the env variable (specified here as test_secret) to the value of the actual secret (specified here as value-of-secret).
Managed to solve this one with extra digging.
The issue is that the secrets manager pulls in the entire secret, which is my case was all of key-value pairs in that secret, but with only 1 entry:
{"key-of-secret":"value-of-secret"}.
In order to access a specific key, and thus retrieve the actual secret value, some extra configuration on the secret ARN needs to be made.
The ARN needs to include the secret's key name you wish to access, appended onto the complete ARN as follows: ::: .
Finally, the updated task definition would look as follows:
"secrets": [
{
"name": "test_secret",
"valueFrom": "arn:aws:secretsmanager:<region>:<acc-id>:secret:<some-secret-id>:<key-of-secret>::"
}
]
This would set the environment variable "test_secret" to the value of the key contained within the secret with id .
Related
I'm using Terraform to deploy an ECS cluster, and task definition has these sections:
{
name = "secret"
valueFrom = "SECRET_NAME"
}
Now, if I'm using a parameter, I can just place a static parameter ARN in the valueFrom field. However, I'm using secrets and Secrets Manager. AWS automatically adds 6 random characters to every deployed secret ARN. I have 4 environments, all taken care of by a single pipeline. I obviously need a way to fit all 4 environments and their secrets in a single valueFrom field. Does this field only take an ARN, or is there some other way for me to do this?
Edit:
I forgot to mention I already have these secrets as an imported resource by name:
data "aws_secretsmanager_secret" "secret" {
name = "secret_name"
}
Is there something I can do with that?
I have a concourse environment deployed using bosh. It is configured with AWS Secrets Manager.
The pipeline secret template is of the form /concourse/{{.Team}}/{{.Secret}}
I have a secret /concourse/team1/general created in AWS Secrets Manager (Other type of secrets) with the below value.
{
"gitbranch": "master",
"hello": "2",
"general": "hi"
}
I have a concourse pipeline hello-world.yml set in team1 team.
---
jobs:
- name: job
public: true
plan:
- task: check-secret
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
run:
path: echo
args: ["((general))"]
This pipeline outputs the value as
{"gitbranch":"master","hello":"2","general":"hi"}
But, if I change the args (last line) in pipeline to args: ["((general.gitbranch))"], then, I get the below error
failed to interpolate task config: cannot access field 'gitbranch' of non-map value ('string') from var: general.gitbranch
Is it possible to access any of the key value pairs in the secret from AWS Secrets Manager, in the concourse pipeline? If yes, how to do so?
Answering my own question.
By creating the secret using cli with the parameter --secret-binary, I was able to achieve to fetch the key value pairs.
(Previously, I was creating the secret from aws console, which got created as a secret string.)
I used the below command to update my secret to create the secret as a binary.
b64key=$(base64 secrets.json)
aws secretsmanager update-secret \
--secret-id /concourse/team1/general \
--secret-binary "$b64key"
I found this using-aws-secrets-manager-with-concourse-ci and it was helpful in solving the issue.
If anyone knows a way to do this in console, kindly let me know.
We have an AWS SecretsManager Secret that was created once. That secret will be updated by an external job every hour.
I have the problem that sometimes the terraform plan/apply fails with the following message:
AWS Provider 2.48
Error: Error refreshing state: 1 error occurred:
* module.xxx.xxx: 1 error occurred:
* module.xxx.aws_secretsmanager_secret_version.xxx:
aws_secretsmanager_secret_version.xxx: error reading Secrets Manager Secret Version: InvalidRequestException: You can't perform this operation on secret version 68AEABC3-34BE-4723-8BF5-469A44F9B1D9 because it was deleted.
We've tried two solutions:
1) Force delete the whole secret via aws cli, but this has the side effect that one of our dependend resources will also be recreated (ecs template definition depends on that secret). This works, but we do not want the side effect of recreating the ecs thing.
2) Manually edit the backend .tfstate file and set the current AWS secret version. Then run the plan again.
Both solution seem to be hacky in a way. What is the best way to solve this issue ?
You can use terraform import to reconcile the state difference before you run a plan or apply.
In your case, this would look like:
terraform import module.xxx.aws_secretsmanager_secret_version.xxx arn:aws:secretsmanager:some_region:some_account_id:secret:example-123456|xxxxx-xxxxxxx-xxxxxxx-xxxxx
I think perhaps the problem you are having is that by default AWS tries to "help you" by not letting you delete secrets automatically until 7 days have elapsed. AWS tries the "help you" by telling you they give you a grace period of 7 days to update your "code" that may rely on this. Which makes automation more difficult.
I have worked around this by setting the recovery window period to "0 days", effectively eliminating that grace period that AWS provides.
Then you can have terraform, rename, or delete your secret at will, either manually (via AWS CLI) or via terraform.
You can update an existing secret by putting in this value FIRST. Then change the name of the secret (if you wish to), or delete it (this terraform section) as desired and run the terraform again after the recovery window days = 0 has been applied.
Here is an example:
resource "aws_secretsmanager_secret" "mySecret" {
name = "your secret name"
recovery_window_in_days = "0"
// this is optional and can be set to true | false
lifecycle {
create_before_destroy = true
}
}
*Note, there is also an option to "create before destroy" you can set on the lifecyle.
https://www.terraform.io/docs/configuration/resources.html
Also, you can use the terraform resource to update the secret values like this:
This example will set the secret values once and then tell terraform to ignore any changes made to the values (username, password in this example) after the initial creation.
If you remove the lifecyle section, then terraform will keep track of whether or not the secret values themselves have changed. If they have changed they would revert back to the value in the terraform state.
If you store your tfstate files in an s3 protected bucket that is safer than not doing so, because they are plaintext in the statefile, so anyone with access to your terraform state file could see your secret values.
I would suggest: 1) figuring out what is deleting your secrets unexpectedly? 2) having your "external job" be a terraform bash script to update the values using a resource as in the example below.
Hope this gives you some ideas.
resource "aws_secretsmanager_secret_version" "your-secret-data" {
secret_id = aws_secretsmanager_secret.your-secret.id
secret_string = <<-EOF
{
"username": "usernameValue",
"password": "passwordValue"
}
EOF
// ignore any updates to the initial values above done after creation.
lifecycle {
ignore_changes = [
secret_string
]
}
}
Is it possible to get AWS access keys without generating one from the console?
I want to be able to create a script that will ask for user/password/(TOTP) and generate temporary access keys in order to perform multiple tasks.
The goal being to be able to give one program to dev so they don't even have to deal with access keys every time since they know their password.
I looked everywhere I believe, but cannot find any resources on if it is even doable.
Thank you!
Yes it is doable.
Both via Command Line Interface (CLI) and API are possible.
The CLI command is:
aws iam create-access-key
The API call is:
CreateAccessKey
See documentation reference below for more information. It is a good read and covers best practice topics like rotating your keys with API or CLI. Another good best practice is NOT using your root account for everyday use.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey_CLIAPI
UPDATE KEY ROTATION EXAMPLE USING CLI:
Before you try this from the CLI - create a new IAM user with Administrator privileges.
Step 1 - configure the CLI to use the Administrator keys:
aws configure
then enter the Access Key ID and Secret Access Key for Administrator.
Step 2 - list the keys for user foo
aws iam list-access-keys --user-name foo
The output will be similar to:
{
"AccessKeyMetadata": [
{
"UserName": "foo",
"AccessKeyId": "AKIAIY*****A7YBHCBEBQ",
"Status": "Active",
"CreateDate": "2018-11-10T14:02:56Z"
}
]
}
Verify there is only one, because a user can have maximum of 2. If they have 2 already, then step 3 will fail.
Step 3 - create a new key for user foo
aws iam create-access-key --user-name foo
you will see output similar to below. This is the only time you will see the secret key for the new set, so you need to preserve it.
{
"AccessKey": {
"UserName": "foo",
"AccessKeyId": "****GAGA*****WEFWEWE",
"Status": "Active",
"SecretAccessKey": "*****sEcReT*****Tasdasd",
"CreateDate": "2018-12-01T19:16:41Z"
}
}
You new key is created and active. Now you need to remove the older key to complete the rotation. I will leave that step up to you.
If you get the error:
An error occurred (InvalidClientTokenId) when calling the ListAccessKeys operation: The security token included in the request is invalid.
then this is a sign you are trying this from an account that your token is old, invalid, or doesnt have the correct privileges.
I would like to get the access key and secret key for a user, while creating the user in CloudFormation.
I followed the AWS Documentation, and I can get the key in Output TAB. My CloudFormation code look like the following:
"Outputs" : {
"MyAccessKeyId": {
"Ref" : "MyAccessKey"
},
"MySecretKey": {
"Fn::GetAtt": [ "MyAccessKey", "SecretAccessKey" ]
}
}
Only issue is that the output contains both access key, and secret key.
So anyone who has access for cloud formation, can get the key.
Is there any way, the secret key would be displayed to me only once, or can be sent to me via mail, when I execute my cloud formation template.
You can try using Lambda backed custom resources with cloud formation. Create the user, then have Lambda request secret key/access key, and use SES to email it.