Im working on creating the CodeBuild which is integrated with SonarQube, So I pass values and sonar credentials directly in my Buildspec.yaml
Instead of Hardcoding directly, I tried to retrieve using the below command from SecretManager as it is mentioned in the below link. But it is not getting the correct values. it throws an error.
Command : '{{resolve:secretsmanager:MyRDSSecret:SecretString:username}}'
Link: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
Error [ERROR] SonarQube server [{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:********:secret:**********:SecretString:SonarURL}}] can not be reached
How I used echo '{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:***:secret:**************:SecretString:*******}}'
Note: All the * inside my commard are the secretname and secreturl
CodeBuild just launched this today - https://aws.amazon.com/about-aws/whats-new/2019/11/aws-codebuild-adds-support-for-aws-secrets-manager/
If you wish to retrieve secrets in your buildspec file, I would recommend to use Systems Manager Parameter Store which is natively integrated with CodeBuild. Systems Manager is a service in itself, search it from the AWS Console homepage, then Paramater Store is in the bottom left of the Systems Manager Console page.
Lets assume you want to include Access Key and Secret Key in buildspec.yml file:
- Create AccessKey/SecretKey pair for a IAM User
- Save the above keys in an SSM parameter store as secure string (e.g. '/CodeBuild/AWS_ACCESS_KEY_ID' and '/CodeBuild/AWS_SECRET_ACCESS_KEY')
- Export the two values in your build environment using the following buildspec directive(s):
version: 0.2
env:
parameter-store:
AWS_ACCESS_KEY_ID_PARAM: /CodeBuild/AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY_PARAM: /CodeBuild/AWS_SECRET_ACCESS_KEY
phases:
build:
commands:
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_PARAM
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_PARAM
# Your Ansible commands below
- ansible-playbook -i hosts ec2-key.yml
[1] Build Specification Reference for CodeBuild - Build Spec Syntax - https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
The dynamic reference syntax you are trying to use only works with the Cloud Formation (CFN) service. In some cases, CFN restricts where these dynamic references to secrets will expand. Specifically, they do not expand in places where the secrets might be visible in the console, such as in EC2 metadata.
If you are trying to setup Code Build via CFN, this may be what you are seeing. However, as shariqmaws mentioned, you can use parameter store and either store your secret there or use parameter store as a pass through to secrets manager (in case you want to use secrets manager to rotate your secrets or for other reasons).
version: 0.2
env:
parameter-store:
AWS_ACCESS_KEY_ID : /terraform-cicd/AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY : /terraform-cicd/AWS_SECRET_ACCESS_KEY
AWS_CODECOMMIT_SSH_ID : /terraform-cicd/AWS_CODECOMMIT_SSH_ID
secrets-manager:
AWS_CODECOMMIT_SSH_PRIVATE: /terraform-cicd/AWS_CODECOMMIT_SSH_PRIVATE
Related
how do you go about making a postgres URI connection string from a Credentials.fromGeneratedSecret() call without writing the secrets out using toString()?
I think I read somewhere making a lambda that does that, but man that seems kinda overkill-ish
const dbCreds = Credentials.fromGeneratedSecret("postgres")
const username = dbCreds.username
const password = dbCreds.password
const uri = `postgresql://${username}:${password}#somerdurl/mydb?schema=public`
Pretty sure I can't do the above. However my hasura and api ECS containers need connection strings like the above, so I figure this is probably a solved thing?
If you want to import a secret that already exists in the Secret's Manager you could just do a lookup of the secret by name or ARN. Take a look at the documentation referring how to get a value from AWS Secrets Manager.
Once you have your secret in the code it is easy to pass it on as an environment variable to your application. With CDK it is even possible to pass secrets from Secrets Manager or AWS Systems Manager Param Store directly onto the CDK construct. One such example would be (as pointed in the documentation):
taskDefinition.addContainer('container', {
image: ecs.ContainerImage.fromRegistry("amazon/amazon-ecs-sample"),
memoryLimitMiB: 1024,
environment: { // clear text, not for sensitive data
STAGE: 'prod',
},
environmentFiles: [ // list of environment files hosted either on local disk or S3
ecs.EnvironmentFile.fromAsset('./demo-env-file.env'),
ecs.EnvironmentFile.fromBucket(s3Bucket, 'assets/demo-env-file.env'),
],
secrets: { // Retrieved from AWS Secrets Manager or AWS Systems Manager Parameter Store at container start-up.
SECRET: ecs.Secret.fromSecretsManager(secret),
DB_PASSWORD: ecs.Secret.fromSecretsManager(dbSecret, 'password'), // Reference a specific JSON field, (requires platform version 1.4.0 or later for Fargate tasks)
PARAMETER: ecs.Secret.fromSsmParameter(parameter),
}
});
Overall, in this case, you would not have to do any parsing or printing of the actual secret within the CDK. You can handle all of that processing within you application using properly set environment variables.
However, only from your question it is not clear what exactly you are trying to do. Still, the provided resources should get you in the correct direction.
Does anybody know a work-around for converting account numbers that start with “00” to string? I am using Mappings in CFn template to assign values based on the account number. I put the account number in quotes for converting it to string and it works well if it does not start with a zero, and I get the following error when it does.:
[/Mappings/EnvMap] map keys must be strings; received numeric [1.50xxx028E9]
Mappings:
EnvMap:
"8727xxxx0":
env: "dev"
"707xxxx78":
env: "test"
"00150xxx280":
env: "prod"
Resources:
rS3Stack:
Type: "AWS::CloudFormation::Stack"
Properties:
TemplateURL: "https://s3.amazonaws.com/some_bucket/nested_cfn/s3.yaml"
Parameters:
pEnvironment: !FindInMap
- EnvMap
- !Ref 'AWS::AccountId'
- env
Your problem is caused by a bug in PyYAML which results from some ambiguity in the YAML 1.1 specification. According to YAML 1.1 an integer must not start with 0 and numbers starting with 0 and are considered octal numbers. So when PyYAML parses the account id it considers the account id not to be an integer, because it's starting with 0, but also not an octal number, because it contains an 8. As it's neither an integer, nor an octal number, PyYAML considers it a string, which is safe to get dumped without surrounding quotes.
A minimal example to reproduce this looks like this:
>>> import sys
>>> import yaml
>>> yaml.dump(["1", "8", "01", "08"], sys.stdout)
- '1'
- '8'
- '01'
- 08
Now you might wonder why a PyYAML bug is mentioned here, while you just want to deploy a CloudFormation stack:
Depending on how you deploy a CloudFormation stack the template might get transformed locally, before it gets deployed. That happens for example when using aws cloudformation package, sam package or sam build to replace local code locations with paths in S3. As reading and writing the template during those transformations is done using PyYAML, it triggers the PyYAML bug mentioned above. There are bug reports for the AWS CLI and the AWS SAM CLI regarding this problem.
As the account id causing the problem is used as a key in your case, your options to work around that problem are limited, as you can't utilize CloudFormation's intrinstic functions. However there are still possible workarounds:
If you're using the AWS CLI, you can switch to using the AWS CLI v2, which doesn't suffer from this bug as it uses ruamel instead of PyYAML. ruamel handles numbers as one would expect, as it implements YAML 1.2, which doesn't contain the ambiguity in its specification.
What you can use no matter if you're using the AWS SAM CLI or the AWS CLI is to convert the transformed template from YAML to JSON and back to YAML which "fixes" that bug as well, as it results in problematic numbers being quoted again. There is a tool called cfn-flip from AWS to do so. You'd have to run this double-flip between packaging and deployment. For the AWS SAM CLI that'd for example look like:
sam build
cfn-flip .aws-sam/build/template.yaml | cfn-flip > .aws-sam/build/template.tmp.yaml
mv .aws-sam/build/template.tmp.yaml .aws-sam/build/template.yaml
sam deploy
With this said, I personally would suggest a completely different workaround and that's to remove that mapping from the template. Hardcoding account ids and environments makes a template less portable, as it limits the accounts/environments this template can be used for. I'd instead provide the environment as a parameter to the CloudFormation template, so it doesn't need to be aware of account ids at all.
I've used the AWS Console to put together an AppSync API that uses resolvers to write to an RDB. I've set up the codegen with amplify add codegen --apiId myId but when I run amplify codegen types, there is no console output and I can't see any types being created in my project.
I understand that I need these auto-generated types when I try to run queries. Should this be working? Where would I expect to see the types get created? I see the introspection schema at app\src\main\graphql\schema.json but I don't see any .java or .kt files getting created.
This is my .graphqlconfig.yml:
projects:
FocalistTest1:
schemaPath: app/src/main/graphql/schema.json
includes:
- app/src/main/graphql/**/*.graphql
excludes:
- ./amplify/**
extensions:
amplify:
codeGenTarget: ''
generatedFileName: ''
docsFilePath: app/src/main/graphql/com/amazonaws/amplify/generated/graphql
region: us-east-2
apiId: [snipped]
maxDepth: 2
extensions:
amplify:
version: 3
The generatedFileName is empty. That should be set to a path to the target file that you want generatedFileName: 'path/to/file'.
Derp. Found them in the generated files section after a build. I had been checking the git files to see if the command had created anything, but they must be under an ignored folder.
I'm looking at this repo and very confused about what's happening here: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/examples/v2/cloud_functions
In other Deployment Manager examples I see the "type" is set to the type of resource being deployed but in this example I see this:
resources:
- name: function
type: cloud_function.py # why not "type: cloudfunctions"?
properties:
# All the files that start with this prefix will be packed in the Cloud Function
codeLocation: function/
codeBucket: mybucket
codeBucketObject: function.zip
location: us-central1
timeout: 60s
runtime: nodejs8
availableMemoryMb: 256
entryPoint: handler
"type" is pointing to a python script (cloud_function.py) instead of a resource type. The script is over 100 lines long and does a whole bunch of stuff.
This looks like a hack, like its just scripting the GCP APIs? The reason I'd ever want to use something like Deployment Manager is to avoid a mess of deployment scripts but this looks like it's more spaghetti.
Does Deployment Manager not support Cloud Functions and this is a hacky workaround or is this how its supposed to work? The docs for this example are bad so I don't know what's happening
Also, I want to deploy multiple function into a single Deployment Manager stack- will have to edit the cloud_function.py script or can I just define multiple resources and have them all point to the same script?
Edit
I'm also confused about what these two imports are for at the top of the cloud_function.yaml:
imports:
# The function code will be defined for the files in function/
- path: function/index.js
- path: function/package.json
Why is it importing the actual code of the function it's deploying?
Deployment manager simply interacts with the different kind of Google APIs. This documentation gives you a list of supported resource types by Deployment manager. I would recommend you to run this command “gcloud deployment-manager types list | grep function” and you will find this “cloudfunctions.v1beta2.function” resource type is also supported by DM.
The template is using a gcp-type (that is in beta).The cloud_functions.py is a template. If you use a template, you can reuse it for multiple resources, you can this see example. For better understanding, easier to read/follow you can check this example of cloud functions through gcp-type.
I wan to add to the answer by Aarti S that gcloud deployment-manager types list | grep function doesn't work for me as I found how to all list of resource types, including resources that are in alpha:
gcloud beta deployment-manager types list --project gcp-types
Or just gcloud beta deployment-manager types list | grep function helps.
I am using CDK to set up code pipelines in AWS. The pipeline stage needs to download the source code from github so uses an oauth token to authenticate the request. I would like to be able to access the token from AWS Parameter Store and NOT from AWS Secret Manager when setting the value in the stage of the pipeline.
There are plenty of examples using Secret Manager to do this. However there are no examples using the Parameter Store or hardcoding the token in plain text within the CDK project.
We are using typescript with CDK 1.3.0.
I have tried storing the token in the Parameter Store. When storing as a secure String you need to additionally specify the version when retrieving the value. However I cannot then cast to a SecretValue that is required to set oauthToken property in the pipeline stage.
Get the value from the Parameter Store ..
// get the secureString
const secureString = ssm.StringParameter.fromSecureStringParameterAttributes(construct,'MySecretParameter', {
parameterName: 'my-secure-parameter-name',
version: 1,
});
I need to cast the secretString to a CDK.SecretValue to then use it to set the oauthToken. I cannot see how to do this.
const sourceAction = new codepipelineactions.GitHubSourceAction({
actionName: 'Source',
owner: owner,
repo: repository,
oauthToken: githubOAuthAccessToken,
output: sourceOutput,
branch: branch,
trigger: codepipelineactions.GitHubTrigger.WEBHOOK,
});
The CDK documentation says that is is advisable to store tokens in Secret Manager.
"It is recommended to use a Secret Manager SecretString to obtain the token"
It does not say that tokens cannot be retrieved from other sources and used. I would be grateful if the situation could be clarified and if anyone stores tokens outside Secrets Manager and is still able to use them to set the Token in the source stage of a pipeline.
You can use cdk.SecretValue.ssmSecure or cdk.SecretValue.plainText:
oauthToken: cdk.SecretValue.ssmSecure('param-name', 'version');
// OR
oauthToken: cdk.SecretValue.plainText('oauth-token-here');
From the doc for plainText:
Do not use this method for any secrets that you care about. The only reasonable use case for using this method is when you are testing.
The previous answer by #jogold does partially work. However, at the time of this writing SecretValue.ssmSecure is not supported by Cloudformation and you will get an error such as: FAILED, SSM Secure reference is not supported in: .
There is an open issue on the CDK roadmap: https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/227. The plaintext option is not truly viable as the secret will be exposed in CFN template.