Does anybody know a work-around for converting account numbers that start with “00” to string? I am using Mappings in CFn template to assign values based on the account number. I put the account number in quotes for converting it to string and it works well if it does not start with a zero, and I get the following error when it does.:
[/Mappings/EnvMap] map keys must be strings; received numeric [1.50xxx028E9]
Mappings:
EnvMap:
"8727xxxx0":
env: "dev"
"707xxxx78":
env: "test"
"00150xxx280":
env: "prod"
Resources:
rS3Stack:
Type: "AWS::CloudFormation::Stack"
Properties:
TemplateURL: "https://s3.amazonaws.com/some_bucket/nested_cfn/s3.yaml"
Parameters:
pEnvironment: !FindInMap
- EnvMap
- !Ref 'AWS::AccountId'
- env
Your problem is caused by a bug in PyYAML which results from some ambiguity in the YAML 1.1 specification. According to YAML 1.1 an integer must not start with 0 and numbers starting with 0 and are considered octal numbers. So when PyYAML parses the account id it considers the account id not to be an integer, because it's starting with 0, but also not an octal number, because it contains an 8. As it's neither an integer, nor an octal number, PyYAML considers it a string, which is safe to get dumped without surrounding quotes.
A minimal example to reproduce this looks like this:
>>> import sys
>>> import yaml
>>> yaml.dump(["1", "8", "01", "08"], sys.stdout)
- '1'
- '8'
- '01'
- 08
Now you might wonder why a PyYAML bug is mentioned here, while you just want to deploy a CloudFormation stack:
Depending on how you deploy a CloudFormation stack the template might get transformed locally, before it gets deployed. That happens for example when using aws cloudformation package, sam package or sam build to replace local code locations with paths in S3. As reading and writing the template during those transformations is done using PyYAML, it triggers the PyYAML bug mentioned above. There are bug reports for the AWS CLI and the AWS SAM CLI regarding this problem.
As the account id causing the problem is used as a key in your case, your options to work around that problem are limited, as you can't utilize CloudFormation's intrinstic functions. However there are still possible workarounds:
If you're using the AWS CLI, you can switch to using the AWS CLI v2, which doesn't suffer from this bug as it uses ruamel instead of PyYAML. ruamel handles numbers as one would expect, as it implements YAML 1.2, which doesn't contain the ambiguity in its specification.
What you can use no matter if you're using the AWS SAM CLI or the AWS CLI is to convert the transformed template from YAML to JSON and back to YAML which "fixes" that bug as well, as it results in problematic numbers being quoted again. There is a tool called cfn-flip from AWS to do so. You'd have to run this double-flip between packaging and deployment. For the AWS SAM CLI that'd for example look like:
sam build
cfn-flip .aws-sam/build/template.yaml | cfn-flip > .aws-sam/build/template.tmp.yaml
mv .aws-sam/build/template.tmp.yaml .aws-sam/build/template.yaml
sam deploy
With this said, I personally would suggest a completely different workaround and that's to remove that mapping from the template. Hardcoding account ids and environments makes a template less portable, as it limits the accounts/environments this template can be used for. I'd instead provide the environment as a parameter to the CloudFormation template, so it doesn't need to be aware of account ids at all.
Related
I need to configure a lambda via serverless.yml to use different provision concurrency for different environments. Below is my lambda configuration:
myLambda:
handler: src/lambdas
name: myLambda
provisionedConcurrency: ${self:custom.pc}
...
custom:
pc: ${env:PC}
The value PC is loaded from environment variable. It works for values greater than 0 but I can't set a value 0 in one environment. What I want to do is to disable provision concurrency in dev environment.
I have read through this doc https://forum.serverless.com/t/conditional-serverless-yml-based-on-stage/1763/3 but it doesn't seem to help in my case.
How can I set provisionedConcurrency conditional based on environment?
Method 1: Stage-based variables via default values
This is a fairly simple trick by using a cascading value variable. The first value is the one you want, the second one being a default, or fallback value. Also called cascading variables.
// serverless.yml
provider:
stage: "dev"
custom:
provisionedConcurrency:
live: 100
staging: 50
other: 10
myLambda:
handler: src/lambdas
name: myLambda
provisionedConcurrency: ${self:custom.provisionedConcurrency.${self:provider.stage}, self:custom.provisionedConcurrency.other}
This above with stage set to dev will default to "other" value of 10, but if you set stage via serverless deploy --stage live then it will use the live value of 100.
See here for more details: https://www.serverless.com/framework/docs/providers/aws/guide/variables#syntax
Method 2: Asynchonous Value via Javascript
You can use an js include and put your conditional logic there. It's called "asynchronous value support". Basically, this allows you to put logic in a javascript file which you include and it can return different values depending on various things (like, what AWS account you're on, or if certain variables are set, or whatever). Basically, it allows you to do this...
provisionedConcurrency: ${file(./detect_env.js):get_provisioned_concurrency}
Which works if you create a javascript file in this folder called detect_env.js, and it has the contents similar to...
module.exports.get_provisioned_concurrency = () => {
if ("put logic to detect which env you are deploying to, eg for live") {
return Promise.resolve('100');
} else {
// Otherwise fallback to 10
return Promise.resolve('10');
}
}
For more info see: https://www.serverless.com/framework/docs/providers/aws/guide/variables#with-a-new-variables-resolver
I felt I had to reply here even though this was asked months ago because none of the answers were even remotely close to the right answer and I really felt sorry for the author or anyone who lands here.
For really sticky problems, I find it's useful to go to the Cloudformation script instead and use the Cloudformation Intrinsic Functions.
For this case, if you know all the environments you could use Fn::FindInMap
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-findinmap.html
Or if it's JUST production which needs 0 then you could use the conditional Fn::If and a boolean Condition test in the Cloudformation template to test if environment equals production, use 0, else use the templated value from SLS.
Potential SLS:
resources:
Conditions:
UseZero: !Equals ["production", ${provider.stage}]
Resources:
myLambda:
ProvisionedConcurrency: !If [UseZero, 0, ${self:custom.pc}]
You can explicitly remove the ProvisionedConcurrency property as well if you want:
resources:
Conditions:
UseZero: !Equals ["production", ${provider.stage}]
Resources:
myLambda:
ProvisionedConcurrency: !If [UseZero, AWS::NoValue, ${self:custom.pc}]
Edit: You can still use SLS to deploy; it simply compiles into a Cloudformation JSON template which you can explicitly modify with the SLS resources field.
The Serverless Framework provides a really useful dashboard tool with a feature called Parameters. Essentially what it lets you do is connect your service to it then you can set different values for different stages and then use those values in your serverless.yml with syntax like ${param:VARAIBLE_NANE_HERE} and it gets replaced at deploy time with the right value for whatever stage you are currently deploying. Super handy. There are also a bunch of other features in the dashboard such as monitoring and troubleshooting.
You can find out more about Parameters at the official documentation here: https://www.serverless.com/framework/docs/guides/parameters/
And how to get started with the dashboard here: https://www.serverless.com/framework/docs/guides/dashboard#enabling-the-dashboard-on-existing-serverless-framework-services
Just using a variable with a null value for dev environments during on deploy/package and SLS will skip this property:
provisionedConcurrency: ${self:custom.variables.provisionedConcurrency}
I'm looking at this repo and very confused about what's happening here: https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/examples/v2/cloud_functions
In other Deployment Manager examples I see the "type" is set to the type of resource being deployed but in this example I see this:
resources:
- name: function
type: cloud_function.py # why not "type: cloudfunctions"?
properties:
# All the files that start with this prefix will be packed in the Cloud Function
codeLocation: function/
codeBucket: mybucket
codeBucketObject: function.zip
location: us-central1
timeout: 60s
runtime: nodejs8
availableMemoryMb: 256
entryPoint: handler
"type" is pointing to a python script (cloud_function.py) instead of a resource type. The script is over 100 lines long and does a whole bunch of stuff.
This looks like a hack, like its just scripting the GCP APIs? The reason I'd ever want to use something like Deployment Manager is to avoid a mess of deployment scripts but this looks like it's more spaghetti.
Does Deployment Manager not support Cloud Functions and this is a hacky workaround or is this how its supposed to work? The docs for this example are bad so I don't know what's happening
Also, I want to deploy multiple function into a single Deployment Manager stack- will have to edit the cloud_function.py script or can I just define multiple resources and have them all point to the same script?
Edit
I'm also confused about what these two imports are for at the top of the cloud_function.yaml:
imports:
# The function code will be defined for the files in function/
- path: function/index.js
- path: function/package.json
Why is it importing the actual code of the function it's deploying?
Deployment manager simply interacts with the different kind of Google APIs. This documentation gives you a list of supported resource types by Deployment manager. I would recommend you to run this command “gcloud deployment-manager types list | grep function” and you will find this “cloudfunctions.v1beta2.function” resource type is also supported by DM.
The template is using a gcp-type (that is in beta).The cloud_functions.py is a template. If you use a template, you can reuse it for multiple resources, you can this see example. For better understanding, easier to read/follow you can check this example of cloud functions through gcp-type.
I wan to add to the answer by Aarti S that gcloud deployment-manager types list | grep function doesn't work for me as I found how to all list of resource types, including resources that are in alpha:
gcloud beta deployment-manager types list --project gcp-types
Or just gcloud beta deployment-manager types list | grep function helps.
Im working on creating the CodeBuild which is integrated with SonarQube, So I pass values and sonar credentials directly in my Buildspec.yaml
Instead of Hardcoding directly, I tried to retrieve using the below command from SecretManager as it is mentioned in the below link. But it is not getting the correct values. it throws an error.
Command : '{{resolve:secretsmanager:MyRDSSecret:SecretString:username}}'
Link: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
Error [ERROR] SonarQube server [{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:********:secret:**********:SecretString:SonarURL}}] can not be reached
How I used echo '{{resolve:secretsmanager:arn:aws:secretsmanager:us-east-1:***:secret:**************:SecretString:*******}}'
Note: All the * inside my commard are the secretname and secreturl
CodeBuild just launched this today - https://aws.amazon.com/about-aws/whats-new/2019/11/aws-codebuild-adds-support-for-aws-secrets-manager/
If you wish to retrieve secrets in your buildspec file, I would recommend to use Systems Manager Parameter Store which is natively integrated with CodeBuild. Systems Manager is a service in itself, search it from the AWS Console homepage, then Paramater Store is in the bottom left of the Systems Manager Console page.
Lets assume you want to include Access Key and Secret Key in buildspec.yml file:
- Create AccessKey/SecretKey pair for a IAM User
- Save the above keys in an SSM parameter store as secure string (e.g. '/CodeBuild/AWS_ACCESS_KEY_ID' and '/CodeBuild/AWS_SECRET_ACCESS_KEY')
- Export the two values in your build environment using the following buildspec directive(s):
version: 0.2
env:
parameter-store:
AWS_ACCESS_KEY_ID_PARAM: /CodeBuild/AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY_PARAM: /CodeBuild/AWS_SECRET_ACCESS_KEY
phases:
build:
commands:
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_PARAM
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_PARAM
# Your Ansible commands below
- ansible-playbook -i hosts ec2-key.yml
[1] Build Specification Reference for CodeBuild - Build Spec Syntax - https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
The dynamic reference syntax you are trying to use only works with the Cloud Formation (CFN) service. In some cases, CFN restricts where these dynamic references to secrets will expand. Specifically, they do not expand in places where the secrets might be visible in the console, such as in EC2 metadata.
If you are trying to setup Code Build via CFN, this may be what you are seeing. However, as shariqmaws mentioned, you can use parameter store and either store your secret there or use parameter store as a pass through to secrets manager (in case you want to use secrets manager to rotate your secrets or for other reasons).
version: 0.2
env:
parameter-store:
AWS_ACCESS_KEY_ID : /terraform-cicd/AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY : /terraform-cicd/AWS_SECRET_ACCESS_KEY
AWS_CODECOMMIT_SSH_ID : /terraform-cicd/AWS_CODECOMMIT_SSH_ID
secrets-manager:
AWS_CODECOMMIT_SSH_PRIVATE: /terraform-cicd/AWS_CODECOMMIT_SSH_PRIVATE
I am using CDK to set up code pipelines in AWS. The pipeline stage needs to download the source code from github so uses an oauth token to authenticate the request. I would like to be able to access the token from AWS Parameter Store and NOT from AWS Secret Manager when setting the value in the stage of the pipeline.
There are plenty of examples using Secret Manager to do this. However there are no examples using the Parameter Store or hardcoding the token in plain text within the CDK project.
We are using typescript with CDK 1.3.0.
I have tried storing the token in the Parameter Store. When storing as a secure String you need to additionally specify the version when retrieving the value. However I cannot then cast to a SecretValue that is required to set oauthToken property in the pipeline stage.
Get the value from the Parameter Store ..
// get the secureString
const secureString = ssm.StringParameter.fromSecureStringParameterAttributes(construct,'MySecretParameter', {
parameterName: 'my-secure-parameter-name',
version: 1,
});
I need to cast the secretString to a CDK.SecretValue to then use it to set the oauthToken. I cannot see how to do this.
const sourceAction = new codepipelineactions.GitHubSourceAction({
actionName: 'Source',
owner: owner,
repo: repository,
oauthToken: githubOAuthAccessToken,
output: sourceOutput,
branch: branch,
trigger: codepipelineactions.GitHubTrigger.WEBHOOK,
});
The CDK documentation says that is is advisable to store tokens in Secret Manager.
"It is recommended to use a Secret Manager SecretString to obtain the token"
It does not say that tokens cannot be retrieved from other sources and used. I would be grateful if the situation could be clarified and if anyone stores tokens outside Secrets Manager and is still able to use them to set the Token in the source stage of a pipeline.
You can use cdk.SecretValue.ssmSecure or cdk.SecretValue.plainText:
oauthToken: cdk.SecretValue.ssmSecure('param-name', 'version');
// OR
oauthToken: cdk.SecretValue.plainText('oauth-token-here');
From the doc for plainText:
Do not use this method for any secrets that you care about. The only reasonable use case for using this method is when you are testing.
The previous answer by #jogold does partially work. However, at the time of this writing SecretValue.ssmSecure is not supported by Cloudformation and you will get an error such as: FAILED, SSM Secure reference is not supported in: .
There is an open issue on the CDK roadmap: https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/227. The plaintext option is not truly viable as the secret will be exposed in CFN template.
When I try to download all log files from a RDS instance, in some cases, I found this error in my python output:
An error occurred (InvalidParameterValue) when calling the
DownloadDBLogFilePortion operation: This file contains binary data and
should be downloaded instead of viewed.
I manage correctly the pagination and the throttling (using The Marker parameter and the sleep function).
This is my calling:
log_page=request_paginated(rds,DBInstanceIdentifier=id_rds,LogFileName=log,NumberOfLines=1000)
rds-> boto3 resource
And this is the definition of my function:
def request_paginated(rds,**kwargs):
return rds.download_db_log_file_portion(**kwargs)
Like I said, most of time this function works but sometime it returns:
"An error occurred (InvalidParameterValue) when calling the
DownloadDBLogFilePortion operation: This file contains binary data and
should be downloaded instead of viewed"
Can you help me please? :)
UPDATE: the problem is a known issue with downloading log files that contain non printable sign. As soon as possible I will try the proposed solution provide by the aws support
LATEST UPDATE: This is an extract of my discussion with aws support team:
There is a known issue with non binary characters when using the boto based AWS cli, however this issue is not present when using the older Java based cli.
There is currently no way to fix the issue that you are experiencing while using the boto based AWS cli, the workaround is to make the API call from the Java based cli
the aws team are aware of this issue and are working on a way to resolve this, however the do not have an ETA for when this will be released.
So the solutions is: use the java API
Giuseppe
LATEST UPDATE: This is an extract of my discussion with aws support team:
There is a known issue with non binary characters when using the boto based AWS cli, however this issue is not present when using the older Java based cli.
There is currently no way to fix the issue that you are experiencing while using the boto based AWS cli, the workaround is to make the API call from the Java based cli
the aws team are aware of this issue and are working on a way to resolve this, however the do not have an ETA for when this will be released. So the solutions is: use the java API
Giuseppe
http://docs.aws.amazon.com/AmazonRDS/latest/APIReference/CommonErrors.html
InvalidParameterValue : An invalid or out-of-range value was supplied
for the input parameter.
Invalid parameter in boto means the data pass does not complied. Probably an invalid name that you specified, possible something wrong with your variable id_rds, or maybe your LogFileName, etc. You must complied with the function arguments requirement.
response = client.download_db_log_file_portion(
DBInstanceIdentifier='string',
LogFileName='string',
Marker='string',
NumberOfLines=123
)
(UPDATE)
For example, LogFileName must be te exact file name exist inside RDS instance.
For the logfile , please make sure the log file EXISTS inside the instance. Use this AWS CLI to get a quick check
aws rds describe-db-log-files --db-instance-identifier <my-rds-name>
Do check Marker (string) and NumberOfLines (Integer) as well. Mismatch type or out of range. Skip them since they are not required, then test it later.