Is it possible to use Ref function on option_settings in AWS? - amazon-web-services

I am using Elastic Beanstalk to deploy a worker tier environment using SQS.
In my .ebextensions I have the following file:
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL:
Ref: WorkerQueue
HttpPath: "/sqs/"
InactivityTimeout: 1650
VisibilityTimeout: 1680
MaxRetries: 1
Resources:
WorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
VisibilityTimeout: 1680
However, this fails with the following error:
"option_settings" in one of the configuration files failed validation. More details to follow.
Invalid option value: 'Ref=WorkerQueue' (Namespace: 'aws:elasticbeanstalk:sqsd', OptionName: 'WorkerQueueURL'): Value does not satisfy regex: '^$|^http(s)?://.+$' [Valid non empty URL starting with http(s)]
It seems that the AWSCloudFormation Ref function cannot be used in the option_settings. Can someone confirm if this is the case?
I have seen some code snippets here on StackOverflow using intrinsic functions in the option_settings, such as in the mount-config.config of this answer and also on this question. So, are these examples using an invalid syntax? Or there are some intrinsic functions or specific resources that can be used on the option_settings?
And lastly, if I cannot use the Ref function, how can I go about this?

Yes, you can reference in .ebextentions, but the syntax is a bit strange. It is shown in the docs here.
You can try something along these lines (note the various quotations marks):
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL: '`{"Ref" : "WorkerQueue"}`'
HttpPath: "/sqs/"
InactivityTimeout: 1650
VisibilityTimeout: 1680
MaxRetries: 1
Resources:
WorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
VisibilityTimeout: 1680
You can also use ImportValue, if you export the WorkerQueue in outputs.
Update
To check the value obtained, you can set it as an env variable, and inspect in EB console:
option_settings:
aws:elasticbeanstalk:application:environment:
SQS_NAME: '`{"Ref" : "WorkerQueue"}`'

After digging further in this issue I made some discoveries I would like to share with future readers.
Ref can be used on option_settings
As #Marcin answer states, the Ref intrinsic function can be used in the option_settings. The syntax is different though:
'`{"Ref" : "ResourceName"}`'
Using Ref on aws:elasticbeanstalk:application:environment (environment variable)
An use case of the above is to store the queue URL in an environment variable, as follows:
option_settings:
aws:elasticbeanstalk:application:environment:
QUEUE_URL: '`{"Ref" : "WorkerQueue"}`'
This will let your .sh script access the URL of the queue:
Note that if you check the Elastic Beanstalk console (Environment > Config > Software), you won't see the actual value:
Using Ref on aws:elasticbeanstalk:sqsd:WorkerQueueURL
If you try to use the following setting:
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL: '`{"Ref" : "WorkerQueue"}`'
HttpPath: "/sqs/"
It will fail:
Invalid option value: '`{"Ref" : "WorkerQueue"}`' (Namespace: 'aws:elasticbeanstalk:sqsd', OptionName: 'WorkerQueueURL'): Value does not satisfy regex: '^$|^http(s)?://.+$' [Valid non empty URL starting with http(s)]
It seems that this configuration option don't accept a reference.
Instead of creating a new queue and assign it to the sqs daemon, you can just update the queue that Elastic Beanstalk creates:
option_settings:
# SQS daemon will use default queue created by EB (AWSEBWorkerQueue)
aws:elasticbeanstalk:sqsd:
HttpPath: "/sqs/"
Resources:
# Update the queue created by EB
AWSEBWorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"

Related

Yaml validation error on TaskDefinition preventing cloudformation from reaching completion

I'm trying to create a cloudformation for a Task Definition but I keep running into validation errors on the command portion of the Task Definition. I've tried a few approaches and can't seem to get it right.
I've been following the documentation here.
Below are 2 of my main attempts and their respective errors:
Command: !Sub
- "--config config.yaml --env ${environment}"
- { environment: !FindInMap [EnvironmentVariables, !Ref Environment, Environment] }
Environment] }
Error:
#/ContainerDefinitions/0/Command: expected type: JSONArray, found: String)
Command: !Sub
- [--config, config.yaml, --env, ${environment}]
- { environment: !FindInMap [EnvironmentVariables, !Ref Environment, Environment] }
Template format error: YAML not well-formed.
There seems to be some issue with how !Sub interacts with a Yaml list but I'm not exactly sure what causes this to be the case.
However the below piece of Yaml managed to solve my problem.
Command:
- --config
- config.yaml
- --env
- !FindInMap [EnvironmentVariables, !Ref Environment, Environment]

SSM Automation Run Command is not longer than default 3600 seconds

I have been working with AWS Systems Manager and I have created a Document to run a command, But it appears there is no way to overwrite the timeout for a run command in an SSM
I have changed the execution timeout here in the parameters but does not work.
also, I added a timeoutSeconds in my Document and it doesn't work either.
This is my Document (I'm using schema version 2.2):
schemaVersion: "2.2"
description: "Runs a Python command"
parameters:
Params:
type: "String"
description: "Params after the python3 keyword."
mainSteps:
- action: "aws:runShellScript"
name: "Python3"
inputs:
timeoutSeconds: '300000'
runCommand:
- "sudo /usr/bin/python3 /opt/python/current/app/{{Params}}"
1: The setting that’s displayed in your screenshot in the Other parameters section is the Delivery Timeout, which is different from the execution timeout.
You must specify the execution timeout value in the Execution Timeout field, if available. Not all SSM documents require that you specify an execution timeout. If a Systems Manager document doesn't require that you explicitly specify an execution timeout value, then Systems Manager enforces the hard-coded default execution timeout.
2: In your document, the timeoutSeconds attribute is in the wrong place. It needs to be on the same level as the action.
...
mainSteps:
- action: "aws:runShellScript"
timeoutSeconds: 300000
name: "Python3"
inputs:
runCommand:
- "sudo /usr/bin/python3 /opt/python/current/app/{{Params}}"
timeoutSeconds: '300000'
Isn't this string but not integer?

Configuring Concourse CI to use AWS Secrets Manager

I have been trying to figure out how to configure the docker version of Concourse (https://github.com/concourse/concourse-docker) to use the AWS Secrets Manager and I added the following environment variables into the docker-compose file but from the logs it doesn't look like it ever reaches out to AWS to fetch the creds. Am I missing something or should this automatically happen when adding these environment variables under environment in the docker-compose file? Here are the docs I have been looking at https://concourse-ci.org/aws-asm-credential-manager.html
version: '3'
services:
concourse-db:
image: postgres
environment:
POSTGRES_DB: concourse
POSTGRES_PASSWORD: concourse_pass
POSTGRES_USER: concourse_user
PGDATA: /database
concourse:
image: concourse/concourse
command: quickstart
privileged: true
depends_on: [concourse-db]
ports: ["9090:8080"]
environment:
CONCOURSE_POSTGRES_HOST: concourse-db
CONCOURSE_POSTGRES_USER: concourse_user
CONCOURSE_POSTGRES_PASSWORD: concourse_pass
CONCOURSE_POSTGRES_DATABASE: concourse
CONCOURSE_EXTERNAL_URL: http://XXX.XXX.XXX.XXX:9090
CONCOURSE_ADD_LOCAL_USER: test: test
CONCOURSE_MAIN_TEAM_LOCAL_USER: test
CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
CONCOURSE_AWS_SECRETSMANAGER_REGION: us-east-1
CONCOURSE_AWS_SECRETSMANAGER_ACCESS_KEY: <XXXX>
CONCOURSE_AWS_SECRETSMANAGER_SECRET_KEY: <XXXX>
CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}
pipeline.yml example:
jobs:
- name: build-ui
plan:
- get: web-ui
trigger: true
- get: resource-ui
- task: build-task
file: web-ui/ci/build/task.yml
- put: resource-ui
params:
repository: updated-ui
force: true
- task: e2e-task
file: web-ui/ci/e2e/task.yml
params:
UI_USERNAME: ((ui-username))
UI_PASSWORD: ((ui-password))
resources:
- name: cf
type: cf-cli-resource
source:
api: https://api.run.pivotal.io
username: ((cf-username))
password: ((cf-password))
org: Blah
- name: web-ui
type: git
source:
uri: git#github.com:blah/blah.git
branch: master
private_key: ((git-private-key))
When storing parameters for concourse pipelines in AWS Secrets Manager, it must follow this syntax,
/concourse/TEAM_NAME/PIPELINE_NAME/PARAMETER_NAME`
If you have common parameters that are used across the team in multiple pipelines, use this syntax to avoid creating redundant parameters in secrets manager
/concourse/TEAM_NAME/PARAMETER_NAME
The highest level that is supported is concourse team level.
Global parameters are not possible. Thus these variables in your compose environment will not be supported.
CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}
Unless you want to change the prefix /concourse, these parameters shall be left to their defaults.
And, when retrieving these parameters in the pipeline, no changes required in the template. Just pass the PARAMETER_NAME, concourse will handle the lookup in secrets manager as per the team and pipeline name.
...
params:
UI_USERNAME: ((ui-username))
UI_PASSWORD: ((ui-password))
...

can we pass parameters to appsec.yml hook ApplicationStart phase?

I want to deploy my application to aws, I am using codeDeploy for this.
following is my appsec.yml file:
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/project
permissions:
- object: /home/ubuntu/project
owner: root
mode: 777
type:
- directory
hooks:
BeforeInstall:
- location: scripts/install_dependencies.sh
timeout: 900
runas: root
AfterInstall:
- location: ./scripts/after-install.sh
timeout: 900
ApplicationStart:
- location: ./scripts/application-start.sh parameter1 parameter2
timeout: 900
ValidateService:
- location: ./scripts/validate-service.sh
timeout: 900
I am not able to pass paramerters to scripts.
Currently this is not possible.
As a workaround, you can design your Hook Scripts to consume System Environment Variables which can be defined on a instance at launch (through user-data) or you can also retrieve the parameters from AWS SSM Parameter Store (specially if they are secrets) using AWS CLI:
https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-cli.html
You can create an if clause using the predefined environment variables. Based on the situation you configure the required values predefined environment variables

AWS Elastic Beanstalk: How to use environment variables in ebextensions?

We are trying to store environment specific application configuration files in s3.
The files are stored in different subdirectories which are named after the environment and also have the environment as part of the file name.
Examples are
dev/application-dev.properties
stg/application-stg.properties
prd/application-prd.properties
The Elastic Beanstalk environments are named dev, stg, prd and alternatively I also have an environment variable defined in Elastic Beanstalk named ENVIRONMENT which can be dev, stg or prd.
My question now is, how do I reference the environment name or ENVIRONMENT variable when downloading the configuration file from a config file in .ebextensions?
I tried using a {"Ref": "AWSEBEnvironmentName" } reference in .ebextensions/myapp.config but get a syntax error when deploying.
The content of .ebextensions/myapp.config is:
files:
/config/application-`{"Ref": "AWSEBEnvironmentName" }`.properties:
mode: "000666"
owner: webapp
group: webapp
source: https://s3.amazonaws.com/com.mycompany.mybucket/`{"Ref": "AWSEBEnvironmentName" }`/application-`{"Ref": "AWSEBEnvironmentName" }`.properties
authentication: S3Access
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Access:
type: S3
roleName: aws-elasticbeanstalk-ec2-role
buckets: com.mycompany.api.config
The error I get is:
The configuration file .ebextensions/myapp.config in application version
manualtest-18 contains invalid YAML or JSON. YAML exception: Invalid Yaml:
mapping values are not allowed here in "<reader>", line 6, column 85:
... .config/stg/application-`{"Ref": "AWSEBEnvironmentName" }`.prop ... ^ ,
JSON exception: Invalid JSON: Unexpected character (f) at position 0..
Update the configuration file.
What is the correct way of referencing an environment variable in a .ebextensions config file in AWS Elastic Beanstalk?
Your .ebextensions config file was almost correct. Substituting the file name with environment variable or AWS resource name won't work, for that do as in Mark's answer to rename the file created in container_commands section.
The source option value trying to access AWS resource name using Ref was correct, it just had to be surrounded by single quote ', like below:
files:
/config/application.properties:
mode: "000666"
owner: webapp
group: webapp
source: 'https://s3.amazonaws.com/com.mycompany.mybucket/`{"Ref": "AWSEBEnvironmentName" }`/application-`{"Ref": "AWSEBEnvironmentName" }`.properties'
authentication: S3Access
And to access environment variables use Fn::GetOptionSetting. Environment variables are in aws:elasticbeanstalk:application:environment namespace.
Below example access an environment variable ENVIRONMENT in source option of files:
files:
"/tmp/application.properties" :
mode: "000666"
owner: webapp
group: webapp
source: 'https://s3.amazonaws.com/com.mycompany.mybucket/`{"Ref": "AWSEBEnvironmentName" }`/application-`{"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "ENVIRONMENT ", "DefaultValue": "dev"}}`.properties'
authentication: S3Auth
I struggled to get this working, until I discovered that the Sub function doesn't appear to be available in ebextensions: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions-functions.html
This means that you need to fall back to Fn::Join and Ref, at least until support for Sub is introduced to ebextensions. It also seems that the files attribute requires a fixed path (and I couldn't use Fn::Join in this context).
My overall solution to this was as follows:
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Auth:
type: S3
buckets: arn:aws:s3:::elasticbeanstalk-xxx
roleName: aws-elasticbeanstalk-ec2-role
files:
"/tmp/application.properties" :
mode: "000644"
owner: root
group: root
source: { "Fn::Join" : ["", ["https://s3-xxx.amazonaws.com/elasticbeanstalk-xxx/path/to/application-", { "Ref" : "AWSEBEnvironmentName" }, ".properties" ]]}
authentication: S3Auth
container_commands:
01-apply-configuration:
command: mkdir -p config && mv /tmp/application.properties config
This will result in an application.properties file (without the environment name qualifier) in a config directory next to the deployed application instance.
If you want to keep the name of the environment as part of the file name using this approach, you will need to adjust the command that moves the file to use another Fn::Join expression to control the filename.
You are almost there .ebextensions are using YAML format, while your trying to use JSON. Use Ref: AWSEBEnvironmentName.
In addition, you can take advantage of Sub function to avoid pesky Join:
!Sub "/config/application-${AWSEBEnvironmentName}.properties"