I defined an environment variable called MY_ENVIRONMENT_VARIABLE in AWS Elastic Beanstalk's Software Configuration tab.
Now I would like to use this environment variable in the "files:" section of a .ebextensions config file.
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Auth:
type: S3
buckets: arn:aws:s3:::SomeS3Bucket
roleName: aws-elasticbeanstalk-ec2-role
files:
"/tmp/application.properties" :
mode: "000644"
owner: root
group: root
source: { "Ref" : "MY_ENVIRONMENT_VARIABLE" }
authentication: S3Auth
container_commands:
01-apply-configuration:
command: mv /tmp/application.properties .
It seems to be possible to reference environment variables in the "container_commands:" section (by using bash scripts) but I couldn't find any references that it is possible inside the "files:" section.
Does anybody have an example of how to use environment variables inside the "files:" section?
Use Fn::GetOptionSetting to retreive environment variables. Environment variables are in aws:elasticbeanstalk:application:environment namespace
files:
"/tmp/application.properties" :
mode: "000644"
owner: root
group: root
source: '`{"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "MY_ENVIRONMENT_VARIABLE", "DefaultValue": "file_path"}}`'
authentication: S3Auth
Note the backtick which perform the command substitution. DefaultValue attribute is optional, which is used in case environment variable is not found.
Above config file will create file /tmp/application.properties with content from the file referenced in environment variable MY_ENVIRONMENT_VARIABLE.
Related
I have an AWS Elasticbeanstalk Python environment and I'd like to change default 1 MB nginx configuration limit for file upload to a bigger value (15 MB).
So I thought to add a config file in .ebextensions:
files:
"/tmp/my.nginx.conf" :
mode: "000755"
owner: root
group: root
content: |
client_max_body_size 15M;
"/tmp/install-nginx-config.sh" :
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
cp /tmp/my.nginx.conf /etc/nginx/conf.d/elasticbeanstalk/01_max-size.conf
container_commands:
01_runmyshellscript:
command: "sudo /tmp/install-nginx-config.sh"
02_reload_nginx:
command: "sudo service nginx reload"
The idea is to add the new config file in /etc/nginx/conf.d/elasticbeanstalk after the elasticbeanstalk nginx config files are created, so to no to to interfere with the creation of the EB environment, like suggested in many articles on Internet.
But if I deploy the above file in .ebextensions then the environment fails and if I SSH the EC2 I find out that in /etc/nginx/conf.d/elasticbeanstalk there is my 01_max-size.conf but the nginx configuration file created by Elasticbeanstalk isn't there anymore.
This is strange, beacause container_commands should be exetuced after the end of environment creation, so I have no clue of how solve this issue.
I am using Elastic Beanstalk to deploy a worker tier environment using SQS.
In my .ebextensions I have the following file:
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL:
Ref: WorkerQueue
HttpPath: "/sqs/"
InactivityTimeout: 1650
VisibilityTimeout: 1680
MaxRetries: 1
Resources:
WorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
VisibilityTimeout: 1680
However, this fails with the following error:
"option_settings" in one of the configuration files failed validation. More details to follow.
Invalid option value: 'Ref=WorkerQueue' (Namespace: 'aws:elasticbeanstalk:sqsd', OptionName: 'WorkerQueueURL'): Value does not satisfy regex: '^$|^http(s)?://.+$' [Valid non empty URL starting with http(s)]
It seems that the AWSCloudFormation Ref function cannot be used in the option_settings. Can someone confirm if this is the case?
I have seen some code snippets here on StackOverflow using intrinsic functions in the option_settings, such as in the mount-config.config of this answer and also on this question. So, are these examples using an invalid syntax? Or there are some intrinsic functions or specific resources that can be used on the option_settings?
And lastly, if I cannot use the Ref function, how can I go about this?
Yes, you can reference in .ebextentions, but the syntax is a bit strange. It is shown in the docs here.
You can try something along these lines (note the various quotations marks):
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL: '`{"Ref" : "WorkerQueue"}`'
HttpPath: "/sqs/"
InactivityTimeout: 1650
VisibilityTimeout: 1680
MaxRetries: 1
Resources:
WorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
VisibilityTimeout: 1680
You can also use ImportValue, if you export the WorkerQueue in outputs.
Update
To check the value obtained, you can set it as an env variable, and inspect in EB console:
option_settings:
aws:elasticbeanstalk:application:environment:
SQS_NAME: '`{"Ref" : "WorkerQueue"}`'
After digging further in this issue I made some discoveries I would like to share with future readers.
Ref can be used on option_settings
As #Marcin answer states, the Ref intrinsic function can be used in the option_settings. The syntax is different though:
'`{"Ref" : "ResourceName"}`'
Using Ref on aws:elasticbeanstalk:application:environment (environment variable)
An use case of the above is to store the queue URL in an environment variable, as follows:
option_settings:
aws:elasticbeanstalk:application:environment:
QUEUE_URL: '`{"Ref" : "WorkerQueue"}`'
This will let your .sh script access the URL of the queue:
Note that if you check the Elastic Beanstalk console (Environment > Config > Software), you won't see the actual value:
Using Ref on aws:elasticbeanstalk:sqsd:WorkerQueueURL
If you try to use the following setting:
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL: '`{"Ref" : "WorkerQueue"}`'
HttpPath: "/sqs/"
It will fail:
Invalid option value: '`{"Ref" : "WorkerQueue"}`' (Namespace: 'aws:elasticbeanstalk:sqsd', OptionName: 'WorkerQueueURL'): Value does not satisfy regex: '^$|^http(s)?://.+$' [Valid non empty URL starting with http(s)]
It seems that this configuration option don't accept a reference.
Instead of creating a new queue and assign it to the sqs daemon, you can just update the queue that Elastic Beanstalk creates:
option_settings:
# SQS daemon will use default queue created by EB (AWSEBWorkerQueue)
aws:elasticbeanstalk:sqsd:
HttpPath: "/sqs/"
Resources:
# Update the queue created by EB
AWSEBWorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
In my docker-compose.yml file I can do the following:
splash:
image: scrapinghub/splash
command: --max-timeout 300
ports:
- "8050:8050"
As you can see, I just pass in the additional options that I want to add to the docker run command that is executed in the image.
When I try to apply this to my Dockerrun.aws.json file that is deployed to Amazon Elastic Beanstalk, I get error when I write the following:
{
"name": "splash",
"image": "scrapinghub/splash",
...
"command": [
"--max-timeout 300"
]
},
So question is, how do I add the --max-timeout parameter to the default command that my Docker Image is executing with my AWS deployment?
You cant customize how AWS start a container by playing with DOCKER RUN options. You have to use .ebextensions .
If you want to increase timeout, create a file in .ebextensions sub directory of your ZIP package (it should already contains the Dockerrun.aws.json file) :
option_settings:
- namespace: aws:elb:policies
option_name: ConnectionSettingIdleTimeout
value: 300
- namespace: aws:elasticbeanstalk:command
option_name: Timeout
value: 300
- namespace: aws:elbv2:loadbalancer
option_name: IdleTimeout
value: 300
I have a project built in Golang and deployed on a Docker instance in AWS.
Internally I create a log file where the program write several logs.
How can I access that log file?
Is there another correct way to logging?
Thanks
You could mount the log file from your container to your EC2 host. You can do this by using the -v flag when running your container:
docker run -v /var/log/my_host_log_file.log:/var/log/your_container_log_file.log your-image
Alternatively, you can configure your app to log to stdout and use syslog as your log driver (using the --log-driver=syslog switch). Your container logs will then be written to /var/log/messages on your host.
If you use AWS, i would suggest to send Logs direct to AWS CloudWatch.
First create a new Log-Group in AWS Cloudwatch, for example "Production". In your Docker-Compose.yml (or via docker run..) add the AWS Logdriver:
logging:
driver: "awslogs"
options:
awslogs-region: "eu-central-1"
awslogs-group: "Production"
awslogs-stream: "MyApp"
Next creat a IAM user with Access to AWS Cloudwatch and add to the Dockerhost the credentials.
Example IAM Policy:
"Version" "2012-10-17"
"Statement"
"Action" "logs:CreateLogStream" "logs:PutLogEvents" "Effect" "Allow" "Resource"
On Ubuntu with systemd:
"Version" "2012-10-17"
"Statement"
"Action"
"logs:CreateLogStream"
"logs:PutLogEvents"
"Effect"
"Allow" "Resource"
And add to the File:
[Service] Environment"AWS_ACCESS_KEY_ID=<aws_access_key_id>"
Environment"AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>"
Run:
systemctl daemon-reload
service docker restart
Now your logs should appear in AWS Cloudwatch.
Thanks for reply.
After a while looking for the solution to the problem, I found it!
Firstly, I needed to mount the file that is inside the instance in the docker-host.
To do this I add a Json file in the root folder of my project called Dockerrun.aws.json
( http://docs.aws.amazon.com/es_es/elasticbeanstalk/latest/dg/create_deploy_docker_image.html#create_deploy_docker_image_dockerrun )
That is the file that declares the shared folder (volumes) (beetwen docker-host and instance) where I save my log file . This line is equivalent to adding -v flag in the docker run command (https://docs.docker.com/engine/tutorials/dockervolumes/#mount-a-host-directory-as-data-volume). I do this this way because I can not add mount to a running instance and i cant stop it by ssh.
{
"AWSEBDockerrunVersion": "1",
"Volumes": [
{
"HostDirectory": "/var/log/",
"ContainerDirectory": "/go/src/app/log"
}
]
}
Then to tell aws that I want to download my log file when I request records. (Tail (last 100 lines), bundle or rotate) I add these files to the .ebextension folder in my project directory. ( http://docs.aws.amazon.com/en_us/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-extend )
Log_bundle.conf
Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_bundle.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Log_rotate.config
Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_rotate.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Log_tail.config
Files:
"/opt/elasticbeanstalk/tasks/publishlogs.d/log_tail.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Finally, I dont try Amazon Could Watch but is the next step.
Regards
If you use ELK (Elasticsearch, Logstash, Kibana), I would suggest to use "logrus"
Get the library
go get github.com/sirupsen/logrus
Then in your project
package main
import (
logrus "github.com/sirupsen/logrus"
)
var log = logrus.New()
func main() {
conn, _ := net.Dial("tcp", "logstash-address")
hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "my-app"}))
log.Hooks.Add(hook)
log.Info("Hello World!")
}
We are trying to store environment specific application configuration files in s3.
The files are stored in different subdirectories which are named after the environment and also have the environment as part of the file name.
Examples are
dev/application-dev.properties
stg/application-stg.properties
prd/application-prd.properties
The Elastic Beanstalk environments are named dev, stg, prd and alternatively I also have an environment variable defined in Elastic Beanstalk named ENVIRONMENT which can be dev, stg or prd.
My question now is, how do I reference the environment name or ENVIRONMENT variable when downloading the configuration file from a config file in .ebextensions?
I tried using a {"Ref": "AWSEBEnvironmentName" } reference in .ebextensions/myapp.config but get a syntax error when deploying.
The content of .ebextensions/myapp.config is:
files:
/config/application-`{"Ref": "AWSEBEnvironmentName" }`.properties:
mode: "000666"
owner: webapp
group: webapp
source: https://s3.amazonaws.com/com.mycompany.mybucket/`{"Ref": "AWSEBEnvironmentName" }`/application-`{"Ref": "AWSEBEnvironmentName" }`.properties
authentication: S3Access
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Access:
type: S3
roleName: aws-elasticbeanstalk-ec2-role
buckets: com.mycompany.api.config
The error I get is:
The configuration file .ebextensions/myapp.config in application version
manualtest-18 contains invalid YAML or JSON. YAML exception: Invalid Yaml:
mapping values are not allowed here in "<reader>", line 6, column 85:
... .config/stg/application-`{"Ref": "AWSEBEnvironmentName" }`.prop ... ^ ,
JSON exception: Invalid JSON: Unexpected character (f) at position 0..
Update the configuration file.
What is the correct way of referencing an environment variable in a .ebextensions config file in AWS Elastic Beanstalk?
Your .ebextensions config file was almost correct. Substituting the file name with environment variable or AWS resource name won't work, for that do as in Mark's answer to rename the file created in container_commands section.
The source option value trying to access AWS resource name using Ref was correct, it just had to be surrounded by single quote ', like below:
files:
/config/application.properties:
mode: "000666"
owner: webapp
group: webapp
source: 'https://s3.amazonaws.com/com.mycompany.mybucket/`{"Ref": "AWSEBEnvironmentName" }`/application-`{"Ref": "AWSEBEnvironmentName" }`.properties'
authentication: S3Access
And to access environment variables use Fn::GetOptionSetting. Environment variables are in aws:elasticbeanstalk:application:environment namespace.
Below example access an environment variable ENVIRONMENT in source option of files:
files:
"/tmp/application.properties" :
mode: "000666"
owner: webapp
group: webapp
source: 'https://s3.amazonaws.com/com.mycompany.mybucket/`{"Ref": "AWSEBEnvironmentName" }`/application-`{"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "ENVIRONMENT ", "DefaultValue": "dev"}}`.properties'
authentication: S3Auth
I struggled to get this working, until I discovered that the Sub function doesn't appear to be available in ebextensions: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions-functions.html
This means that you need to fall back to Fn::Join and Ref, at least until support for Sub is introduced to ebextensions. It also seems that the files attribute requires a fixed path (and I couldn't use Fn::Join in this context).
My overall solution to this was as follows:
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Auth:
type: S3
buckets: arn:aws:s3:::elasticbeanstalk-xxx
roleName: aws-elasticbeanstalk-ec2-role
files:
"/tmp/application.properties" :
mode: "000644"
owner: root
group: root
source: { "Fn::Join" : ["", ["https://s3-xxx.amazonaws.com/elasticbeanstalk-xxx/path/to/application-", { "Ref" : "AWSEBEnvironmentName" }, ".properties" ]]}
authentication: S3Auth
container_commands:
01-apply-configuration:
command: mkdir -p config && mv /tmp/application.properties config
This will result in an application.properties file (without the environment name qualifier) in a config directory next to the deployed application instance.
If you want to keep the name of the environment as part of the file name using this approach, you will need to adjust the command that moves the file to use another Fn::Join expression to control the filename.
You are almost there .ebextensions are using YAML format, while your trying to use JSON. Use Ref: AWSEBEnvironmentName.
In addition, you can take advantage of Sub function to avoid pesky Join:
!Sub "/config/application-${AWSEBEnvironmentName}.properties"