Below is the setup of the application which runs a docker container into elasticbeanstalk.
step 1:
created the parent folder say apptest and inside that I have placed the Dockerfile, package.json and small hello world server.js node app.
step 2:
inside the parent folder apptest ran the command eb init which crated a hidden folder .elasticbeanstalk with config.yml comprising default settings.
step 3:
added .ebextensions with a config file 01_run.config, comprising the below configuration to update the instance type.
option_settings:
aws:autoscaling:launchconfiguration :
InstanceType : "m4.xlarge"
Note that till now no environment is created yet. And since I have extensions created, should override the defaults with the instance type pointed to m4.xlarge.
step 4:
Now ran the command eb create apptest-dev ( say for example ) and created the environment
Problem:
when the environment is created, it did not have the m4.xlarge, rather got created with default instance type as t2.micro. But when uploaded the zipped version of this folder contents into the environment from console ( excluding the .elasticbeanstalk folder ), ebextensions folder configuration is picked up. Its only with the option_settings not getting worked, but the other resources like files, commands are getting reflected both from command line and from the file upload.
I feel its kind of very small thing missing which I am not able to figure it out from blogs and documentation. Thanks for the help in advance.
During eb create, the EBCLI passes it's own defaults for many of the option settings, among which is the instance type. Since, the EBCLI does not parse .ebextensions, and the Beanstalk service prefers the defaults passed by the EBCLI, the instance type specified in your .ebextensions are disregarded.
There are two ways to get around this:
call eb config after eb create. In the interactive mode, change the instance type, and save and exit.
call eb create as eb create -i m4.xlarge
Related
I'm facing a similar issue to this question while trying to implement the new hooks logic on a AWS Linux 2 managed platform running Docker.
I have created my file inside this .platform/hooks/postdeploy/configure_nginx.sh
which is living in the src/ folder of my app.
I can see the file in my host after the deploy but it's never executed /var/app/current/src/.platform/hooks/postdeploy/configure_nginx.sh
It has the right privileges and I can run it if I ssh into my instance.
The EBS environment is initialized through a docker-compose file where I start 2 public containers and my custom app (in which I have put the hook folders)
The script is not being executed so I'm a bit lost on where I need to put it. FYI, my eb deploy is simply copying a Dockerrun file which is grabbing an image from one of my ECR so basically nothing is done in there.
Thanks for your help!
[SOLUTION]
I've found how it should work.
You have to create the .platform folder at the same level as the .ebextensions one.
In my case I'm deploying a zip archive simply containing my Dockerrun.aws.json and both .ebextensions and .platform folders.
So remember to zip it with the whole package before deploying it to your EBS environment.
On Elastic Beanstalk, with an AWS Linux 2 based environment, updating the Environment Properties (i.e. environment variables) of an environment causes all generated files to be deleted. It also doesn't run container_commands as part of this update.
So, for example, I have a Django project with collectstatic in the container commands:
05_collectstatic:
command: |
source $PYTHONPATH/activate
python manage.py collectstatic --noinput --ignore *.scss
This collects static files to a folder called staticfiles as part of deploy. But when I do an environment variable update, staticfiles is deleted. This causes all static files on the application to be broken until I re-deploy, which is extremely undesirable.
This behavior did not occur on AWS Linux 1 based environments. The difference appears to be that AWS Linux 2 based environments replace the /var/app/current folder during environment variable changes, where AWS Linux 1 based environments did not do this.
How do I fix this?
Research
I can verify that the container commands are not being run during an environment variable change by monitoring /var/log/cfn-init.log; no new entries are added to this log.
This happens with both rolling update type "disabled" and "immutable".
This happens even if I convert the environment command to be a platform hook, despite the fact that hooks are listed as running when environment properties are updated.
It seems to me like there are two potential solutions, but I don't know of an Elastic Beanstalk setting for either:
Have environment variable changes leave /var/app/current rather than replacing it.
Have environment variable changes run container commands.
The Elastic Beanstalk docs on container commands say "Leader-only container commands are only executed during environment creation and deployments, while other commands and server customization operations are performed every time an instance is provisioned or updated." Is this a bug in Elastic Beanstalk?
Related question: EB: Trigger container commands / deploy scripts on configuration change
The solution is to use a Configuration deployment platform hook for any commands that change the files in the deployment directory. Note that this is different from an Application deployment platform hook.
Using the example of the collectstatic command, the best thing to do is to move it from a container command to a pair of hooks, one for standard deployments and one for configuration changes.
To do this, remove the collectstatic container command. Then, make two identical files:
.platform/confighooks/predeploy/predeploy.sh
.platform/hooks/predeploy/predeploy.sh
Each file should have the following code:
#!/bin/bash
source $PYTHONPATH/activate
python manage.py collectstatic --noinput --ignore *.scss
You need two seemingly redundant files because different hooks have different trigger conditions. Scripts in hooks run when you deploy the app whereas scripts in confighooks run when you change the configuration of the app.
Make sure to make both of these files executable according to git or else you will run into a "permission denied" error when you try to deploy. You can check if they are executable via git ls-files -s .platform; you should see 100755 before any shell files in the output of this command. If you see 100644 before any of your shell files, run git add --chmod=+x -- .platform/*/*/*.sh to make them executable.
Background
Our AWS Elastic Beanstalk environment, running the latest version of the pre-configured "Python 3.7 on 64-bit Amazon Linux 2" platform branch, has a lot of custom configuration and environment properties.
Now we would like to switch this environment to the "Python 3.8 on 64-bit Amazon Linux 2" platform branch.
Basically, the goal is to clone the environment, keeping the current configuration (other than platform branch and version) and environment properties.
Problem
Unfortunately, when cloning, it is not possible to switch between different platform branches (we can only switch between platform versions within the same platform branch).
The documentation suggests that a blue/green deployment is required here. However, a blue/green deployment involves creating a new environment from scratch, so we would still need some other way to copy our configuration settings and environment properties.
Question
What would be the recommended way to copy the configuration settings and/or environment properties from the original environment into a newly created environment?
I suppose we could use eb config to download the original configuration, modify the environment name, platform branch and version, and so on, and then use eb config --update on the new environment. However, that feels like a hack.
Summary
save current config: eb config save <env name>
use a text editor to modify the platform branch in the saved config file
create new environment based on modified config file: eb create --cfg <config name> (add --sample to use the sample application)
if necessary, delete local config files
if necessary, use eb printenv and eb setenv to copy environment properties
EDIT: For some reason the saved config does not include all security group settings, so it may be necessary to check those manually, using the EB console (configuration->instances).
Background
AWS support have confirmed that using eb config is the way to go, and they referred to the online documentation for details.
Unfortunately, the documentation for the eb cli does not provide all the answers.
The following is based on my own adventures using the latest version of the eb cli (3.20.2) with botocore 1.21.50, and documentation at the time of writing (Sep 30, 2021). Note there's a documentation repo on github but it was last updated six months ago and does not match the latest online docs...
eb config
Here's a screenshot from the eb config docs:
Indeed, if you call eb config my-env or eb config my-env --display, environment properties are not shown.
However, this does not hold for eb config save: YAML files created using eb config save actually do include environment properties*.
*Beware, if your environment properties include secrets (e.g. passwords), these also end up in your saved configs, so make sure you don't commit those to version control.
Moreover, it is currently also possible to set environment properties using eb config --update.
This implies we should be able to "copy" both configuration settings and environment properties in one go.
EDIT: After some testing it turns out eb config save does not always get the complete set of environment properties: some properties may be skipped. Not yet sure why... Step 5 below might help in those cases.
Walk-through
Not sure if this is the best way to do it, but here's what seems to work for me:
Suppose we have an existing EB environment called py37-env with lots of custom configuration and properties, running the Python 3.7 platform branch.
The simplest way to "clone" this would be as follows:
Step 1: download the existing configuration
Download the configuration for the existing environment:
eb config save py37-env
By default, the config file will end up in our project directory as .elasticbeanstalk/saved_configs/py37-env-sc.cfg.yml.
The saved config file could look like this (just an example, also see environment manifest):
EnvironmentConfigurationMetadata:
Description: Configuration created from the EB CLI using "eb config save".
DateCreated: '1632989892000'
DateModified: '1632989892000'
Platform:
PlatformArn: arn:aws:elasticbeanstalk:eu-west-1::platform/Python 3.7 running on 64bit Amazon Linux 2/3.3.5
OptionSettings:
aws:elasticbeanstalk:application:environment:
MY_ENVIRONMENT_PROPERTY: myvalue
aws:elasticbeanstalk:command:
BatchSize: '30'
BatchSizeType: Percentage
aws:elb:policies:
ConnectionDrainingEnabled: true
aws:elb:loadbalancer:
CrossZone: true
aws:elasticbeanstalk:environment:
ServiceRole: aws-elasticbeanstalk-service-role
aws:elasticbeanstalk:healthreporting:system:
SystemType: enhanced
aws:autoscaling:launchconfiguration:
IamInstanceProfile: aws-elasticbeanstalk-ec2-role
EC2KeyName: my-key
aws:autoscaling:updatepolicy:rollingupdate:
RollingUpdateType: Health
RollingUpdateEnabled: true
EnvironmentTier:
Type: Standard
Name: WebServer
AWSConfigurationTemplateVersion: 1.1.0.0
Also see the list of available configuration options in the documentation.
Step 2: modify the saved configuration
We are only interested in the Platform, so it is sufficient here to replace 3.7 by 3.8 in the PlatformArn value.
If necessary, you can use e.g. eb platform list to get an overview of valid platform names.
Step 3: create a new environment based on the modified config file
eb create --cfg py37-env-sc
This will deploy the most recent application version. Use --version <my version> to deploy a specific version, or use --sample to deploy the sample application, as described in the docs.
This will automatically look for files in the default saved config folder, .elasticbeanstalk/saved_configs/.
If you get a ServiceError or InvalidParameterValueError at this point, make sure only to pass in the name of the file, i.e. without the file extension .cfg.yml and without the folders.
Step 4: clean up local saved configuration file
Just in case you have any secrets stored in the environment properties.
Step 5: alternative method for copying environment properties
If environment properties are not included in the saved config files, or if some of them are missing, here's an alternative way to copy them (using bash).
This might not be the most efficient implementation, but I think it serves to illustrate the approach. Error handling was omitted, for clarity.
source_env="py37-env" # or "$1"
target_env="py38-env" # or "$2"
# get the properties from the source environment
source_env_properties="$(eb printenv "$source_env")"
# format the output so it can be used with `eb setenv`
mapfile -t arg_array < <(echo "$source_env_properties" | grep "=" | sed -e 's/ =/=/g' -e 's/= /=/g' -e 's/^ *//g')
# copy the properties to the target environment
eb setenv -e "$target_env" "${arg_array[#]}"
This has the advantage that it does not store any secrets in local files.
How to call a custom configuration file per environment when using php's composer and amazon's elastic beanstalk?
for example, when having a "composer.dev.json" & "composer.master.json" files (one of them could request files from a "dev" version/branch and the other one for stable files).
Elastic Beanstalk has composer built-in. In order to use it, the only thing you need to do is adding a "composer.json" file in your root folder. In case you want to use a custom composer file for each version, it can also be accomplished relatively easily. Create a folder named ".elasticbeanstalk" and its relative file "optionsettings.branch-name" - it should be created using the "eb cli", particularity, with the commands "eb init" and "eb branch"- more info can be found here http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cmd-commands.html.
After we have the optionsettings file, we should define a param named "COMPOSER" in there and just have it to point to the custom composer file we would like to run, for example:
[aws:elasticbeanstalk:application:environment]
COMPOSER=composer.dev.json
on your next deploment, it should run the composer using composer.dev.json
While trying to setup an elastic beanstalk worker application using the command line tools (eb tools), my configuration file (optionsettings.MyApp-env) gets overwritten when I start/update/stop the environment.
These are the steps to reproduce:
Using the CLI tools' command eb init I've created a new application in Elastic Beanstalk.
The config file in the .elasticbeanstalk folder had the following line:
OptionSettingFile=/Users/doron/projects/workers/my-worker/.elasticbeanstalk/optionsettings.MyWorkerName-dev
After running eb start for the first time, that file was created with some values.
I went ahead and changed its contents according to http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html so it'll be configured as I want (environment parameters, autoscaling servers amount, etc...).
To apply the changes I've tried the following:
Update the existing environment with eb update.
Terminate the existing environment with eb stop and build it from scratch with eb start.
In both cases the optionsettings file get changed after running the command (update or start).
The new content of the file looks more like the vanilla version I got after calling the first eb start with all sorts of configuration parameters that I added - removed completely.
Is there another way of configuring the environment (not the software on the machine, but the configuration that exists in the console - instance type, regions, autoscaling, rotating updates, etc...) ?
I realise that this is an old thread but in case anyone comes across this, as I did, then check out this thread on the AWS forums for Elasticbeanstalk https://forums.aws.amazon.com/thread.jspa?messageID=395052#395052
It explains how settings set in the .elasticbeanstalk/optionsettings. file are set using the API in such a way that they can't be changed later, unlike those set in the .ebextensions/*.config files.
Also, in an incredibly annoying move, the optionsettings file will often set some settings in it which you want to set in the .config file however it automatically re-creates the optionsettings file when running eb start and there's very little that seems possible. This makes using the eb command line tools close to impossible to use if you want to change something like the WSGIPath.