While trying to setup an elastic beanstalk worker application using the command line tools (eb tools), my configuration file (optionsettings.MyApp-env) gets overwritten when I start/update/stop the environment.
These are the steps to reproduce:
Using the CLI tools' command eb init I've created a new application in Elastic Beanstalk.
The config file in the .elasticbeanstalk folder had the following line:
OptionSettingFile=/Users/doron/projects/workers/my-worker/.elasticbeanstalk/optionsettings.MyWorkerName-dev
After running eb start for the first time, that file was created with some values.
I went ahead and changed its contents according to http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html so it'll be configured as I want (environment parameters, autoscaling servers amount, etc...).
To apply the changes I've tried the following:
Update the existing environment with eb update.
Terminate the existing environment with eb stop and build it from scratch with eb start.
In both cases the optionsettings file get changed after running the command (update or start).
The new content of the file looks more like the vanilla version I got after calling the first eb start with all sorts of configuration parameters that I added - removed completely.
Is there another way of configuring the environment (not the software on the machine, but the configuration that exists in the console - instance type, regions, autoscaling, rotating updates, etc...) ?
I realise that this is an old thread but in case anyone comes across this, as I did, then check out this thread on the AWS forums for Elasticbeanstalk https://forums.aws.amazon.com/thread.jspa?messageID=395052#395052
It explains how settings set in the .elasticbeanstalk/optionsettings. file are set using the API in such a way that they can't be changed later, unlike those set in the .ebextensions/*.config files.
Also, in an incredibly annoying move, the optionsettings file will often set some settings in it which you want to set in the .config file however it automatically re-creates the optionsettings file when running eb start and there's very little that seems possible. This makes using the eb command line tools close to impossible to use if you want to change something like the WSGIPath.
Related
I'm facing a similar issue to this question while trying to implement the new hooks logic on a AWS Linux 2 managed platform running Docker.
I have created my file inside this .platform/hooks/postdeploy/configure_nginx.sh
which is living in the src/ folder of my app.
I can see the file in my host after the deploy but it's never executed /var/app/current/src/.platform/hooks/postdeploy/configure_nginx.sh
It has the right privileges and I can run it if I ssh into my instance.
The EBS environment is initialized through a docker-compose file where I start 2 public containers and my custom app (in which I have put the hook folders)
The script is not being executed so I'm a bit lost on where I need to put it. FYI, my eb deploy is simply copying a Dockerrun file which is grabbing an image from one of my ECR so basically nothing is done in there.
Thanks for your help!
[SOLUTION]
I've found how it should work.
You have to create the .platform folder at the same level as the .ebextensions one.
In my case I'm deploying a zip archive simply containing my Dockerrun.aws.json and both .ebextensions and .platform folders.
So remember to zip it with the whole package before deploying it to your EBS environment.
Background
Our AWS Elastic Beanstalk environment, running the latest version of the pre-configured "Python 3.7 on 64-bit Amazon Linux 2" platform branch, has a lot of custom configuration and environment properties.
Now we would like to switch this environment to the "Python 3.8 on 64-bit Amazon Linux 2" platform branch.
Basically, the goal is to clone the environment, keeping the current configuration (other than platform branch and version) and environment properties.
Problem
Unfortunately, when cloning, it is not possible to switch between different platform branches (we can only switch between platform versions within the same platform branch).
The documentation suggests that a blue/green deployment is required here. However, a blue/green deployment involves creating a new environment from scratch, so we would still need some other way to copy our configuration settings and environment properties.
Question
What would be the recommended way to copy the configuration settings and/or environment properties from the original environment into a newly created environment?
I suppose we could use eb config to download the original configuration, modify the environment name, platform branch and version, and so on, and then use eb config --update on the new environment. However, that feels like a hack.
Summary
save current config: eb config save <env name>
use a text editor to modify the platform branch in the saved config file
create new environment based on modified config file: eb create --cfg <config name> (add --sample to use the sample application)
if necessary, delete local config files
if necessary, use eb printenv and eb setenv to copy environment properties
EDIT: For some reason the saved config does not include all security group settings, so it may be necessary to check those manually, using the EB console (configuration->instances).
Background
AWS support have confirmed that using eb config is the way to go, and they referred to the online documentation for details.
Unfortunately, the documentation for the eb cli does not provide all the answers.
The following is based on my own adventures using the latest version of the eb cli (3.20.2) with botocore 1.21.50, and documentation at the time of writing (Sep 30, 2021). Note there's a documentation repo on github but it was last updated six months ago and does not match the latest online docs...
eb config
Here's a screenshot from the eb config docs:
Indeed, if you call eb config my-env or eb config my-env --display, environment properties are not shown.
However, this does not hold for eb config save: YAML files created using eb config save actually do include environment properties*.
*Beware, if your environment properties include secrets (e.g. passwords), these also end up in your saved configs, so make sure you don't commit those to version control.
Moreover, it is currently also possible to set environment properties using eb config --update.
This implies we should be able to "copy" both configuration settings and environment properties in one go.
EDIT: After some testing it turns out eb config save does not always get the complete set of environment properties: some properties may be skipped. Not yet sure why... Step 5 below might help in those cases.
Walk-through
Not sure if this is the best way to do it, but here's what seems to work for me:
Suppose we have an existing EB environment called py37-env with lots of custom configuration and properties, running the Python 3.7 platform branch.
The simplest way to "clone" this would be as follows:
Step 1: download the existing configuration
Download the configuration for the existing environment:
eb config save py37-env
By default, the config file will end up in our project directory as .elasticbeanstalk/saved_configs/py37-env-sc.cfg.yml.
The saved config file could look like this (just an example, also see environment manifest):
EnvironmentConfigurationMetadata:
Description: Configuration created from the EB CLI using "eb config save".
DateCreated: '1632989892000'
DateModified: '1632989892000'
Platform:
PlatformArn: arn:aws:elasticbeanstalk:eu-west-1::platform/Python 3.7 running on 64bit Amazon Linux 2/3.3.5
OptionSettings:
aws:elasticbeanstalk:application:environment:
MY_ENVIRONMENT_PROPERTY: myvalue
aws:elasticbeanstalk:command:
BatchSize: '30'
BatchSizeType: Percentage
aws:elb:policies:
ConnectionDrainingEnabled: true
aws:elb:loadbalancer:
CrossZone: true
aws:elasticbeanstalk:environment:
ServiceRole: aws-elasticbeanstalk-service-role
aws:elasticbeanstalk:healthreporting:system:
SystemType: enhanced
aws:autoscaling:launchconfiguration:
IamInstanceProfile: aws-elasticbeanstalk-ec2-role
EC2KeyName: my-key
aws:autoscaling:updatepolicy:rollingupdate:
RollingUpdateType: Health
RollingUpdateEnabled: true
EnvironmentTier:
Type: Standard
Name: WebServer
AWSConfigurationTemplateVersion: 1.1.0.0
Also see the list of available configuration options in the documentation.
Step 2: modify the saved configuration
We are only interested in the Platform, so it is sufficient here to replace 3.7 by 3.8 in the PlatformArn value.
If necessary, you can use e.g. eb platform list to get an overview of valid platform names.
Step 3: create a new environment based on the modified config file
eb create --cfg py37-env-sc
This will deploy the most recent application version. Use --version <my version> to deploy a specific version, or use --sample to deploy the sample application, as described in the docs.
This will automatically look for files in the default saved config folder, .elasticbeanstalk/saved_configs/.
If you get a ServiceError or InvalidParameterValueError at this point, make sure only to pass in the name of the file, i.e. without the file extension .cfg.yml and without the folders.
Step 4: clean up local saved configuration file
Just in case you have any secrets stored in the environment properties.
Step 5: alternative method for copying environment properties
If environment properties are not included in the saved config files, or if some of them are missing, here's an alternative way to copy them (using bash).
This might not be the most efficient implementation, but I think it serves to illustrate the approach. Error handling was omitted, for clarity.
source_env="py37-env" # or "$1"
target_env="py38-env" # or "$2"
# get the properties from the source environment
source_env_properties="$(eb printenv "$source_env")"
# format the output so it can be used with `eb setenv`
mapfile -t arg_array < <(echo "$source_env_properties" | grep "=" | sed -e 's/ =/=/g' -e 's/= /=/g' -e 's/^ *//g')
# copy the properties to the target environment
eb setenv -e "$target_env" "${arg_array[#]}"
This has the advantage that it does not store any secrets in local files.
I have recently start using aws cdk as a newbie. so i ran lot of commands that i had no idea about.
now i want to remove all settings like env variables i created or profiles and start from scratch. what should in un install to achieve that?
I'm not totally sure what you're trying to reset but here's a few suggestions that might help:
Remove Deployed CDK Stacks
cdk destroy stack_name
Note: You'll have to do this for every stack you've deployed. This can also be done through "CloudFormation" in the AWS dashboard in your browser.
Remove CLI Settings
As per https://docs.amazonaws.cn/en_us/cli/latest/userguide/cli-configure-files.html
To remove a setting, use an empty string as the value, or manually delete the setting in your config and credentials files in a text editor.
Example:
aws configure set cli_pager ""
Remove Profiles
Unsure if you can do this easily through the CLI but you can just manually remove them from your config files. There are only two config files and they can be found using https://docs.amazonaws.cn/en_us/cli/latest/userguide/cli-configure-profiles.html
~/.aws/credentials (Linux & Mac) or %USERPROFILE%.aws\credentials (Windows)
~/.aws/config (Linux & Mac) or %USERPROFILE%.aws\config (Windows)
If you need more specific help on how to undo something then please provide an example of what exactly you ran that you would like to undo.
Below is the setup of the application which runs a docker container into elasticbeanstalk.
step 1:
created the parent folder say apptest and inside that I have placed the Dockerfile, package.json and small hello world server.js node app.
step 2:
inside the parent folder apptest ran the command eb init which crated a hidden folder .elasticbeanstalk with config.yml comprising default settings.
step 3:
added .ebextensions with a config file 01_run.config, comprising the below configuration to update the instance type.
option_settings:
aws:autoscaling:launchconfiguration :
InstanceType : "m4.xlarge"
Note that till now no environment is created yet. And since I have extensions created, should override the defaults with the instance type pointed to m4.xlarge.
step 4:
Now ran the command eb create apptest-dev ( say for example ) and created the environment
Problem:
when the environment is created, it did not have the m4.xlarge, rather got created with default instance type as t2.micro. But when uploaded the zipped version of this folder contents into the environment from console ( excluding the .elasticbeanstalk folder ), ebextensions folder configuration is picked up. Its only with the option_settings not getting worked, but the other resources like files, commands are getting reflected both from command line and from the file upload.
I feel its kind of very small thing missing which I am not able to figure it out from blogs and documentation. Thanks for the help in advance.
During eb create, the EBCLI passes it's own defaults for many of the option settings, among which is the instance type. Since, the EBCLI does not parse .ebextensions, and the Beanstalk service prefers the defaults passed by the EBCLI, the instance type specified in your .ebextensions are disregarded.
There are two ways to get around this:
call eb config after eb create. In the interactive mode, change the instance type, and save and exit.
call eb create as eb create -i m4.xlarge
How to call a custom configuration file per environment when using php's composer and amazon's elastic beanstalk?
for example, when having a "composer.dev.json" & "composer.master.json" files (one of them could request files from a "dev" version/branch and the other one for stable files).
Elastic Beanstalk has composer built-in. In order to use it, the only thing you need to do is adding a "composer.json" file in your root folder. In case you want to use a custom composer file for each version, it can also be accomplished relatively easily. Create a folder named ".elasticbeanstalk" and its relative file "optionsettings.branch-name" - it should be created using the "eb cli", particularity, with the commands "eb init" and "eb branch"- more info can be found here http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cmd-commands.html.
After we have the optionsettings file, we should define a param named "COMPOSER" in there and just have it to point to the custom composer file we would like to run, for example:
[aws:elasticbeanstalk:application:environment]
COMPOSER=composer.dev.json
on your next deploment, it should run the composer using composer.dev.json