How can I set secrets.yml for test environment for CircleCI with Rspec? My secrets.yml is not on git.
When I run my tests on CircleCI, they fail with error:
ArgumentError: Missing required arguments: aws_access_key_id, aws_secret_access_key
This solution worked out for me:
Create config/secrets.ci.yml with test environment variables.
Add circle.yml to the root directory:
circle.yml
machine:
ruby:
version:
2.2.2
dependencies:
override:
- mv config/secrets.ci.yml config/secrets.yml
Another option is to use CircleCI Environment Variables. This would allow you to load in secret info without it being in your repository or file system at all.
Not to mention that the AWS CLI will automatically read those variables from the environment if they are present.
Related
Background
Our AWS Elastic Beanstalk environment, running the latest version of the pre-configured "Python 3.7 on 64-bit Amazon Linux 2" platform branch, has a lot of custom configuration and environment properties.
Now we would like to switch this environment to the "Python 3.8 on 64-bit Amazon Linux 2" platform branch.
Basically, the goal is to clone the environment, keeping the current configuration (other than platform branch and version) and environment properties.
Problem
Unfortunately, when cloning, it is not possible to switch between different platform branches (we can only switch between platform versions within the same platform branch).
The documentation suggests that a blue/green deployment is required here. However, a blue/green deployment involves creating a new environment from scratch, so we would still need some other way to copy our configuration settings and environment properties.
Question
What would be the recommended way to copy the configuration settings and/or environment properties from the original environment into a newly created environment?
I suppose we could use eb config to download the original configuration, modify the environment name, platform branch and version, and so on, and then use eb config --update on the new environment. However, that feels like a hack.
Summary
save current config: eb config save <env name>
use a text editor to modify the platform branch in the saved config file
create new environment based on modified config file: eb create --cfg <config name> (add --sample to use the sample application)
if necessary, delete local config files
if necessary, use eb printenv and eb setenv to copy environment properties
EDIT: For some reason the saved config does not include all security group settings, so it may be necessary to check those manually, using the EB console (configuration->instances).
Background
AWS support have confirmed that using eb config is the way to go, and they referred to the online documentation for details.
Unfortunately, the documentation for the eb cli does not provide all the answers.
The following is based on my own adventures using the latest version of the eb cli (3.20.2) with botocore 1.21.50, and documentation at the time of writing (Sep 30, 2021). Note there's a documentation repo on github but it was last updated six months ago and does not match the latest online docs...
eb config
Here's a screenshot from the eb config docs:
Indeed, if you call eb config my-env or eb config my-env --display, environment properties are not shown.
However, this does not hold for eb config save: YAML files created using eb config save actually do include environment properties*.
*Beware, if your environment properties include secrets (e.g. passwords), these also end up in your saved configs, so make sure you don't commit those to version control.
Moreover, it is currently also possible to set environment properties using eb config --update.
This implies we should be able to "copy" both configuration settings and environment properties in one go.
EDIT: After some testing it turns out eb config save does not always get the complete set of environment properties: some properties may be skipped. Not yet sure why... Step 5 below might help in those cases.
Walk-through
Not sure if this is the best way to do it, but here's what seems to work for me:
Suppose we have an existing EB environment called py37-env with lots of custom configuration and properties, running the Python 3.7 platform branch.
The simplest way to "clone" this would be as follows:
Step 1: download the existing configuration
Download the configuration for the existing environment:
eb config save py37-env
By default, the config file will end up in our project directory as .elasticbeanstalk/saved_configs/py37-env-sc.cfg.yml.
The saved config file could look like this (just an example, also see environment manifest):
EnvironmentConfigurationMetadata:
Description: Configuration created from the EB CLI using "eb config save".
DateCreated: '1632989892000'
DateModified: '1632989892000'
Platform:
PlatformArn: arn:aws:elasticbeanstalk:eu-west-1::platform/Python 3.7 running on 64bit Amazon Linux 2/3.3.5
OptionSettings:
aws:elasticbeanstalk:application:environment:
MY_ENVIRONMENT_PROPERTY: myvalue
aws:elasticbeanstalk:command:
BatchSize: '30'
BatchSizeType: Percentage
aws:elb:policies:
ConnectionDrainingEnabled: true
aws:elb:loadbalancer:
CrossZone: true
aws:elasticbeanstalk:environment:
ServiceRole: aws-elasticbeanstalk-service-role
aws:elasticbeanstalk:healthreporting:system:
SystemType: enhanced
aws:autoscaling:launchconfiguration:
IamInstanceProfile: aws-elasticbeanstalk-ec2-role
EC2KeyName: my-key
aws:autoscaling:updatepolicy:rollingupdate:
RollingUpdateType: Health
RollingUpdateEnabled: true
EnvironmentTier:
Type: Standard
Name: WebServer
AWSConfigurationTemplateVersion: 1.1.0.0
Also see the list of available configuration options in the documentation.
Step 2: modify the saved configuration
We are only interested in the Platform, so it is sufficient here to replace 3.7 by 3.8 in the PlatformArn value.
If necessary, you can use e.g. eb platform list to get an overview of valid platform names.
Step 3: create a new environment based on the modified config file
eb create --cfg py37-env-sc
This will deploy the most recent application version. Use --version <my version> to deploy a specific version, or use --sample to deploy the sample application, as described in the docs.
This will automatically look for files in the default saved config folder, .elasticbeanstalk/saved_configs/.
If you get a ServiceError or InvalidParameterValueError at this point, make sure only to pass in the name of the file, i.e. without the file extension .cfg.yml and without the folders.
Step 4: clean up local saved configuration file
Just in case you have any secrets stored in the environment properties.
Step 5: alternative method for copying environment properties
If environment properties are not included in the saved config files, or if some of them are missing, here's an alternative way to copy them (using bash).
This might not be the most efficient implementation, but I think it serves to illustrate the approach. Error handling was omitted, for clarity.
source_env="py37-env" # or "$1"
target_env="py38-env" # or "$2"
# get the properties from the source environment
source_env_properties="$(eb printenv "$source_env")"
# format the output so it can be used with `eb setenv`
mapfile -t arg_array < <(echo "$source_env_properties" | grep "=" | sed -e 's/ =/=/g' -e 's/= /=/g' -e 's/^ *//g')
# copy the properties to the target environment
eb setenv -e "$target_env" "${arg_array[#]}"
This has the advantage that it does not store any secrets in local files.
In a Symfony 4.3 application using symfony/dotenv 4.3.11 and aws/aws-sdk-php 3.173.13:
I'd like to authenticate the AWS SDK using credentials provided via environment variables, and I'd like to use the dotenv component to provide those environment variables.
This should be possible: Setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables is one way to automatically authenticate the with the aws sdk. And DotEnv should turn your configuration into environment variables.
However, when I set these variables in my .env.local or .env files, I get the following error:
Aws\Exception\CredentialsException: Error retrieving credentials from the instance profile metadata service.
This does not work:
.env.local:
AWS_ACCESS_KEY_ID=XXX
AWS_SECRET_ACCESS_KEY=XXXXXX
$ ./bin/console command-that-uses-aws-sdk
This works:
$ AWS_ACCESS_KEY_ID=XXX AWS_SECRET_ACCESS_KEY=XXXXXX ./bin/console command-that-uses-aws-sdk
Debug info:
I made a symfony command that outputs the environment variables in $_ENV. With AWS_ACCESS_KEY_ID/SECRET in .env.local, sure enough it appears as an environment variable:
...
[SYMFONY_DOTENV_VARS] => MEQ_ENV,APP_ENV,APP_SECRET,DATABASE_URL,AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_REGION,AWS_ACCOUNT
[AWS_ACCESS_KEY_ID] => XXX
[AWS_SECRET_ACCESS_KEY] => XXXXXX
...
The aws php client documentation states:
The SDK uses the getenv() function to look for the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables.
=> it uses getenv() not $_ENV.
But the Symfony Dotenv component (by default) just populates $_ENV and doesn't call putenv therefore your settings in .env files are not accessible by getenv().
Here are some options:
call Dotenv())->usePutenv(true) (but as symfony states: Beware that putenv() is not thread safe, that's why this setting defaults to false)
call putenv() manually exclusively for the aws setting
Wrap the aws client in your own symfony service and inject the settings from .env
Is it a good approach to commit the .elasticbeanstalk/config.yml inside the git repo of a project which uses eb deploy?
We want to deploy using our CI and so we can not use the interactive eb init.
What we are thinking now is to define our dev, uat and prod inside that config.yml (if possible) and to point to that environment using eb deploy.
We saw that we could perform eb init with all necessary parameters in ebcli version2 but not in version 3 anymore? So it seems the approach is changed?
Can someone explain how to deploy EB for multiple environments, without interaction?
We want to deploy using our CI and so we can not use the interactive eb init
You can suppress the interactive mode as follows:
eb init --platform <platform-name> --region <region-name> <application-name>
Is it a good approach to commit the .elasticbeanstalk/config.yml inside the git repo of a project which uses eb deploy?
Can someone explain how to deploy EB for multiple environments, without interaction?
By design, the EBCLI avoids committing the .elasticbeanstalk/ directory since it can contain developer-specific information, which when committed to VC can cause confusion. So, it's best avoided from VC. You are free to commit it to version control. Ensure there's no sensitive information here. Logs, and saved configurations are usually stored in .elasticbeanstalk/.
You can copy pertinent portions of the .elasticbeanstalk/config.yml file into root-level file from which CI could read information such as the environment name to use.
Locally, you could create a pre-commit Git hook that would read the default environment name from the .elasticbeanstalk/config.yml file into the root-level file -- let's call it .environment_config.sh. It could be a statement as simple as export BEANSTALK_ENVIRONMENT_NAME=<environment name from .elasticbeanstalk/config.yml>
On the CI server:
3.1. Ensure PWD is git init-ed. Systems such as Jenkins usually are git init-ed with the necessary branch, so CI can simply source .environment_config.sh at this point and load the name of the environment to deploy.
3.2. eb init --platform <platform-name> --region <region-name> <application-name>
3.3. eb use $BEANSTALK_ENVIRONMENT_NAME
3.4. eb deploy
(You could combine 3.3. and 3.4. by performing eb deploy $BEANSTALK_ENVIRONMENT_NAME instead; I just wanted to demonstrate the use of eb use)
The EB CLI is really meant to be used from a workstation. I think you'd be better off scripting your CI with the AWS CLI.
A deployment with eb deploy will archive your code in S3 (or CodeCommit), create a new application version then update the environment with the new version label. All of those operations are supported with AWS CLI commands.
Or, you could write your own deployment script in Python with boto3. That's an easy option too. That's basically what the EB CLI is.
I am trying to deploy to elastic beanstalk using the cli.
The command I run is
eb create --modules ebtargets/goapi -v
I get the following error
WARNING: You have uncommitted changes.
INFO: Getting version label from git with git-describe
INFO: Uploading archive to s3 location: goapi/app-d4ec2-160630_135740.json
Uploading goapi/app-d4ec2-160630_135740.json to S3. This may take a while.
Upload Complete.
INFO: Creating AppVersion app-d4ec2-160630_135740
--- Waiting for application versions to be pre-processed ---
Finished processing application version app-d4ec2-160630_135740
--- Creating modules ---
ERROR: ServiceError - ApplicationVersion app-d4ec2-160630_135740 must specify an environment name in env.yaml
I have the key EnvironmentName in my env.yaml file. Any help on this?
I had a similar problem and I noticed you have to check your env.yaml into version control otherwise it won't be used:
git add env.yaml
git commit -m "add elastic beanstalk environment config"
Check that you have an .elasticbeanstalk folder into your ebtargets/goapi
If necessary, use eb init --modules ebtargets/goapi.
More information here
I'm getting the following error updating composer in an AWS Elastic Beanstalk deployment.
The HOME or COMPOSER_HOME environment variable must be set for composer to run correctly
The command causing the error actually sets the environment variable on the same line. It's part of a .ebextensions command
commands:
01_update_composer:
command: export COMPOSER_HOME=/var/home && composer.phar self-update
How can I fix this? Is there something else that could cause that error?
I have tried setting the environment in this configuration env: but that made no difference. I've also added printenv && composer.phar self-update and can confirm the variable is set.
Replace with this:
command: export COMPOSER_HOME=/root && /usr/bin/composer.phar self-update 1.0.0-alpha11
There is a composer issue. issue