I have a dockerized Django application that I want to deploy on SAP Cloud Platform via cloudfoundry cli utility. I have added couple of User Provided Services with their own set of credentials. For example, I added AWS S3 as a User Provided Service and provided credentials for the same.
Now those credentials are available as environment variable as
VCAP_SERVICES={"user-provided":[{
"label": "user-provided",
"name": "s3",
"tags": [
],
"instance_name": "s3",
"binding_name": null,
"credentials": {
"aws_access_key": "****",
"aws_secret_key": "****",
"bucket": "****",
"region": "****",
"endpoint": "*****"
},
"syslog_drain_url": "",
"volume_mounts": [
]
}]}
I have .env file where in I have variables defined, eg. AWS_ACCESS_KEY. Usually I pass string value to the variable which is then consumed by my app. But, given that I have configured it via User Provided Service mechanism and credentials are already there, I was wondering how do I get to access those credentials.
There are a few ways to extract service information in Python applications.
You can do it programmatically using the cfenv library. You would simply integrate this library into the start-up of your application. This generally offers the most flexibility, but can sometimes be difficult to integrate with frameworks, depending on how frameworks expect the configuration to be feed in.
You can generate environment variables or configuration files, from your example the .env file, on the fly. This can be done using a .profile script. A .profile script, if placed into the root of your application will execute prior to your application but inside the runtime container. This allows you to adjust the configuration of your application at just the last moment.
A .profile script is just a shell script and in it, you can use tools like jq or sed to extract information from the VCAP_SERVICES environment variable and put that information elsewhere (possibly other environment variables or into a .env file).
Because you are pushing a Python application, the .profile could also execute a Python script. The Python buildpack will run and guarantee that a Python runtime is available on the PATH for use by your .profile script. Thus you can do something like this, to execute a Python script.
.profile script:
python $HOME/.profile.py
.profile.py script (made up this name, you can call it anything):
#!/usr/bin/env python3
print("Hello from python .profile script")
You can even import Python libraries in your script that are included in your requirements.txt file from this script.
Related
so my goal is to deploy a serverless Dockerized NextJS application on ECS/Fargate.
So when I docker build my project using the command docker build . -f development.Dockerfile --no-cache -t myapp:latest everything is running successfully except Docker build doesn't consider the env file in my project's root directory. Once build finishes, I push the Docker image to Elastic Container Repository(ECR) and my Elastic Container Service(ECS) references that ECR.
So naturally, my built image doesn't have a ENV file(contains the API keys and DB credentials), and as a result my app is deployed but all of the services relying on those credentials are failing because there isn't an ENV file in my container and all of the variables become undefined or null.
To fix this issue I looked at this AWS doc and implemented a solution that stores my .env file in AWS S3 and that S3 ARN gets refrenced in the container service where the .env file is stored. However, that didn't workout and I think it's because of the way I'm setting my
next.config.js to reference my environmental files in my local codebase. I also tried to set my environmental variables manually(very unsecure, screenshot below) when configuring the container in my task defination, and that didn't work either.
My next.confg.js
const dotEnvConfig = { path: `../../${process.env.NODE_ENV}.env` };
require("dotenv").config(dotEnvConfig);
module.exports = {
serverRuntimeConfig: {
// Will only be available on the server side
xyzKey: process.env.xyzSecretKey || "",
},
publicRuntimeConfig: {
// Will be available on both server and client
appUrl: process.env.app_url || "",
},
};
So on my local codebase in the root directory I have two files development.env (local api keys) and production.env(live api keys) and my next.config.js is located in /packages/app/next.config.js
So apparently it was just a plain NextJS's way of handling env variables.
In next.config.js
module.exports = {
env: {
user: process.env.SQL_USER || "",
// add all the env var here
},
};
and to call the environmental variable user in the app all you have to do is call process.env.user and user will reference process.env.SQL_USER in my local .env file where it will be stored as SQL_USER="abc_user"
You should be setting the environment variables in the ECS task definition. In order to prevent storing sensitive values in the task definition you should use AWS Parameter Store, or AWS Secrets Manager, as documented here.
I am using AWS SSM Run Command with the AWS-RunShellScript document to run a script on an AWS Linux 1 instance. Part of the script includes using an environment variable. When I run the script myself, everything is fine. But when I run the script with SSM, it can't see the environment variable.
This variable needs to be passed to a Python script. I had originally been trying os.environ['VARIABLE'] to no effect.
I know that AWS SSM uses root privileges and so I have put a line exporting the variable in the root ~/.bashrc file, yet it still can not see the variable. The root user can see it when I run it myself.
Is it not possible for AWS SSM to use environment variables, or am I not exporting it correctly? If it is not possible, I'll try using AWS KMS instead to store my variable.
~/.bashrc
export VARIABLE="VALUE"
script.sh
"$VARIABLE"
Security is important, hence why I don't want to just store the variable in the script.
SSM does not open an actual SSH session so passing environment variables won't work. It's essential a daemon running on the box that's taking your requests and processing them. It's a very basic product: it doesn't support any of the standard features that come with SSH such as SCP, port forwarding, tunneling, passing of env variables etc. An alternative way of passing a value you need to a script would be to store it in AWS Systems Manager Parameter Store, and have your script pull the variable from the store.
You'll need to update your instance role permissions to have access to ssm:GetParameters for the script you run to access the value stored.
My solution to this problem:
set -o allexport; source /etc/environment; set +o allexport
-o allexport enables all variables in /etc/environment to be exported. +o allexport disables this feature.
For more information see the Set builtin documentation
I have tested this solution by using the AWS CLI command aws ssm send-command:
"commands": [
"set -o allexport; source /etc/environment; set +o allexport",
"echo $TEST_VAR > /home/ec2-user/app.log"
]
I am running bash script in my SSM command document, so I just source the profile/script to have env variables ready to be used by the subsequent commands. For example,
"runCommand": [
"#!/bin/bash",
". /tmp/setEnv.sh",
"echo \"myVar: $myVar, myVar2: $myVar2\""
]
You can refer to Can a shell script set environment variables of the calling shell? for sourcing your env variables. For python, you will have to parse your source profile/script, see Emulating Bash 'source' in Python
What is the best way to handle deployment to staging and production for the Dockerrun.aws.json file? Is there a way to pass variables to the image value, etc or have multiple Dockerrun.aws.json files one for each environment? Currently my staging env gets the image tagged as staging and production gets the images tagged as production but I need the Dockerrun.aws.json different for each env? I either want to do something like:
"image": "${IMAGE}",
where IMAGE is defined in the configs for each environment or separate each file out. So I can setup each container differently based on staging or production.
Old question but in case it can help others, I wanted to do pretty much the same thing and automate it, so as a quick way to do it I came up with a simple shell script.
The idea was to have a Dockerrun.aws.json template file that would hold a dynamic ENV property, then depending on the desired environment, the script would use this template and generate the appropriate Dockerrun.aws.json file.
Example:
Create a shell script with the following content:
#!/bin/bash
# current script directory path
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# $1 will be the environement name passed to the script : it can only be dev or prod
# if empty, we ask for user input for convenience
if [ "$1" == "" ]; then
echo -n "Enter your the environment (either 'dev' or 'prod') and press [ENTER]:"
read ENV
else
ENV=$1
fi
# check if environment name is valid
if [ "$ENV" == "dev" ] || [ "$ENV" == "prod" ] ; then
# move to shell script directory
cd $DIR
# generate Dockerfile from template by replacing the ENV property by the input
echo "Generating Dockerrun.aws.json..."
sed -e "s/\${ENV}/$ENV/g" Dockerrun.aws.json.template > Dockerrun.aws.json
# do other things here if necessary
else
echo "$ENV is not a valid environment name, accepted values : env & prod"
exit 0
fi
Then create your Dockerrun.aws.json.template file:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "php-app",
"image": "phpfpm-image-${ENV}",
#...
},
{
"name": "nginx-proxy",
"image": "nginx-image-${ENV}",
#...
}
]
}
Now, just put the shell script where your Dockerrun.aws.jon.template file resides, and run it like so:
sh yourscript.sh dev
It will generate a valid file for you to use for the given environment.
This is a simple example that gives you a basic idea of what do to, then you can build something much more complex from it. I personally use it to pick all the right config files (.ebextensions, etc.) and then zip the whole thing to upload on beanstalk.
One way could be having a Makefile and have separate commands for deploying in separate environments e.g "make deploy-staging" or "make deploy-prod". Internally, the command would generate Dockerrun.aws.json dynamically by interpolating env specific values to a template Dockerrun.aws file.
I'm new to Docker, and my previous experience is with deploying Java web applications (running in Tomcat containers) to Elastic Beanstalk. The pipeline I'm used to goes something like this: a commit is checked into git, which triggers a Jenkins job, which builds the application JAR (or WAR) file, publishes it to Artifactory, and then deploys that same JAR to an application in Elastic Beanstalk using eb deploy. (Apologies if "pipeline" is a reserved term; I'm using it conceptually.)
Incidentally, I'm also going to be using Gitlab for CI/CD instead of Jenkins (due to organizational reasons out of my control), but the jump from Jenkins to Gitlab seems straight-forward to me -- certainly moreso than the jump from deploying WARs directly to deploying Dockerized containers.
Moving over into the Docker world, I imagine the pipeline will go something like this: a commit is checked into git, which triggers the Gitlab CI, which will then build the JAR or WAR file, publish it to Artifactory, then use the Dockerfile to build the Docker image, publish that Docker image into Amazon ECR (maybe?)... and then I'm honestly not sure how the Elastic Beanstalk integration would proceed from there. I know it has something to do with the Dockerrun.aws.json file, and presumably needs to call the AWS CLI.
I just got done watching a webinar from Amazon called Running Microservices and Docker on AWS Elastic Beanstalk, which stated that in the root of my repo there should be a Dockerrun.aws.json file which essentially defines the integration to EB. However, it seems that JSON file contains a link to the individual Docker image in ECR, which is throwing me off. Wouldn't that link change every time a new image is built? I'm imagining that the CI would need to dynamically update the JSON file in the repo... which almost feels like an anti-pattern to me.
In the webinar I linked above, the host created his Docker image and pushed it ECR manually, with the CLI. Then he manually uploaded the Dockerrun.aws.json file to EB. He didn't need to upload the application however, since it was already contained within the Docker image. This all seems odd to me and I question whether I'm understanding things correctly. Will the Dockerrun.aws.json file need to change on every build? Or am I thinking about this the wrong way?
In the 8 months since I posted this question, I've learned a lot and we've already moved onto different and better technology. But I will post what I learned in answer to my original question.
The Dockerrun.aws.json file is almost exactly the same as an ECS task definition. It's important to use the Multi-Docker container deployment version of Beanstalk (as opposed to the single container), even if you're only deploying a single container. IMO they should just get rid of the single-container platform for Beanstalk as it's pretty useless. But assuming you have Beanstalk set to the Multi-Container Docker platform, then the Dockerrun.aws.json file looks something like this:
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "my-container-name-this-can-be-whatever-you-want",
"image": "my.artifactory.com/docker/my-image:latest",
"environment": [],
"essential": true,
"cpu": 10,
"memory": 2048,
"mountPoints": [],
"volumesFrom": [],
"portMappings": [
{
"hostPort": 80,
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/aws/elasticbeanstalk/my-image/var/log/stdouterr.log",
"awslogs-region": "us-east-1",
"awslogs-datetime-format": "%Y-%m-%d %H:%M:%S.%L"
}
}
}
]
}
If you decide, down the road, to convert the whole thing to an ECS service instead of using Beanstalk, that becomes really easy, as the sample JSON above is converted directly to an ECS task definition by extracting the "containerDefinitions" part. So the equivalent ECS task definition might look something like this:
[
{
"name": "my-container-name-this-can-be-whatever-you-want",
"image": "my.artifactory.com/docker/my-image:latest",
"environment": [
{
"name": "VARIABLE1",
"value": "value1"
}
],
"essential": true,
"cpu": 10,
"memory": 2048,
"mountPoints": [],
"volumesFrom": [],
"portMappings": [
{
"hostPort": 0,
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/aws/ecs/my-image/var/log/stdouterr.log",
"awslogs-region": "us-east-1",
"awslogs-datetime-format": "%Y-%m-%d %H:%M:%S.%L"
}
}
}
]
Key differences here are that with the Beanstalk version, you need to map port 80 to port 80 because a limitation of running Docker on Beanstalk is that you cannot replicate containers on the same instance, whereas in ECS you can. This means that in ECS you can map your container port to host port "zero," which really just tells ECS to pick a random port in the ephemeral range which allows you to stack multiple replicas of your container on a single instance. Secondly with ECS if you want to pass in environment variables, you need to inject them directly into the Task Definition JSON. In Beanstalk world, you don't need to put the environment variables in the Dockerrun.aws.json file, because Beanstalk has a separate facility for managing environment variables in the console.
In fact, the Dockerrun.aws.json file should really just be thought of as a template. Because Docker on Beanstalk uses ECS under-the-hood, it simply takes your Dockerrun.aws.json as a template and uses it to generate its own Task Definition JSON, which injects the managed environment variables into the "environment" property in the final JSON.
One of the big questions I had at the time when I first asked this question was whether you had to update this Dockerrun.aws.json file every time you deployed. What I discovered is that it comes down to a choice of how you want to deploy things. You can, but you don't have to. If you write your Dockerrun.aws.json file so that the "image" property references the :latest Docker image, then there's no need to ever update that file. All you need to do is bounce the Beanstalk instance (i.e. restart the environment), and it will pull whatever :latest Docker image is available from Artifactory (or ECR, or wherever else you publish your images). Thus, all a build pipeline would need to do is publish the :latest Docker image to your Docker repository, and then trigger a restart of the Beanstalk environment using the awscli, with a command like this:
$ aws elasticbeanstalk restart-app-server --region=us-east-1 --environment-name=myapp
However, there are a lot of drawbacks to that approach. If you have a dev/unstable branch that publishes a :latest image to the same repository, you become at risk of deploying that unstable branch if the environment happens to restart on its own. Thus, I would recommend versioning your Docker tags and only deploying the version tags. So instead of pointing to my-image:latest, you would point to something like my-image:1.2.3. This does mean that your build process would have to update the Dockerrun.aws.json file on each build. And then you also need to do more than just a simple restart-app-server.
In this case, I wrote some bash scripts that made use of the jq utility to programmatically update the "image" property in the JSON, replacing the string "latest" with whatever the current build version was. Then I would have to make a call to the awsebcli tool (note that this is a different package than the normal awscli tool) to update the environment, like this:
$ eb deploy myapp --label 1.2.3 --timeout 1 || true
Here I'm doing something hacky: the eb deploy command unfortunately takes FOREVER. (This was another reason we switched to pure ECS; Beanstalk is unbelievably slow.) That command hangs for the entire deployment time, which in our case could take up to 30 minutes or more. That's completely unreasonable for a build process, so I force the process to timeout after 1 minute (it actually continues the deployment; it just disconnects my CLI client and returns a failure code to me even though it may subsequently succeed). The || true is a hack that effectively tells Gitlab to ignore the failure exit code, and pretend that it succeeded. This is obviously problematic because there's no way to tell if the Elastic Beanstalk deployment really did fail; we're assuming it never does.
One more thing on using eb deploy: by default this tool will automatically try to ZIP up everything in your build directory and upload that entire ZIP to Beanstalk. You don't need that; all you need is to update the Dockerrun.aws.json. In order to do this, my build steps were something like this:
Use jq to update Dockerrun.aws.json file with the latest version tag
Use zip to create a new ZIP file called deploy.zip and put Dockerrun.aws.json inside it
Make sure a file called .elasticbeanstalk/config.yml is in place (described below)
Run the eb deploy ... command
Then you need a file in the build directory at .elasticbeanstalk/config.yml which looks like this:
deploy:
artifact: deploy.zip
global:
application_name: myapp
default_region: us-east-1
workspace_type: Application
The awsebcli knows to automatically look for this file when you call eb deploy. And what this particular file says is to look for a file called deploy.zip instead of trying to ZIP up the whole directory itself.
So the :latest method of deployment is problematic because you risk deploying something unstable; the versioned method of deployment is problematic because the deployment scripts are more complicated, and because unless you want your build pipelines to take 30+ minutes, there's a chance that the deployment won't be successful and there's really no way to tell (outside of monitoring each deployment yourself).
Anyways, it's a bit more work to set up, but I would recommend migrating to ECS whenever you can. (Better still to migrate to EKS, though that's a lot more work.) Beanstalk has a lot of problems.
The Amazon Elastic Beanstalk blurb says:
Elastic Beanstalk lets you "open the hood" and retain full control ... even pass environment variables through the Elastic Beanstalk console.
http://aws.amazon.com/elasticbeanstalk/
How to pass other environment variables besides the one in the Elastic Beanstalk configuration?
As a heads up to anyone who uses the .ebextensions/*.config way: nowadays you can add, edit and remove environment variables in the Elastic Beanstalk web interface.
The variables are under Configuration → Software Configuration:
Creating the vars in .ebextensions like in Onema's answer still works.
It can even be preferable, e.g. if you will deploy to another environment later and are afraid of forgetting to manually set them, or if you are ok with committing the values to source control. I use a mix of both.
Only 5 values is limiting, or you may want to have a custom environment variable name. You can do this by using the configuration files. Create a directory at the root of your project called
.ebextensions/
Then create a file called environment.config (this file can be called anything but it must have the .config extension) and add the following values
option_settings:
- option_name: CUSTOM_ENV
value: staging
After you deploy your application you will see this new value under
Environment Details -> Edit Configuration -> Container
for more information check the documentation here:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#customize-containers-format-options
Update
To prevent committing to your repository values like API keys, secrets and so on, you can put a placeholder value.
option_settings:
- option_name: SOME_API_KEY
value: placeholder-value-change-me
Later you can go to the AWS admin panel (Environment Details -> Edit Configuration -> Container) and update the values there. In my experience these values do not change after subsequent deployments.
Update 2
As #Benjamin stated in his comment, since the new look and feel was rolled out July 18, 2013 it is possible to define any number of environment variables directly from the console:
Configuration > Software Configuration > Environment Properties
In the 2016 Java8 Tomcat8 AMI, ElasticBeanstalk fails to set environment variables from the web configuration. They are really setting jvm -D properties instead.
--
"The following properties are passed into the application as environment variables. Learn more."
This statement is incorrect for the Java Tomcat ami. Amazon does not set these as environment variables. They are set as System properties passed on the command line to Tomcat as a -D property for jvm.
The method in Java to get environment variables is not the same for getting a property.
System.getenv vs System.getProperty
I ssh'd into the box and verified that the environment variable was never set. However, in the tomcat logs I can see the -D property is set.
I've changed my code to check for both locations now as a workaround.
AWS will interpret CloudFormation template strings in your environment variables. You can use this to access information about your EB environment inside your application:
In the AWS web interface the following will be evaluated as the name of your environment (note the back ticks):
`{ "Ref" : "AWSEBEnvironmentName" }`
Or, you can use an .ebextensions/*.config and wrap the CloudFormation template in back ticks (`):
{
"option_settings": [
{
"namespace": "aws:elasticbeanstalk:application:environment",
"option_name": "ENVIRONMENT_NAME",
"value": "`{ \"Ref\" : \"AWSEBEnvironmentName\" }`"
}
]
}
Alternatively, you could use the Elastic Beanstalk CLI to set environment variables.
To set an environment variable: eb setenv FOO=bar
To view the environment variables: eb printenv
Environment Details -> Edit Configuration -> Container
This seems to be the only way to set ENVs with dynamic values in beanstalk. I came up with a workaround that works for my multi-docker setup:
1) Add this to your Dockerfile before building + uploading to your ECS
repository:
CMD eval `cat /tmp/envs/env_file$`; <base image CMD goes here>;
2) In your Dockerrun.aws.json file create a volume:
{
"name": "env-file",
"host": {
"sourcePath": "/var/app/current/envs"
}
}
3) Mount volume to your container
{
"sourceVolume": "env-file",
"containerPath": "/tmp/envs",
"readOnly": true
}
4) In your .ebextensions/options.config file add a container_commands
block like so:
container_commands:
01_create_mount:
command: "mkdir -p envs/"
02_create_env_file:
command: { "Fn::Join" : [ "", [ 'echo "', "export ENVIRONMENT_NAME=" , { "Ref", "RESOURCE" }, ';" > envs/env_file;' ] ] }
5) eb deploy and your ENVS should be available in your docker container
You can add more ENVs by adding more container_commands like:
02_create_env_file_2:
command: { "Fn::Join" : [ "", [ 'echo "', "export ENVIRONMENT_NAME_2=" , { "Ref", "RESOURCE2" }, ';" >> envs/env_file;' \] \] }
Hope this helps!