I'm looking for a way to read data from an .env file trough the build process of an image thus option docker-compose is not option as I am using AWS CDK and with "cdk deploy" command the building process initiates same way as running "docker build" command and as far as I see there is not such an option.
I'd imagine this will work similarly. I wanted to do the same thing for React and found this article, which helped. The env.sh file in there would be copied to your Dockerfile along with the .env file.
https://www.freecodecamp.org/news/how-to-implement-runtime-environment-variables-with-create-react-app-docker-and-nginx-7f9d42a91d70/
Related
We run these two commands (the first one is async and the other runs synchronously)
#async BUT does something funky and doesn't run the Dockerfile image as-is
gcloud alpha builds triggers run staging-deploy --branch master
# sync BUT runs the image the way it's supposed to run!!!
gcloud builds submit --config cloudbuild.yaml
both are using our cloudbuild.yaml
steps:
- name: gcr.io/$PROJECT_ID/continuous-deploy
args: ['${_SERVICE}', '${_DOWNLOAD_URL}']
timeout: 1000s
substitutions:
_SERVICE: none
_DOWNLOAD_URL: none
timeout: 1100s
Our Dockerfile is very very simple
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
RUN mkdir -p ./monobuild
COPY . ./monobuild/
WORKDIR "/monobuild"
#NOTE: This file in google cloud build trigger MUST be in root of monorepo BUT I don't know why
#NOTE: This command receives any arguments to docker
#ie. for "docker run {image} {args}", it receives the args
ENTRYPOINT ["./downloadAndExtract.sh"]
Sooo, when I run the SECOND command, it completely uses the docker image obeying the Dockerfile. When I run the first command, it's ignoring all my Dockerfile stuff and trying to run scripts in my git repo(which is very frustrating and not what I want).
We HAD this directory structure
- gitroot
- stagingDeploy
- Dockerfile
- deployStaging.sh # part of Dockerfile
- cloudbuild.yaml
- prodDeploy
- Dockerfile
- prodDeploy.sh #part of Docker file
- cloudbuild.yaml
Of course, only the second command works with this directory structure. The first command CANNOT find deployStaging.sh UNTIL we ln -s stagingDeploy/deployStaging.sh from our gitrepo root and we have around 5 deploy directories and now our git repo root is fully polluted.
It is to say the least very frustrating and we are not sure how to clean this up so prodDeploy contains all the prod deploy scripts and staging, the staging ones and get rid of all root files.
Of course, we now have a corrupted git repo directory structure with a whole slew of files in the root directory from various different builds(sometimes conflicting on accident as files get the same names sometimes).
EDIT: Not really much to share on configuration of twitter as each one just points to the yaml file is all like so
thanks,
Dean
My app currently uses a folder called "Documents" that is located in the root of the app. This is where it stores supporting docs, temporary files, uploaded files etc. I'm trying to move my app from Azure to Beanstalk and I don't know how to give permissions to this folder and sub-folders. I think it's supposed to be done using .ebextensions but I don't know how to format the config file. Can someone suggest how this config file should look? This is an ASP.NET app running on Windows/IIS.
Unfortunately, you cannot use .ebextensions to set permissions to files/folders within your deployment directory.
If you look at the event hooks for an elastic beanstalk deployment:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-windows-ec2.html#windows-container-commands
You'll find that commands run before the ec2 app and web server are set up, and
container_commands run after the ec2 app and web server are setup, but before your application version is deployed.
The solution is to use a wpp.targets file to set the necessary ACLs.
The following SO post is most useful
Can Web Deploy's setAcl provider be used on a sub-directory?
Given below is the sample .ebextensions config file to create a directory/file and modify the permissions and add some content to the file
====== .ebextensions/custom_directory.config ======
commands:
create_directory:
command: mkdir C:\inetpub\AspNetCoreWebApps\backgroundtasks\mydirectory
command: cacls C:\inetpub\AspNetCoreWebApps\backgroundtasks\mydirectory /t /e /g username:W
files:
"C:/inetpub/AspNetCoreWebApps/backgroundtasks/mydirectory/mytestfile.txt":
content: |
This is my Sample file created from ebextensions
ebextensions go into the root of the application source code through a directory called .ebextensions. For more information on how to use ebextensions, please go through the documentation here
Place a file 01_fix_permissions.config inside .ebextensions folder.
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/49_change_permissions.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo chown -R ec2-user:ec2-user tmp/
Following that you can set your folder permissions as you want.
See this answer on Serverfault.
There are platform hooks that you can use to run scripts at various points during deployment that can get you around the shortcomings of the .ebextension Commands and Platform Commands that Napoli describes.
There seems to be some debate on whether or not this setup is officially supported, but judging by comments made on the AWS github, it seems to be not explicitly prohibited.
I can see where Napoli's answer could be the more standard MS way of doing things, but wpp.targets looks like hot trash IMO.
The general scheme of that answer is to use Commands/Platform commands to copy a script file into the appropriate platform hook directory (/opt/elasticbeanstalk/hooks or C:\Program Files\Amazon\ElasticBeanstalk\hooks\ ) to run at your desired stage of deployment.
I think its worth noting that differences exist between platforms and versions such as Amazon Linux 1 and Linux 2.
I hope this helps someone. It took me a day to gather that info and what's on this page and pick what I liked best.
Edit 11/4 - I would like to note that I saw some inconsistencies with the File .ebextension directive when trying to place scripts drirectly into the platform hook dir's during repeated deployments. Specifically the File directive failed to correctly move the backup copies named .bak/.bak1/etc. I would suggest using a Container Command to copy with overwriting from another directory into the desired hook directory to overcome this issue.
I source my environment specific variables from .env file (I'm using dotenv package). This file is not version controlled.
Using code pipeline stage codebuild how do I create this .env file and its contents ?
I'm thinking, I have to use buildspec to create and add content to the .env file, but have no idea how ?
Thanks
So this is actually a simple script, taking the second option of the first answer you can add the following to your buildspec.yml:
phases:
pre_build:
commands:
- printenv > .env
the shell script printenv > .env is all that is needed to output all of your process env to a file.
My suggestion would be to get rid of dotenv and use real environment variables.
The problem with dotenv is that it simply mocks process.env inside your node process. It does not really modify environment variables for you. They are not available outside of your node process.
I would use direnv instead and replace your .env files with .envrc for local development.
For CI and production environments, the environment variables are now defined in your CodeBuild project so .envrc files are not needed.
Please see this related answer for more information from another angle.
If you wish to continue using dotenv, you have several options:
Store the .env file in S3 and retrieve it during the build process.
Construct the .env file using shell scripts and environment variables defined on the CodeBuild project.
I'm sure there are others but they are all hacky for me.
This is Driving me nuts.
I had a working .ebextensions config file in my Project which was working fine.
Recently my single instance failed and a new one got initiated. My configuration failed to run so i tried to troubleshoot what went wrong. I didn't find anything suspicious so i just created a new .config with a very simple command but it still fails!!
I validated my config file with an online yaml validator.
I Connected to the instance through remote desktop and saw that .ebextensions folder is actually created within the wwwroot and then it disappears meaning that it got successfully picked up by elastic beanstalk.
I also granted all permissions to everyone on the test folder just to make sure this is not the reason.
Whichever i tried the old configuration or this test command it just does not work and elastic beanstalk just ignores it!
Any info of what might be wrong is appreciated.
commands:
01_Dowork:
command: mkdir kakarot
cwd: c:\\testdir
waitForCompletion: 0
I think everything under 01_DoWork needs to be indented (command, cwd, waitForCompletion). Also, make sure you're using spaces and not tabs.
Check the properties on your config file in VS. It should be (I think) both 'Content' and 'Copy if Newer'. Also, make sure that it gets packaged into the msdeploy package. It's a .zip file in/below your obj directory.
The command will error-out of it's already succeeded, so you would want to either ignore errors or add this. I found this syntax on another SO post but don't know who to credit for it :-/. The errorlevel will cause your command to not run if the directory already exists.
test: test ! -d c:\\testdir\\kakarot
If you're creating a package.zip (that inside has a deploy manifest json file plus the actual site.zip content) for a Windows deployment, it appears the .ebextensions directory needs to be inside package.zip, alongside the manifest json, not inside the site.zip, contrary to the current documentation.
I have the following directory layout
./www/index.php
Dockerfile
apache-config.conf
I then zip up the file and upload it onto Elastic Beanstalk, but I get the following error message:
[Instance: i-572d1ae8] Command failed on instance. Return code: 1 Output: Dockerfile and Dockerrun.aws.json are both missing, abort deployment. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03build.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
The log file contains the following message:
AppDeployPreHook/01unzip.sh] : Completed activity. Result:
Archive: /opt/elasticbeanstalk/deploy/appsource/source_bundle
creating: /var/app/current/DI_Test/
inflating: /var/app/current/DI_Test/.DS_Store
creating: /var/app/current/__MACOSX/
creating: /var/app/current/__MACOSX/DI_Test/
inflating: /var/app/current/__MACOSX/DI_Test/._.DS_Store
inflating: /var/app/current/DI_Test/apache-config.conf
inflating: /var/app/current/DI_Test/Dockerfile
inflating: /var/app/current/__MACOSX/DI_Test/._Dockerfile
creating: /var/app/current/DI_Test/www/
inflating: /var/app/current/DI_Test/www/.DS_Store
creating: /var/app/current/__MACOSX/DI_Test/www/
inflating: /var/app/current/__MACOSX/DI_Test/www/._.DS_Store
inflating: /var/app/current/DI_Test/www/index.php
[2015-10-29T14:01:33.279Z] INFO [2865] - [Application deployment/StartupStage0/AppDeployPreHook/03build.sh] : Starting activity...
[2015-10-29T14:01:33.579Z] INFO [2865] - [Application deployment/StartupStage0/AppDeployPreHook/03build.sh] : Activity execution failed, because: Dockerfile and Dockerrun.aws.json are both missing, abort deployment (ElasticBeanstalk::ExternalInvocationError)
caused by: Dockerfile and Dockerrun.aws.json are both missing, abort deployment (Executor::NonZeroExitStatus)
Why is my dockerfile not being read here? I can build the image and run it just fine without Elastic Beanstalk.
Sharing another possibility:
You were testing eb and you've just added your Dockerfile.
According to the doc
If git is installed, EB CLI uses the git archive command to create a
.zip file from the contents of the most recent git commit command.
That's saying if your Dockerfile is not committed, it won't be zipped in therefore your instance can't find it.
Compress those stuff within your project folder(go into your project folder and zip everything), instead of zipping your project folder.
You don't need to include parent directory in zip file. Suppose you want to zip ./www/index.php ,Dockerfile and apache-config.conf then,
zip dist.zip Dockerfile ./www/index.php apache-config.conf
The upload dist.zip.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.deployment.source.html
I've come across this issue as well. It seems a Dockerrun.aws.json file is required as well. I believe that is because other files are included in the app. If you had only a Dockerfile and nothing else, it wouldn't be required.
Here's the documentation: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html
In my case there was a spelling mistake in Dockerfile name i.e DockerFile. Elastic Beanstalk is case sensitive.
Super newbie mistake, but when I had this problem it was because my Dockerfile was actually Dockerfile.txt. After removing the .txt extension and creating a plain file it worked fine.
I had the same issue. The solution is to go to the actual directory that you are trying to zip and then run: zip name_of_file.zip *. Then the Dockerfile will get the exact same zip filename "Dockerfile", which is what Elastic Beanstalk is looking for.
If you zip from outside your directory, the Dockerfile's name will be "directory/Dockerfile", and that is what causes Elastic Beanstalk to fail.
I had a different problem. Using directly json, the Dockerrun files can have the form of Dockerrun_${DOCKER_TAG}.aws.json.
Solution: In zips, only Dockerrun.aws.json is allowed.
Two possible problems:
1) When zipping the folder enter the folder and zip from the command line or in finder select each file individually then zip together.
Example: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/applications-sourcebundle.html
2) Elastic Beanstalk is case sensitive so make sure your file is labeled "Dockerfile" not "dockerfile"
I was trying to upload a zip which contained a singular file Dockerrun.aws.json. But docs state:
If you use only a Dockerfile or only a Dockerrun.aws.json file to deploy your application, you don't need to create a .zip file
So you can simply upload either one or another without zipping it.
To document just another way to get into this - added by mistake Dockerfile in the .ebignore file. I was testing non-docker and docker deploys and worked from the same directory. Dockerfile is not obviously needed with regular deploys but should not be added to .ebignore when doing docker deploys.
I added the dockerrun.aws.json and config.yml in a folder called .elasticbeanstalk/ which is in the root folder.