I have a django application running locally and i've set up the project on CircleCi with python and postgres images.
If I understand correctly what is happening, CircleCi would use the images to build a container to test my application with code database.
Then I'm using the job heroku/deploy-via-git to deploy it to Heroku when the tests are passed.
Now I think Heroku is using some images too to run the application.
I would like to get the image used by heroku to run my site locally on another machine.
So pull the image then push it to Docker Hub and finally download it back to my computer to only have to use a docker compose up.
Here is my CircleCi configuration's file
version: 2.1
docker-auth: &docker-auth
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
orbs:
python: circleci/python#1.5.0
heroku: circleci/heroku#0.0.10
jobs:
build-and-test:
docker:
- image: cimg/python:3.10.2
- image: cimg/postgres:14.1
environment:
POSTGRES_USER: theophile
steps:
- checkout
- run:
command: pip install -r requirements.txt
name: Install Deps
- run:
name: Run MIGRATE
command: python manage.py migrate
- run:
name: Run loaddata from Json
command: python manage.py loaddata datadump.json
- run:
name: Run tests
command: pytest
workflows:
heroku_deploy:
jobs:
- build-and-test
- heroku/deploy-via-git:
requires:
- build-and-test
I don't know if it is possible, if not, what should be the best way to proceed ? (I assume that there is a lot of possibilites)
I was considering to build an image from my local directory with docker compose up then use this image direclty on CircleCi, then i would be able to use this image an on other computer. But building images into images with CircleCi seems really messy and I'm not sure how I should proceed.
I've tried to pull images from Heroku but it seems I can only pull the code or get/modify the database but I can't get the image builds itself.
I hope this question is relevant and clear, as the CircleCi and Heroku documentation seems not clear and it's my first post on stackoverflow !
Thanks in advance
Heroku's platform is proprietary, so we can't be sure how it works internally.
We know that their stacks are based on Ubuntu LTS releases, and we know that they use open-source buildpacks to compile application slugs from source code, but details about the underlying infrastructure are murky. They certainly don't provide base images like heroku/python:3.11.0 for you to download.
If you want to use the same image locally, on CircleCI, and Heroku, a better option would be to start deploying with Heroku's Container Registry instead of Git. This allows you to build an image locally, push it into the container registry, and release it as the next version of your application.
I suggest you read the entire documentation page linked above, but the short version is:
Log into the container registry using the Heroku CLI:
heroku container:login
Assuming you already have a Dockerfile for your application, build and push an image:
heroku container:push web
In this case we are building from Dockerfile and pushing the resulting image to be used as a web process.
Release your application:
heroku container:release web
That's a basic Docker deployment from your local machine, and even if that's not your final plan I suggest you start by getting that working.
From there, you have options. One option would be to move this flow to CircleCI—continue to build images there, but have CircleCI push the resulting container to Heroku's Container Registry.
Another option might be as you suggest in your question: to build images locally and use them with both CircleCI and Heroku.
I have followed the following article on getting my unit tests working in a Docker image and publishing via Azure DevOps pipeline.
Running Your Unit Tests With Visual Studio Team Services and Docker Compose
Each one of my unit tests projects have a very basic Dockerfile:
First Dockerfile for Application Tests
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
COPY . /app
WORKDIR /app/Application.Tests
RUN dotnet restore
Second DockerFile for Infrastructure tests.
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
COPY . /app
WORKDIR /app/Infrastructure.Tests
RUN dotnet restore
My docker-compose for both images:
application.tests:
image: ${DOCKER_REGISTRY-}applicationtests
build:
context: .
dockerfile: Application.Tests/Dockerfile
entrypoint: >
dotnet test --results-directory /testresults --logger trx
/p:CollectCoverage=true
/p:CoverletOutputFormat=cobertura
/p:CoverletOutput=/testresults/Application.Tests.cobertura.xml
/p:Exclude="[xunit.*]"
volumes:
- /opt/vsts/work/_temp:/testresults
infrastructure.tests:
image: ${DOCKER_REGISTRY-}infrastructuretests
build:
context: .
dockerfile: Infrastructure.Tests/Dockerfile
entrypoint: >
dotnet test --results-directory /testresults --logger trx
/p:CollectCoverage=true
/p:CoverletOutputFormat=cobertura
/p:CoverletOutput=/testresults/Infrastructure.Tests.cobertura.xml
/p:Exclude="[xunit.*]"
volumes:
- /opt/vsts/work/_temp:/testresults
Is there a way to combine the definition for each image in docker-compose? I understand that I can combine the tests into a single project, but want to maintain a 1:1 relationship between project and test project.
I should also add that both projects are in the same solution (.NET Core).
Is there a way to run the tests for all projects in a single
azure-pipeline task?
It's not available with DockerCompose task. According to your scenario, you should avoid combining multiple containers into one image. As the answer in this thread pointed that images should be kept light, and run one service per container.
Docker Compose Task is just the way to manage these services with multiple containers to run in your application. .
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration
If you insist on combing everything together, Docker compose is not fit for your case.
You could have a try with multiple stage dockerfile to combine these dockerfiles into one. This will end up with a single image when your run docker build
Currently our company is creating individual software for B2B customers.
Some applications can be used for multiple customers.
Usually we can host the application in the cloud and deploy everything with Docker.
Running a GitLab pipeline and deploying etc. is fine for that.
Now we got some customers who rely on an external installation.
Since some of them still use Windows Server (2008 tho), I can not install a proper Docker environment on there and we need to install an Apache Tomcat and run the application inside the tomcat.
Question: How to deal with that? I would need a pipeline to create a docker image and a war file.
Simply create two completely independent pipelines?
Handle everything in a single pipeline?
Our current gitlab-ci.yml file for the .war
image: maven:latest
variables:
MAVEN_CLI_OPTS: "-s settings.xml -q -B"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
cache:
paths:
- .m2/repository/
- target/
stages:
- build
- test
- deploy
build:
stage: build
script:
- mvn $MAVEN_CLI_OPTS compile
test:
stage: test
script:
- mvn $MAVEN_CLI_OPTS test
install:
stage: deploy
script:
- mvn $MAVEN_CLI_OPTS install
artifacts:
name: "datahub-$CI_COMMIT_REF_SLUG"
paths:
- target/*.war
Using to separate delivery pipeline is preferable: you are dealing with two very installation processes, and you need to be sure which one is running for a given client.
Having two separate GitLab pipeline allows for said client to chose the right one.
What I am trying to do is to enable Continuous delivery from GitLab to my compute engine on Google Cloude. I have Ubuntu 16.04 TSL running over there. I did install all components needed to run my project like: Swift, vapor, nginx.
I have manage to install Gitlab runner as well and created a runner whcihc is accessible from my gitlab repo. Everytime I do push on master the runner triggers. What happen is a failure due to:
could not create leading directories of '/home/gitlab-runner/builds/2bbbbbd/0/Server/Packages/vapor.git': Permission denied
If I change the permissions to chmod -R 777 It will hange on running for build stage visible on gitlab pipeline.
I did something like:
sudo chown -R gitlab-runner:gitlab-runner /home/gitlab-runner/builds
sudo chown -R gitlab-runner:gitlab-runner /home/gitlab-runner/cache
but this haven't help, the error is same Permission denied
Below you have my .gitlab-ci.yml
before_script:
- swift --version
stages:
- build
- deploy
job_build:
stage: build
before_script:
- vapor clean
script:
- vapor build --release
only:
- master
job_run_app:
stage: deploy
script:
- echo "Deploy a API"
- vapor run --name=App --env=production
environment:
name: production
job_run_frontend:
stage: deploy
script:
- echo "Deploy a Frontend"
- vapor run --name=Frontend --env=production
environment:
name: production
But that haven't pass to next stage eg. deploy. I had waited more then 14h for that but with out result.
And... I have few more questions:
Gitlab runner creates builds under location /home/gitlab-runner/builds/ in this location every new job have own folder. for eg. /home/gitlab-runner/builds/2bbbbbd/ in which is my project and the commands are executed. So what happens when the first one is running and I do deploy new version? the ports are blocked by the first instance and so on?
If I want to enable supervisor how do I do that with this when every time I deploy folder is different?
Can anyone explain or show me or point me to tutorial how do Continuous deployment with out docker?
How to start a service using GitLab runner
Thanks to long deep search I finally found an answer! The full article can be found above.
Briefly GitLab CI documentation recommends using dpl for deployment. Gitlab runner run test and process should end. The runner is designed to kill all created processes after finishing each build. The GitLab runner is unable to perform operations outside the catalogue.
I've set up a docker django container and made its image using build using tutorial here. The tutorial shows how to make a basic django application and mounts the application to "/code", which as I understand, is contained in a data-volume.
However I want to understand that how will I be able to update and develop this code, and be able to ship/deploy it. Since when I make a commit, it's doesn't take account any changes in the code, since it's a part of the data volume.
Is there any way I can make the django code a part of the image, or update the image with the updated code?
In my experience Docker serves two purposes:
To be able to develop code in a containerized environment. This is very useful as I am now able to get new developers on my team ready to work in about 5 mins Previously, this could have taken anywhere from an hour to several hours for misc issues, especially on older projects.
To be able to package an application in a containerized environment. This is also a great time saver as the only requirement for the environment is to have Docker installed.
When you are developing your code you should mount the source/volume so that your changes are always reflected inside the container. When you want to package an app for deployment you should COPY the source into the container and package it appropriately.
Here is a docker-compose file I use to (1) build an image to develop against, (2) develop my code, and (3) ship it (I'm using spring boot):
version: '3.7'
services:
dev:
image: '${MVN_BUILDER}'
container_name: '${CONTAINER_NAME}'
ports:
- '8080:8080'
volumes:
- './src:/build/src'
- './db:/build/db'
- './target:/build/target'
- './logs:/build/logs'
command: 'mvn spring-boot:run -Drun.jvmArguments="-Xmx512m" -Dmaven.test.skip=true'
deploy:
build:
context: .
dockerfile: Dockerfile-Deploy
args:
MVN_BUILDER: '${MVN_BUILDER}'
image: '${DEPLOYMENT_IMAGE}'
container_name: '${CONTAINER_NAME}'
ports:
- '8080:8080'
maven:
build:
context: .
dockerfile: Dockerfile
image: '${MVN_BUILDER}'
container_name: '${CONTAINER_NAME}'
I would run docker-compose build maven to build my base image. This is needed so that when I run my code in a container all the dependencies are installed in the image. The Dockerfile for this essentially copies to pom.xml into the image and downloads the dependencies needed for the app. Note this would need to be performed anytime dependencies change. Here is the Dockerfile to build that image that is referenced in the maven service:
### BUILD a maven builder. This will contain all mvn dependencies and act as an abstraction for all mvn goals
FROM maven:3.5.4-jdk-8-alpine as builder
#Copy Custom Maven settings
#COPY settings.xml /root/.m2/
# create app folder for sources
RUN mkdir -p /build
RUN mkdir -p /build/logs
# The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile.
WORKDIR /build
COPY pom.xml /build
#Download all required dependencies into one layer
RUN mvn -B dependency:go-offline dependency:resolve-plugins
RUN mvn clean install
Next, I would run docker-compose up dev to start my dev service and begin to develop my application. This service mounts my code into the container and uses Maven to start a spring boot application. Anytime I change the code spring boot restarts the server and my changes are reflected.
Finally, once I am happy with my application I build an image that has my application packaged for deployment using docker-compose build deploy. I use a two-stage build process to first copy the source into a container and package it for deployment as a Jar then that Jar is put into the 2nd stage where I can simply run java -jar build/app.jar (in the container) to start my application and the first stage is removed. That's It! Now you can deploy this latest image anywhere Docker is installed.
Here is what that last Dockerfile (Dockerfile-Deploy) looks like:
ARG MVN_BUILDER
### Stage 1 - BUILD image
FROM $MVN_BUILDER as builder
COPY src /build/src
RUN mvn clean package -PLOCAL
### Stage 2 - Deploy Jar
FROM openjdk:8
RUN mkdir -p /build
COPY --from=builder /build/target/*.jar /build/app.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","build/app.jar"]
Here the .env file in the same directory as the docker-compose file. I use it to abstract image/container names and simply bump up the version number in one place when a new image is needed.
MVN_BUILDER=some/maven/builder:0.1
DEPLOYMENT_IMAGE=some/deployment/spring:0.1
CONTAINER_NAME=spring-container
CONTAINER_NAME_DEBUG=spring-container-debug
I think it's too late to answer your question, however, it might be beneficial for others who reach out.
The tutorial you mentioned is a bit tricky to use for the first-timers, so, I change the structure a little bit. I assume that you have a docker registry account (like Dockerhub) for the purpose of publishing the images to. This is required if you want to access the image on a remote host (you can copy the actual image file but is not recommended).
creating a project
Assume that you are going to create a website with Django and dockerize it, first, you do:
django-admin startproject samplesite
It creates a directory samplesite that includes the following (I added requirements.txt):
db.sqlite3 manage.py requirements.txt samplesite
adding Dockerfile and docker-compose.yml
For the Dockerfile, as you can see, nothing is changed compared to the Dockerfile.
FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
However for the docker-compose.yml:
version: '3'
services:
db:
image: postgres
web:
build: .
image: yourUserNameOnDockerHub/mywebsite:0.1 # this line is added
command: python manage.py runserver 0.0.0.0:8000
#volumes:
# - .:/code
ports:
- "8000:8000"
depends_on:
- db
docker-compose.yml is also almost identical to the one mentioned in the tutorial, with volume commented and one line added image: mywebsite:0.1. This line allows us to track the built image and deploy it whenever we want. The volume mounting is not related to the code you write and was put there to take out the dynamic content that is changed by Django (sqlite, uploaded files, etc.).
build and run
If you run docker-compose up the first time everything works fine, however, because of the new line added, when you change the code after the first time, the changes will not reflect in the container that runs. This is because upon each docker-compose up, compose will look for mywebsite:0.1 (that already exists) and does not build a new image and creates a container based on the old one. As we need that image name and tag to publish/deploy our image, we need to instead use:
docker-compose up --build
It will re-build an image with the changes reflected. Every time you make some changes, run it and a new fresh image is created that can be seen with (note that although the name and tag remain unchanged, change in image id shows that this is a new image):
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
yourUserNameOnDockerHub/mywebsite 0.1 033c9d2bfac0 7 seconds ago 974MB
publishing and deployment
If you have set up an account on Dockerhub (or any other registry) you can publish the image for later use or deployment on a remote server:
docker push yourUserNameOnDockerHub/mywebsite:01
If you want to deploy it on a remote host and want to use docker-compose again, Just change the docker-compose.yml to:
version: '3'
services:
db:
image: postgres
web:
image: yourUserNameOnDockerHub/mywebsite:0.1
command: python manage.py runserver 0.0.0.0:8000
#volumes:
# - .:/code
ports:
- "8000:8000"
depends_on:
- db
Note that the build: . line is removed (as we are going to run it only). When developing locally, whenever you run docker-compose up --build a new image will be created and tagged and a container based on that will run in the compose stack. If you thought that you are happy with the changes, you follow the publishing step to make it live on the server.
When you want to update an image, lets say due to your application code changes, you use COPY during your image-build, so in the Dockerfile you do something like
COPY /you/code/on/the/host /var/www
Also see my answer about "volumes" and image-building https://stackoverflow.com/a/39314602/3625317 to clarify why your code is missing in the build
In step 9 of the tutorial you set a volume. This volume will link your current directory and your container /code directory. In other words, they will be the same.
So any updates to your local files will change the files in your container as well. Remember that you will need to restart your app so the changes can take place.
Before you deploy your image, you will need to create a second docker compose file. This file will remove the volume so the code will stay inside the container and won't change from outside. You can follow the steps provided in docker compose documentation.