How to set environmental variables properly Gitlab CI/CD and Docker - django

Question

I am new to Docker and CI/CD with Gitlab CI/CD. I have .env file in the root directory of my Django project which contains my environmental variable e.g SECRET_KEY=198191891. The .env file is included in .gitignore. I have set up these variables in Gitlab settings for CI/CD. However, the environment variables set in Gitlab CI/CD settings seem to be unavailable
Also, how should the Gitlab CI/CD automation process create a user and DB to connect to and run the tests? When creating the DB and user for the DB on my local machine, I logged into the container docker exec -it <postgres_container_name> /bin/sh and created Postgres user and DB.
Here are my relevant files.
docker-compose.yml
version: "3"
services:
postgres:
image: postgres
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data/
web:
build: .
command: /usr/local/bin/gunicorn writer.wsgi:application -w 2 -b :8000
environment:
DEBUG: ${DEBUG}
DB_HOST: ${DB_HOST}
DB_NAME: ${DB_NAME}
DB_USER: ${DB_USER}
DB_PORT: ${DB_PORT}
DB_PASSWORD: ${DB_PASSWORD}
SENDGRID_API_KEY: ${SENDGRID_API_KEY}
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
AWS_STORAGE_BUCKET_NAME: ${AWS_STORAGE_BUCKET_NAME}
depends_on:
- postgres
- redis
expose:
- "8000"
volumes:
- .:/writer-api
redis:
image: "redis:alpine"
celery:
build: .
command: celery -A writer worker -l info
volumes:
- .:/writer-api
depends_on:
- postgres
- redis
celery-beat:
build: .
command: celery -A writer beat -l info
volumes:
- .:/writer-api
depends_on:
- postgres
- redis
nginx:
restart: always
build: ./nginx/
ports:
- "80:80"
depends_on:
- web
volumes:
pgdata:
.gitlab-ci.yml
image: tmaier/docker-compose:latest
services:
- docker:dind
before_script:
- docker info
- docker-compose --version
stages:
- build
- test
- deploy
build:
stage: build
script:
- echo "Building the app"
- docker-compose build
test:
stage: test
variables:
script:
- echo "Testing"
- docker-compose run web coverage run manage.py test
deploy-staging:
stage: deploy
only:
- develop
script:
- echo "Deploying staging"
- docker-compose up -d
deploy-production:
stage: deploy
only:
- master
script:
- echo "Deploying production"
- docker-compose up -d
Here are my settings for my variables
Here is my failed pipeline job

Answer 1

The SECRET_KEY variable will be available to all your CI jobs, as configured. However, I don't see any references to it in your Docker Compose file to pass it to one or more of your services. For the web service to use it, you'd map it in like the other variables you already have.
web:
build: .
command: /usr/local/bin/gunicorn writer.wsgi:application -w 2 -b :8000
environment:
SECRET_KEY: ${SECRET_KEY}
DEBUG: ${DEBUG}
…
As for creating the database, you should wrap up whatever you currently run interactively in the postgres container in a SQL file or shell script, and then bind-mount it into the container's initialization scripts directory under /docker-entrypoint-initdb.d. See the Initialization scripts section of the postgres image's description for more details.

Answer 2

In my experience the best way to pass environment variables to a container docker is creating an environment file, which works to development environment and production environment:
GitLab CI/CD variables
You must create an environment file on GitLab CI/CD, following the next steps, on your project GitLab you must go to:
settings > CI/CD > Variables
on that you must create a ENV_FILE
Demo image
Next on your build stage in .gitlab-ci.yml copy the ENV_FILE to local process
.gitlab-ci.yml
build:
stage: build
script:
- cp $ENV_FILE .env
- echo "Building the app"
- docker-compose build
your Dockerfile should be stay normally so it doesn't have to change
Dockerfile
FROM python:3.8.6-slim
# Rest of setup goes here...
COPY .env .env

Answer 3

In order for compose variable substitution to work, when the user is not added to docker group, you must add the -E flag to sudo.
script:
- sudo -E docker-compose build