I am provisioning a server with a Django Stack via Ansible and getting the app from bitbucket, I am using https://github.com/jcalazan/ansible-django-stack, but I have had to tweak it a bit in order to make it work with a private bitbucket repo.
Now it's authenticating correctly but giving me the following error
failed: [default] => {"failed": true} msg: youtubeadl: ERROR (not
running) youtubeadl: ERROR (abnormal termination)
When performing this task:
- name: Restart Supervisor
supervisorctl: name={{ application_name }} state=restarted
Reading gunicorn ERROR (abnormal termination), I would like to add the project to the PYTHONPATH, any ideas how to approach this with an Ansible task, or am I missing something?
Thanks
PYTHONPATH is just another environment variable, so you can use the best practices explained in the FAQ. If it's only needed for the one task, it'd look something like:
- name: Restart Supervisor
supervisorctl: name={{ application_name }} state=restarted
environment:
PYTHONPATH: "{{ ansible_env.PYTHONPATH }}:/my/path"
Something changed. I tried this answer above, but doesn't works. After some digging, and trying:
- name: Restart Supervisor
supervisorctl: name={{ application_name }} state=restarted
environment:
PYTHONPATH: "{{ ansible_env.PATH }}:/my/path"
This should be correct answer.
Related
I develop in local k8s cluster with minikube and skaffold. Using Django and DRF for the API.
I'm working on a number of models.py and one thing that is starting to get annoying is anytime I run a ./manage.py command like (showmigrations, makemigrations, etc.) it triggers a skaffold rebuild of the API nodes. It takes less than 10 seconds, but getting annoying none the less.
What should I exclude/include specifically from my skaffold.yaml to prevent this?
apiVersion: skaffold/v2beta12
kind: Config
build:
artifacts:
- image: postgres
context: postgres
sync:
manual:
- src: "**/*.sql"
dest: .
docker:
dockerfile: Dockerfile.dev
- image: api
context: api
sync:
manual:
- src: "**/*.py"
dest: .
docker:
dockerfile: Dockerfile.dev
local:
push: false
deploy:
kubectl:
manifests:
- k8s/ingress/development.yaml
- k8s/postgres/development.yaml
- k8s/api/development.yaml
defaultNamespace: development
It seems that ./manage.py must be recording some state locally, and thus triggering a rebuild. You need to add these state files to your .dockerignore.
Skaffold normally logs at a warning level, which suppresses details of what triggers sync or rebuilds. Run Skaffold with -v info and you'll see more detail:
$ skaffold dev -v info
...
[node] Example app listening on port 3000!
INFO[0336] files added: [backend/src/foo]
INFO[0336] Changed file src/foo does not match any sync pattern. Skipping sync
Generating tags...
- node-example -> node-example:v1.20.0-8-gc9335b0ad-dirty
INFO[0336] Tags generated in 80.293621ms
Checking cache...
- node-example: Not found. Building
INFO[0336] Cache check completed in 1.844615ms
Found [minikube] context, using local docker daemon.
Building [node-example]...
I am trying to add a tensorboard container to an existing microservice structure running behind traefik. Unfortunately, the traefik version is 1.5 so a lot of recent articles are not helpful.
Since there is a default service on www.my-server.com/, I am trying to have traefik redirect to the tensorboard service from www.my-server.com/tensorboard/. Here is my docker-compose (the part relevant for tensorboard)
tensorboard:
build: "./docker/build/images/tensorflow"
container_name: tensorboard
command: tensorboard --logdir=runs --port=8888 --host=0.0.0.0
labels:
- log.level=debug
- traefik.enable=true
- traefik.frontend.rule=Host:www.my-server.com;PathPrefix:/tensorboard/
volumes:
- ./machine_learning:/opt/src
ipc: host
When I visit www.my-server.com/tensorboard/ I get "Not Found". If I remove the host argument from the command I get "Bad Gateway". I don't understand what either of these mean but I think one of them is being able to reach the service but the service is getting the request with the prefix tensorboard and is complaining.
How do I make this work?
Turns out that the following command will solve this problem
tensorboard --logdir mylogdir --bind_all --path_prefix=/tensorboard
I'm currently setting up a build/test pipeline for my app (django) using Google Cloud Build (and testing using cloud-build-local).
In order to properly run the tests I need to start a mysql dependency (I use docker-compose for this ). The issue is that when running docker-compose in a cloud-build step, database init scripts are not properly run, I get a
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0-init.sql
ERROR: Can't initialize batch_readline - may be the input source is a directory or a block device.
(running docker-compose out of google-cloud-build properly works)
Here's my docker-compose file:
version: '3.3'
services:
mysql:
image: mysql:5.7
restart: always
environment:
MYSQL_DATABASE: 'dev'
MYSQL_USER: 'dev'
MYSQL_PASSWORD: 'dev'
MYSQL_ROOT_PASSWORD: 'root'
ports:
- '3306:3306'
expose:
- '3306'
volumes:
- reports-db:/var/lib/mysql-reports
- ./dev/databases/init.sql:/docker-entrypoint-initdb.d/0-init.sql
- ... (other init scripts)
volumes:
reports-db:
And cloudbuild.yaml :
steps:
...
- id: 'tests-dependencies'
name: 'docker/compose:1.24.1'
args: ['up', '-d']
...
Files being organized like this:
parent_dir/
dev/
databases/
init.sql
cloudbuild.yaml
docker-compose.yml
...
(all commands are run from parent_dir/)
When I run
cloud-build-local --config=cloudbuild.yaml --dryrun=false .
I get a
...
Step #2 - "tests-dependencies": mysql_1 | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/0-init.sql
Step #2 - "tests-dependencies": mysql_1 | ERROR: Can't initialize batch_readline - may be the input source is a directory or a block device.
...
Knowing that running docker-compose up directly works properly
I'm suspecting that the way volumes are mounted is incorrect but can't find why/how.
If anyone has any input on this it will be really useful :)
Thanks in advance.
Looks like it's an issue proper to cloud-build-local, properly works on GCP
I'm setting up circle-ci to automatically build/deploy to AWS ECR &ECS.
But build is failed due to no Dockerfile.
Maybe this is because I set docker-compose for multiple docker images.
But I don't know how to resolve this issue.
Is there no way to make DockerFile instead of docker-compose?
front: React
backend: Golang
ci-tool: circle-ci
db: mysql
article
├ .circleci
├ client
├ api
└ docker-compose.yml
I set .circleci/config.yml.
version: 2.1
orbs:
aws-ecr: circleci/aws-ecr#6.0.0
aws-ecs: circleci/aws-ecs#0.0.8
workflows:
build_and_push_image:
jobs:
- aws-ecr/build-and-push-image:
region: AWS_REGION
account-url: AWS_ECR_ACCOUNT_URL
repo: 'article-ecr-jpskgc'
tag: '${CIRCLE_SHA1}'
- aws-ecs/deploy-service-update:
requires:
- aws-ecr/build-and-push-image
family: 'article-task-jpskgc'
cluster-name: 'article-cluster-jpskgc'
service-name: 'article-service-jpskgc'
container-image-name-updates: 'container=article-container-jpskgc,tag=${CIRCLE_SHA1}'
Here is the source code in github.
https://github.com/jpskgc/article
I expect build/deploy via circle-ci to ECR/ECS to success, but it actually fails.
This is the error log on circle-ci.
Build docker image
Exit code: 1
#!/bin/bash -eo pipefail
docker build \
\
-f Dockerfile \
-t $AWS_ECR_ACCOUNT_URL/article-ecr-jpskgc:${CIRCLE_SHA1} \
.
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/circleci/project/Dockerfile: no such file or directory
Exited with code 1
You must use a Dockerfile, check out the documentation for the orb you are using. Please read through them here. Also docker-compose ≠ docker, therefore I will confirm that one cannot be used in substitution for the other.
Given your docker-compose.yml, I have a few suggestions for your general setup and CI.
For reference here is the docker-compose.yml in question:
version: '3'
services:
db:
image: mysql
ports:
- '3306:3306'
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: article
MYSQL_USER: docker
MYSQL_PASSWORD: docker
nginx:
restart: always
build:
dockerfile: Dockerfile.dev
context: ./nginx
ports:
- '3050:80'
api:
build:
dockerfile: Dockerfile.dev
context: ./api
volumes:
- ./api:/app
ports:
- 2345:2345
depends_on:
- db
tty: true
environment:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
client:
build:
dockerfile: Dockerfile.dev
context: ./client
volumes:
- /app/node_modules
- ./client:/app
ports:
- 3000:3000
From the above we have the various components, just as you have stated:
MySQL Database
Nginx Loadbalancer
Client App
API Server
Here are my recommendations for each component:
MySQL Database
Since you are deploying to AWS I recommend deploying a MySQL instance on the free tier, please follow this documentation: https://aws.amazon.com/rds/free. With this you can remove your database from CI, which is recommended as ECS is not the ideal service to run a MySQL server.
Nginx Loadbalancer
Because you are using ECS, this is not required as AWS handles all load balancing for you and is redundant.
Client App
Because this is a react application, you shouldn't deploy to ECS -- this is not cost effective you would rather deploy this to Amazon S3. There are many resources on how to do this. You may follow this guide though you may have to make a few change based of the structure of your repository.
This will reduce your overall cost and it makes more sense than an entire Docker container running just to serve static files.
API Server
This is the only thing that should be running in ECS, and all you need to do is point to the correct Dockerfile in your configuration for it be built and pushed successfully.
You may therefore edit your circle ci config as follows, assuming we are using the same Dockerfile in your docker-compose.yml:
build_and_push_image:
jobs:
- aws-ecr/build-and-push-image:
region: AWS_REGION
dockerfile: Dockerfile.dev
path: ./api
account-url: AWS_ECR_ACCOUNT_URL
repo: 'article-ecr-jpskgc'
tag: '${CIRCLE_SHA1}'
Things to Note
My answer does not include:
How to load balance your API service please follow these docs on how to do so: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html
Details on setting up the MySQL server, it assumed you will follow the AWS documentation provided above.
Things you must do:
Point your client app to the API server, this will probably require a code change from what I've seen.
I want to stress that you must Load balance your API server according to these docs yet again.
You do not need to edit your docker-compose.yml
I'm trying to provision my infrastructure on AWS using Ansible playbooks. I have the instance, and am able to provision docker-engine, docker-py, etc. and, I swear, yesterday this worked correctly and I haven't changed the code since.
The relevant portion of my playbook is:
- name: Ensure AWS CLI is available
pip:
name: awscli
state: present
when: aws_deploy
- block:
- name: Add .boto file with AWS credentials.
copy:
content: "{{ boto_file }}"
dest: ~/.boto
when: aws_deploy
- name: Log in to docker registry.
shell: "$(aws ecr get-login --region us-east-1)"
when: aws_deploy
- name: Remove .boto file with AWS credentials.
file:
path: ~/.boto
state: absent
when: aws_deploy
- name: Create docker network
docker_network:
name: my-net
- name: Start Container
docker_container:
name: example
image: "{{ docker_registry }}/example"
pull: true
restart: true
network_mode: host
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone
My {{ docker_registry }} is set to my-acct-id.dkr.ecr.us-east-1.amazonaws.com and the result I'm getting is:
"msg": "Error pulling my-acct-id.dkr.ecr.us-east-1.amazonaws.com/example - code: None message: Get http://: http: no Host in request URL"
However, as mentioned, this worked correctly last night. Since then I've made some VPC/subnet changes, but I'm able to ssh to the instance, and run docker pull my-acct-id.dkr.ecr.us-east-1.amazonaws.com/example with no issues.
Googling has led me not very far as I can't seem to find other folks with the same error. I'm wondering what changed, and how I can fix it! Thanks!
EDIT: Versions:
ansible - 2.2.0.0
docker - 1.12.3 6b644ec
docker-py - 1.10.6
I had the same problem. Downgrading docker-compose pip image on that host machine from 1.9.0 to 1.8.1 solved the problem.
- name: Install docker-compose
pip: name=docker-compose version=1.8.1
Per this thread: https://github.com/ansible/ansible-modules-core/issues/5775, the real culprit is requests. This fixes it:
- name: fix requests
pip: name=requests version=2.12.1 state=forcereinstall