Building elasticsearch indexes from another container - django

I have a Django project that uses django-elasticsearch-dsl. The project is dockerized, so elasticsearch and the web projects leave in separate containers.
Now my goal is to recreate and repopulate the indices running
python manage.py search_index --rebuild
In order to do that, I try to run the command from the container of the web service the following way:
docker-compose exec web /bin/bash
> python manage.py search_index --rebuild
Not surprsiginly, I get an error
Failed to establish a new connection: [Errno 111] Connection refused)
apparently because python tried to connect to elasticsearch using localhost:9200.
So the question is, how do I tell the management command the host where elasticsearch lives ?
Here's my docker-compose.yml file:
version: '2'
services:
web:
build: .
restart: "no"
command: ["python3", "manage.py", "runserver", "0.0.0.0:8000"]
env_file: &envfile
- .env
environment:
- DEBUG=True
ports:
- "${DJANGO_PORT}:8000"
networks:
- deploy_network
depends_on:
- elasticsearch
- db
elasticsearch:
image: 'elasticsearch:2.4.6'
ports:
- "9200:9200"
- "9300:9300"
networks:
- deploy_network
db:
image: "postgres"
container_name: "postgres"
restart: "no"
env_file: *envfile
ports:
- "5432:5432"
volumes:
- db_data:/var/lib/postgresql/data
volumes:
db_data:
networks:
deploy_network:
driver: bridge
UPDATE:
In the Django project's settings I setup the elasticsearch dsl host:
# settings.py
ELASTICSEARCH_DSL = {
'default': {
'hosts': 'localhost:9200'
}
}

Since your Django project and Elasticsearch are in 2 separate containers, setting ELASTICSEARCH_DSL's host to 'localhost:9200' won't work, in this case localhost refers to localhost inside Django container.
So you need to set it like this:
# settings.py
ELASTICSEARCH_DSL = {
'default': {
'hosts': 'elasticsearch:9200'
}
}

Related

How to deploy django based website running on a docker compose to my local home network?

I have the following setup:
docker-compose.yml
# Mentioning which format of dockerfile
version: "3.9"
# services or nicknamed the container
services:
# web service for the web
web:
# you should use the --build flag for every node package added
build: .
# Add additional commands for webpack to 'watch for changes and bundle it to production'
command: python manage.py runserver 0.0.0.0:8000
volumes:
- type: bind
source: .
target: /code
ports:
- "8000:8000"
depends_on:
- db
environment:
- "DJANGO_SECRET_KEY=django-insecure-m#x2vcrd_2un!9b4la%^)ou&hcib&nc9fvqn0s23z%i1e5))6&"
- "DJANGO_DEBUG=True"
expose:
- 8000
db:
image: postgres:13
#
volumes:
- postgres_data:/var/lib/postgresql/data/
# unsure of what this environment means.
environment:
- "POSTGRES_HOST_AUTH_METHOD=trust"
# - "POSTGRES_USER=postgres"
# Volumes set up
volumes:
postgres_data:
and a settings file as
ALLOWED_HOSTS = ['0.0.0.0', 'localhost', '127.0.0.1']
#127.0.0.1 is my localhost address.
With my host's IP as 192.168.0.214
Can you please help me deploy the django site on my host's local network?
Do I have to set up something on my router?
Or could you direct me towards resources(understanding networking) which will help me understand the same.

Unable to connect to Postgres DB living in one docker-compose file with django app in separate docker-compose file

I have a large monorepo Django app that I want to break into two separate repositories (1 to handle external api requests and the other to handle my front end that I plan on showing to users). I would still like to have both django apps have access to the same db when running things locally. Is there a way for me to do this? I'm running docker for both and am having issues with my front end facing django app being able to connect to the Postgres DB i have set up in a separate docker-compose file than the one I made for my front end app.
External API docker-compose file (Postgres DB docker image gets created here when running docker-compose up --build)
---
version: "3.9"
services:
db:
image: postgres:13.4
ports:
- "5432:5432"
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
api:
restart: always
build: .
image: &img img-one
command: bash start.sh
volumes:
- .:/app
ports:
- "8000:8000"
depends_on:
- db
env_file:
- variables.env
Front end facing docker-compose file (This is the one I want to be able to connect to the DB above):
---
version: "3.9"
services:
dashboard:
restart: always
build: .
image: &img img-two
volumes:
- .:/code
ports:
- "8010:8010"
depends_on:
- react-app
env_file:
- variables.env
react-app:
restart: always
build: .
image: *img
command: yarn start
env_file:
- variables.env
volumes:
- .:/app
- /app/node_modules
ports:
- "3050:3050"
Below is the database configuration I have set up in the front end django app that I want to connect to the DB but I keep getting connection refused errors when I try to run python manage.py runserver
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": os.environ.get("DB_NAME", "postgres"),
"USER": os.environ.get("DB_USERNAME", "postgres"),
"PASSWORD": os.environ.get("DB_PASSWORD", "postgres"),
"HOST": os.environ.get("DB_HOSTNAME", "db"),
"PORT": os.environ.get("DB_PORT", 5432),
}
}
Any ideas on how to fix this issue? (For reference, I've also tried changing HOST to localhost instead of db but still get the same connection refused errors)

Django + ElasticSearch + Docker - Connection Timeout no matter what hostname I use

I'm having issues connecting with my Elasticsearch container since day 1.
First I was using elasticsearch as the hostname, then I've tried the container name web_elasticsearch_1, and finally I'd set a Static IP address to the container and passed it in my configuration file.
PYPI packages:
django==3.2.9
elasticsearch==7.15.1
elasticsearch-dsl==7.4.0
docker-compose.yml
version: "3.3"
services:
web:
build:
context: .
dockerfile: local/Dockerfile
image: project32439/python
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
env_file:
- local/python.env
depends_on:
- elasticsearch
elasticsearch:
image: elasticsearch:7.10.1
environment:
- xpack.security.enabled=false
- discovery.type=single-node
networks:
default:
ipv4_address: 172.18.0.10
settings.py
# Elasticsearch
ELASTICSEARCH_HOST = "172.18.0.10"
ELASTICSEARCH_PORT = 9200
service.py
from django.conf import settings
from elasticsearch import Elasticsearch, RequestsHttpConnection
es = Elasticsearch(
hosts=[{"host": settings.ELASTICSEARCH_HOST, "port": settings.ELASTICSEARCH_PORT}],
use_ssl=False,
verify_certs=False,
connection_class=RequestsHttpConnection,
)
traceback
HTTPConnectionPool(host='172.18.0.10', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f1973ebd6d0>, 'Connection to 172.18.0.10 timed out. (connect timeout=5)'))
By default Docker Compose uses a bridge network to provision inter-container communication. You can read more about this network at the Debian Wiki.
What matters for you, is that by default Docker Compose creates a hostname that equals the service name in the docker-compose.yml file. So update your file:
version: "3.3"
services:
web:
build:
context: .
dockerfile: local/Dockerfile
image: project32439/python
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
env_file:
- local/python.env
depends_on:
- elasticsearch
elasticsearch:
image: elasticsearch:7.10.1
environment:
- xpack.security.enabled=false
- discovery.type=single-node
And now you can connect with elasticsearch:9200 instead of 172.18.0.10 from your web container. For more info see this article.

Can't connect to MySQL server on db. nodename nor server provided or not known using Docker

I'm trying to connect my application to my mysql database which I've got up and running in a docker-compose file. I'm using flask and trying to connect using DBUtils
I keep getting the error message described in my title:
(pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'db' ([Errno 8] nodename nor servname provided, or not known)"))
I've tried using the IPAddresses of my docker instances as well as several other solutions in similar problems discussed here on StackOverflow:
Docker-Compose can't connect to MySQL
Connecting to MySQL from Flask Application using docker-compose.
however, the offered solutions don't seem to be working for me.
my docker-compose file looks as follows:
version: '3.3'
services:
db:
image: mysql:8.0
container_name: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: 'myPassword'
MYSQL_DATABASE: 'databaseName'
volumes:
- .:/dockerFiles
ports:
- "3306:3306"
expose:
- "3306"
phpmyadmin:
image: phpmyadmin/phpmyadmin
container_name: phpmyadmin
restart: always
ports:
- "8080:80"
volumes:
- /sessions
and my connect string looks as follows:
def connect_db():
# Connects to the database and takes care of the connection
return PersistentDB(
creator=pymysql, host='db',
user='root', password='myPassword', database='databaseName', port=3306,
autocommit=True, charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
The error [Errno 8] nodename nor servname provided, or not known states that it can't find the database server. Thus, host='db' will only work if your Python code is inside a docker in the same network of the database service db. Try adding a service for the Python code in your docker-compose:
version: '3.3'
services:
db:
image: mysql:8.0
container_name: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: 'myPassword'
MYSQL_DATABASE: 'databaseName'
volumes:
- .:/dockerFiles
ports:
- "3306:3306"
expose:
- "3306"
web:
build: . # where is your dockerfile
command: sh -c 'python app.py' # this should be the command to start your application
ports:
- "8082:8082"
volumes:
- .:/code # it depends on the WORKDIR of your dockerfile
links:
- db

django-redis connection error inside docker

django views.py
import redis
import jwt
from access import utils
import os
redis_url = os.environ['REDIS_URI']
R = redis.StrictRedis(redis_url)
def set(request):
R.set('foo', 'bar')
return JsonResponse({"code":200,"msg":"success"})
docker-compose
version: "3"
services:
rango:
container_name: rango
build: ./
command: python backend/manage.py runserver 0.0.0.0:8000
# command: npm start --prefix frontend/rango-frontend/
working_dir: /usr/src/rango
environment:
REDIS_URI: redis://redis_db:6379
ports:
- "8000:8000"
tty: true
links:
- elasticsearch
- node
- redis
#elastic search
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.0
ports:
- "9200:9200"
#node
node:
image: node:10.13.0
#redis
redis:
image: redis
environment:
- ALLOW_EMPTY_PASSWORD=yes
ports:
- "6379:6379"
here i am connecting redis from django inside docker.
it is giving me exceptions connexctions refused.
Please have a look into my code and shared screenshot below
By default, docker compose makes containers discoverable with a hostname identical to the container name. Your redis container is thus discoverable via the hostname redis. However, your Django container is using the hostname redis_db.
Update your docker-compose.yml and change the REDIS_URI to reference the correct hostname:
REDIS_URI: redis://redis:6379