Any calls to Heroku redis causes flask app to timeout - flask

I've been developing a web app for the usage of computers and I'm attempting to store some information about active computers on redis. I'm hosting my Flask web app on Heroku and using Heroku redis.
When I'm testing on my local machine using the Heroku Redis URL, there are no problems and everything seems to be fine.
The problem I'm having is when I push and host on heroku. As soon as there is any interaction with the redis database the workers give a H12 web request timeout and none of my pages will load.
at=error code=H12 desc="Request timeout" method=GET path="/" host=my_app ... dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https
At first I thought it was a problem with connecting to the redis database, but on launch I can see in the logs that it connects without a problem. I also thought it might be because I'm initializing some lists and hashes, but after removing those and simply doing a "set" on launch caused the same issue. The weird thing is that if I remove any gets, sets, etc. but I leave the connection to the redis database, everything runs fine. However, as soon as I try a simple get or set I get the aforementioned error. I've provided my code for initializing the redis instance below and note that I am using a blueprint model as shown out below. (I've also tried moving the initialization of the redis instance to the init file but it still gives the same results).
- app
- application
- __init__
- models
models.py
import os
from urllib.parse import urlparse
import redis
url = urlparse(os.environ.get("REDIS_URL"))
my_redis = redis.Redis(host=url.hostname, port=url.port, username=url.username,
password=url.password, ssl=True, ssl_cert_reqs=None)

The issue is that you'll need to change REDIS_URL to REDIS_TLS_URL or set ssl=False. It's better to set REDIS_TLS_URL for security reason though.
If you'll need to work both on the Heroku env and your local env, set a persistent Heroku env var such as
heroku config:set HEROKU=1
Then on you app.py add
if 'HEROKU' in os.environ:
url = urlparse(os.environ.get("REDIS_TLS_URL"))
else:
url = urlparse(os.environ.get("REDIS_URL"))

Related

Docker Image works with docker-compose up but not on Amazon ECS or Heroku

I am trying to host a Django project with a Postgres database in a Docker container. The project is a practice e-commerce site with a database for product info. I was able to get it working with docker-compose up and accessed the site running in the container at localhost:8000 but when I tried hosting it on AWS it didn't work. I uploaded the image to ECR and started a cluster. When I tried running a task with the image, it showed PENDING but as soon as I tried to refresh, the task was gone. I tried setting up cloudwatch logs but they were empty since the task was stopping immediately after starting. After that I tried hosting on Heroku. I was able to deploy the image but when I tried to open the app it showed an error (shown below).
It feels like the image is just failing immediately whenever I try hosting it anywhere, but it works fine when I use docker-compose up. I think I'm making a very basic mistake (I'm a total beginner at all this) but not sure what it is. Thanks for taking the time to help.
I'll also add my Dockerfile and docker-compose.yml
Error Message from Heroku
2022-11-25T05:13:31.719689+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=hk-comic-app.herokuapp.com request_id=ea683b1d-e869-4ea9-98aa-2b9ed08f7219 fwd="98.116.68.242" dyno= connect= service= status=503 bytes= protocol=https
2022-11-25T05:22:36.083750+00:00 app[api]: Scaled to app#1:Free by user
2022-11-25T05:22:39.300239+00:00 heroku[app.1]: Starting process with command `python3`
2022-11-25T05:22:39.895200+00:00 heroku[app.1]: State changed from starting to up
2022-11-25T05:22:40.178736+00:00 heroku[app.1]: Process exited with status 0
2022-11-25T05:22:40.228638+00:00 heroku[app.1]: State changed from up to crashed
2022-11-25T05:22:40.232742+00:00 heroku[app.1]: State changed from crashed to starting
2022-11-25T05:22:43.937389+00:00 heroku[app.1]: Starting process with command `python3`
2022-11-25T05:22:44.610097+00:00 heroku[app.1]: State changed from starting to up
2022-11-25T05:22:45.130636+00:00 heroku[app.1]: Process exited with status 0
2022-11-25T05:22:45.180808+00:00 heroku[app.1]: State changed from up to crashed
2022-11-25T05:23:09.462805+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path="/" host=hk-comic-app.herokuapp.com request_id=f4cc3e04-0257-4336-94b3-7e48094cabd4 fwd="98.116.68.242" dyno= connect= service= status=503 bytes= protocol=https
Dockerfile
FROM python:3.9-slim-buster
ENV PYTHONUNBUFFERED=1
WORKDIR /django
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
docker-compose.yml
version: "3"
services:
app:
build: .
volumes:
- .:/django
- ./wait-for-it.sh:/wait-for-it.sh
ports:
- 8000:8000
image: app:django
container_name: django_container
command: /wait-for-it.sh db:5432 -- python3 manage.py runserver 0.0.0.0:8000
depends_on:
- db
db:
image: postgres
volumes:
- ./data/db:/var/lib/postgresql/data
environment:
- POSTGRES_DB=comic_db
- POSTGRES_USER=comic_user
- POSTGRES_PASSWORD=password
container_name: postgres_db
Heroku doesn't use docker-compose.yml. You'll need to make a few changes:
Update your Dockerfile to include a command that should be used to run your application, e.g.
CMD gunicorn project_name.wsgi
This shouldn't impact your development environment since your docker-compose.yml overrides the command. You'll need to make sure Gunicorn (or whatever WSGI server you choose to use) is declared as a dependency.
Update your Django application to get its PostgreSQL connection string from the DATABASE_URL environment variable that Heroku provides. A common way to do that is by adding a dependency on dj-database-url and then changing your DATABASES setting accordingly:
DATABASES["default"] = dj_database_url.config()
I suggest you read the documentation for that library as there's more than one way to use it.
For example, you can optionally set a default connection for development via a default argument here. Or, if you prefer, you could set your own DATABASE_URL environment variable in your docker-compose.yml.
Provision a PostgreSQL database for your application. Make sure to do the first step to check whether you already have a database.
Then redeploy.

testdriven.io django raise 500 on heroku when tried to access postgres

I am following tutorial in https://testdriven.io/courses/tdd-django/deployment/ .
Everything worked perfectly when I request get without accessing postgres,
http://ancient-waters-04623.herokuapp.com/ping/ will return status 200
I done the migration and seed data
$ heroku run python manage.py migrate
$ heroku run python manage.py loaddata movies.json
when i run http://ancient-waters-04623.herokuapp.com/api/movies/ in my browser, it will give me error 500
the logs:
$ heroku logs --app=ancient-waters-04623 --tail
2020-09-07T10:13:51.045281+00:00 heroku[router]: at=info method=GET path="/ping/" host=ancient-waters-04623.herokuapp.com request_id=1895c42e-27d8-4d67-b5e8-98d0e5c3a3bd fwd="210.186.32.252" dyno=web.1 connect=0ms service=4ms status=200 bytes=225 protocol=http
I tried connect the database via dbeaver, and it connected and has correct data loaded.
I'm newbie in heroku either django, any help will be appreciated.
Thanks in advance
I was also getting a 500 back when accessing
https://APP_NAME.herokuapp.com/api/movies/
I turned on debug mode in Dockerfile.prod, and was able to view the DRF Browsable API with the expected data.
This led me to verify...
if not DEBUG:
REST_FRAMEWORK = {
"DEFAULT_RENDERER_CLASSES": (
"rest_framework.renderers.JSONRenderer",
)
}
...in settings.py, where I determined I was missing the final ',' (see also: https://testdriven.io/courses/tdd-django/deployment/#H-1-browsable-api)
After fixing that and disabling debug mode again in Dockerfile.prod, I rebuilt the image, pushed it to the heroku container registry and released it. Now, I get the expected JSON response from
https://APP_NAME.herokuapp.com/api/movies/

Docker + Django + PostgreSQL + Heroku = Failed to bind to $PORT within 60 seconds

So I am building a Dockerized Django project and I want to deploy it to Heroku, but I am having a lot of issues. My issues are exactly the same as this post:
Docker + Django + Postgres Add-on + Heroku
Except I cannot use CMD python3 manage.py runserver 0.0.0.0:$PORT since I receive an invalid port pair error.
I'm just running
heroku container:push web
heroku container:release web
heroku open
After going to the site it stays loading until it says an error occurred. My log shows the following:
System check identified no issues (0 silenced).
2019-05-03T11:38:47.708761+00:00 app[web.1]: May 03, 2019 - 11:38:47
2019-05-03T11:38:47.709011+00:00 app[web.1]: Django version 2.2.1, using settings 'loan_app.settings.heroku'
2019-05-03T11:38:47.709012+00:00 app[web.1]: Starting development server at http://0.0.0.0:8000/
2019-05-03T11:38:47.709014+00:00 app[web.1]: Quit the server with CONTROL-C.
2019-05-03T11:38:55.505334+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=GET path="/" host=albmej-loan-application.herokuapp.com request_id=9037f839-8421-46f2-943a-599ec3cc6cb6 fwd="129.161.215.240" dyno= connect= service= status=503 bytes= protocol=https
2019-05-03T11:39:45.091840+00:00 heroku[web.1]: State changed from starting to crashed
2019-05-03T11:39:45.012262+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
2019-05-03T11:39:45.012440+00:00 heroku[web.1]: Stopping process with SIGKILL
2019-05-03T11:39:45.082756+00:00 heroku[web.1]: Process exited with status 137
The app works locally through a virtual environment and using Docker but just not on Heroku. Not sure what else to try. You can find my code at: https://github.com/AlbMej/Online-Loan-Application
Maybe I have some glaring problems in my Dockerfile or docker-compose.yml
The Answer is not correct.
If you use container with Dockerfile, you do not need any Profile.
Just use the $PORT variable to let heroku to determine which port to use.
https://help.heroku.com/PPBPA231/how-do-i-use-the-port-environment-variable-in-container-based-apps
Quick solution is to change your Procfile to this:
web: python subfolder/manage.py runserver 0.0.0.0:$PORT
That would work, but keep in mind that you are using the development server on production, which is a really bad idea! But if you are just toying around, that's ok.
However, if you're using this as a production app with real data, you should use a real production server. Then your Procfile would look like this:
web: gunicorn yourapp.wsgi --log-file -

Django restframework, Django channels, Ionic 2 - websocket handshake error

I am currently working on a project using the technologies mentioned in the thread title.
I got it all running from the browser (the application is hosted on heroku), but when I try to connect to the websockets from my Ionic 2 application, I always get an error establishing handshake.
2016-09-17T15:02:03.200133+00:00 app[web.1]: 2016-09-17 15:02:03,199 DEBUG Connection http.response!uvRVDyvolYEG did not get successful WS handshake.
2016-09-17T15:02:03.200498+00:00 app[web.1]: 2016-09-17 15:02:03,200 DEBUG WebSocket closed before handshake established
2016-09-17T15:02:03.169206+00:00 heroku[router]: at=info method=GET path="/1/" host=musicmashup-jukebox.herokuapp.com request_id=c46960d7-bb8f-45bf-b8be-5a934c771d96 fwd="212.243.230.222" dyno=web.1 connect=0ms service=7ms status=400 bytes=74
Now one idea was, that it could be a CORS problem. So I installed django-cors-middleware in hope this could solve the problem - well it did not.
But I think the app does not add any headers to the Daphne server at all.
At the moment I have no idea anymore, if the problem is on the client or on the server side.
Has anyone experienced similar problems?
EDIT:
Found out that websockets and CORS do not have anything to do with each other Why is there no same-origin policy for WebSockets? Why can I connect to ws://localhost?
So my guess is, that the server may reject the origin header sent by the client. I will see if I can get my hands on the headers being sent
This issue was fixed in Daphne v.1.0.3
https://github.com/django/daphne/commit/07dd777ef11f5091931fbb22bdacb9e4aefea7da
You need to also update channels and asgi-redis if it's used.
Alright, the problem was somehow related to the origin header. Ionic seems to be sending a origin header containing "file://..", which was getting rejected / blocked by the websocket server.
Unfortunately I did not find a way to configure the webserver on heroku to either ignore this or set another origin header on incoming packets.
My Procfile on heroku:
web: daphne app.asgi:channel_layer --port $PORT--bind 0.0.0.0 -v2
worker: python manage.py runworker -v2
What I did then, was moving the whole application to a self hosted Ubuntu server and putting an nginx in front of Daphne, where I created a rule to override the origin header of incoming packets.
That's how it can be done.. I hope this can help some people.
Thank you platzhersch,
it worked for me with the following nginx rule:
proxy_set_header Origin http://$host;

ALLOWED_HOSTS is set, but Django is still giving a 500 error when DEBUG=False

In Django 1.5, ALLOWED_HOSTS became a required setting for when DEBUG=False. However, in my app, I have ALLOWED_HOSTS = ['.mycompany.com'], and I'm hitting the app via a "http://mysubdomain.mycompany.com" URL, but Django is still giving a 500 error/response.
With DEBUG=True, everything works fine, so it appears that Django is just ignoring that setting. Is there something else that could cause Django to bork out when DEBUG=False?
Is there a way to make Django spit out some details, in the log, about what its problem is? By default, it just spits out something like the following in the log:
at=info method=GET path="/admin/login/?next=/admin/" host=mysubdomain.mycompany.com request_id=69e99b68-5b57-4f3f-a10d-898deb507ae4 fwd="123.456.789.101" dyno=web.1 connect=1ms service=161ms status=500 bytes=253
which is no help at all...