Django REST Framework slow on Gunicorn - django

Question

I deployed a new application made on Django REST Framework running with Gunicorn.
The application is deployed on 4 different servers and they are listening on a port (8001) which is consumed by an HAProxy load balancer.
Unfortunately I am suffering many performance problems: many requests takes seconds to be served, even 30 or 60 seconds.
Sometimes even the basic entry endpoint (like https://my.api/api/v2 which basically returns the list of the available endpoints) requires 10/20 seconds to respond.
I don't think the problem is the load balancer because I have latencies connecting directly to any backend servers with my client, so bypassing the load balancer.
And I don't think the problem is the database because the call to https://my.api/api/v2 as guest (not logged with any user) shouldn't make any database query.
This is a performance test made with hey (https://github.com/rakyll/hey) on the basic endpoint without authorization:
me#staging2:~$ hey -n 10000 -c 100 https://my.api/api/v2/
Summary:
Total: 38.9165 secs
Slowest: 18.6772 secs
Fastest: 0.0041 secs
Average: 0.3099 secs
Requests/sec: 256.9604
Total data: 20870000 bytes
Size/request: 2087 bytes
Response time histogram:
0.004 [1] |
1.871 [9723] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
3.739 [0] |
5.606 [175] |■
7.473 [0] |
9.341 [35] |
11.208 [29] |
13.075 [0] |
14.943 [0] |
16.810 [0] |
18.677 [37] |
Latency distribution:
10% in 0.0054 secs
25% in 0.0122 secs
50% in 0.0322 secs
75% in 0.2378 secs
90% in 0.3417 secs
95% in 0.3885 secs
99% in 8.5935 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0021 secs, 0.0041 secs, 18.6772 secs
DNS-lookup: 0.0001 secs, 0.0000 secs, 0.0123 secs
req write: 0.0001 secs, 0.0000 secs, 0.0153 secs
resp wait: 0.3075 secs, 0.0039 secs, 18.6770 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0150 secs
Status code distribution:
[200] 10000 responses
This is my Gunicorn configuration:
bind = '0.0.0.0:8001'
backlog = 2048
workers = 1
worker_class = 'sync'
worker_connections = 1000
timeout = 120
keepalive = 5
spew = False
daemon = False
pidfile = None
umask = 0
user = None
group = None
tmp_upload_dir = None
errorlog = '-'
loglevel = 'debug'
accesslog = '-'
access_log_format = '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'
And Gunicorn is currently running with the following command:
/path/to/application/bin/python3 /path/to/application/bin/gunicorn --env DJANGO_SETTINGS_MODULE=settings.production -c /path/to/application/settings/gunicorn_conf.py --user=deployer --log-file=/path/to/application/django-application-gunicorn.log --chdir /path/to/application/django-application --daemon wsgi:application
What tests I can do to find out what is causing my performance problems?