Where is robots.txt coming from (Nginx + Gunicorn + Django)? - django

Question

I am running a server with Nginx + Gunicorn + Django.
I have configured Django to show a robots.txt file, and this works just fine when I test Django on its own, without Nginx+Gunicorn.
However, when I run the whole stack, /robots.txt suddenly gets replaced with a different file that I didn't write.
Why is this happening? I never wrote anything to tell Nginx or Gunicorn to do such a thing. Is there a default somewhere that I need to overwrite or deactivate?
The folder /etc/nginx/sites-enabled contains only my own sites file, and it doesn't mention /robots explicitly. It just has a location block targeting / which gets routed to Gunicorn+Django, and for every site except robots.txt this works just fine.
My Gunicorn config:
[Unit]
Description=gunicorn daemon
After=network.target
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=root
Group=www-data
WorkingDirectory=/root/lod/collab/collab
ExecStart=/root/lod/CollabVenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/root/lod/collab.sock collab.wsgi:application
[Install]
WantedBy=multi-user.target
My Nginx config:
server {
# if no Host match, close the connection to prevent host spoofing
listen 80 default_server;
listen [::]:80 default_server;
return 444;
}
server {
# redirect any http requests to https
listen 80;
listen [::]:80;
server_name example.com www.example.com localhost;
return 301 https://example.com$request_uri;
}
server {
# this is the main server application, serving example.com with https
listen 443;
listen [::]:443;
server_name example.com;
# SSL certificate
ssl on;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
# allow large request bodies, which may be necessary for some JSON commands
client_max_body_size 4G;
# ignore a missing favicon
location = /favicon.ico { access_log off; log_not_found off; }
# Django media
location /media {
alias /root/lod/CollabData/media_public; # your Django project's media files
}
# Django static
location /static {
autoindex on;
alias /root/lod/collab/collab/static_processed; # your Django project's static files
}
# send all non-media requests to the Django server
location / {
# necessary for SSL
# note:
# I have no idea what most of these do. I just followed tutorials on the internet.
# Feel free to improve this if you know what you are doing.
proxy_redirect off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host:443;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
# the connection to Gunicorn
proxy_pass http://unix:/root/lod/collab.sock;
}
}
EDIT:
It just started working correctly, even though I made no changes since the last time I opened the site. Maybe some background process or cache took a while to update? I'm leaving this question open in case someone can explain what caused this, so that other people won't have to spend as much time with this problem as I did.