I am going nuts here, I am trying to get django logging to work, which led me to need to check the gunicorn logs which I found were not setup and when i try to set them up via the config file it throws a 502 and I don't know how to track it down...
according to this... https://www.digitalocean.com/community/tutorials/how-to-use-the-django-one-click-install-image
the config file on the DO deployment is at... /etc/gunicorn.d/gunicorn.py
now when I go to that file and add a line like this as per the gunicorn docs tell me to do in the config file, and then I restart gunicorn it throws a 502 at me...
"""gunicorn WSGI server configuration."""
from multiprocessing import cpu_count
from os import environ
def max_workers():
return cpu_count() * 2 + 1
max_requests = 1000
worker_class = 'gevent'
workers = max_workers()
#my added config
errorlog = '/var/log/gunicorn/error.log'
Interesting to note is that when I srestart gunicorn with
sudo service gunicorn restart
it goes smoothly the first time and then if I try to restart it again it gives me this...
stop: Unknown instance:
gunicorn start/running, process 30943
I have tried contacting digital ocean about this and so far they have not been helpful, any ideas of where to go next?
This was cause by the permissions of the log files for some reason. Through some more testing I have found that the log files need to be already in place, and their permissions need to be set to no less than 666.
I have no idea why they need to be this high...
Related
I was deploying a django app and it failed because for some reason the gunicorn.socket file was not created even though before adding nginx it worked perfectly fine so I searched the internet and found this answer where the guy says that the reason for this is the virtual environment but I'm sure there must be a way around it using venv right?
the log I get from nginx:
connect() to unix:/run/gunicorn.sock failed (11 1: Connection refused) while connecting to upstream,
error from gunicorn:
gunicorn.socket: Failed with result 'service-start-limit-hit'.
I'm 100% sure the problem is with gunicorn not with the setup of nginx becuase I did check for the gunicorn file and it did not exist.
I don't know django, but I am going to assume that this issue is similar to an issue I saw trying to get the MySQL service to start on one of my servers today (see here: https://stackoverflow.com/a/55141733/708323)
Basically, the "start-limit-hit" message is a red herring - service start tries to start which ever service name you provide multiple times, and if after the fifth failed attempt (for me, at least) it crashes out on the sixth attempt with "start-limit-hit". You'll need to investigate the actual syslog (possibly /var/log/syslog) to see what the real errors are that are preventing the service from starting on attempts 1-5.
Check whether you have proper permissions for the user/group mentioned inside the gunicorn.service file.
I had the same issue, my gunicorn file had:
[Service]
User= www-data
Group= www-data
I changed the permission of gunicorn service file, socket file and django directory and it worked fine.
chown www-data:www-data /etc/systemd/system/gunicorn.service
chown www-data:www-data /etc/systemd/system/gunicorn.socket
chown www-data:www-data -R myproject/
Hope, it helps.
I have configured my subdomain on route53 to point to my ec2 public IP, right after I did this I got a welcome to nginx page. It was good this far. Now I add a custom cofig file in nginx/sites-available(also did link it to sites-enabled and also reloaded nginx). I don't see anything new. I still see the same welcome page. I wonder why this happened.I also have proxy_pass to :3000 which doesn't seem to work.
Here are the few things I tried and the result I got.
changed index.html file as mentioned in the root(/usr/share/nginx/html) - Still welcome page
tried server_name with both default_server and my subdomain(http://management.teampapercloud.in) - Still welcome page
stopped nginx service with service nginx stop - Surprisingly it still shows welcome page.
tried sudo lsof -i :80 (when nginx is running it has 2 diff nginx processed running one with root as user and the other www-data), when nginx is not running sudo lsof -i :80 gives nothing.
at this point i wondered if the domain hitting the server at all or not. I tried to ping, got my ec2 ip. tried DNS checker got my ec2 ip.
Now I am so frustrated stopped my ec2 instance. Now the welcome page is gone. which means I believe my ec2 instance is doing something to show the page, But I don't know which service.
checked top and service --status-all | grep + I don't see any strange services running nor services like Apache. as apache is not installed at all.
As a final try, I even uninstalled nginx. - I still see the same welcome page.
Updated all the index.html files I could find - Still same welcome page.
At this point, I am left with no ideas, and don't understand what is happening. Help is much appreciated.
Thanks
You have written, you reloaded nginx, try again (sometimes we forget to check common things):
sudo service nginx -t # check if configuration syntax is ok
sudo service nginx reload #reload configuration
sudo service nginx restart # restart server
Another important thing, check realtime log in terminal
error log: tail -f /var/log/nginx/error.log
access log: tail -f /var/log/nginx/access.log
and visit the page from browser and check the log in terminal.
Note: Log files can be in different directory in your server.
check log files with this: ls /var/log/nginx/
May be restart the os, if you have access to restart.
ec2 can have any page cache settings.
Are you using something cloudflare name server that can cache a
page, or, Any proxy server or something, that can cache the page.
You can even try browser incognito window, force reload in browser
(ctrl + shift + R).
The changes are not being applied because you're not making any changes to root directory which is
/var/www/html$
as this is a default public folder for your web server. Also if you wish you can change this default path to the one you want.
As you can see I have "index.nginx-debian.html" which on changing takes effect immediately.
I know that I can use logging to log all the requests that my django server gets. However I would like to log the requests along with headers and body that my django server makes to other servers. How can I access this info?
What type of logs do you really want?
1 - in development mode, you can get the request as .txt using: python manage.py runserver > logs.txt - source: stackoverflow
2 - The Apache2 webserver provides two important logs: error.log or access.log. If you are using debian/ubuntu, you can get it in?
/var/log/apache2/error.log or access.log
I have a Django site that uses Gunicorn and Nginx. Occasionally, I'll have a problem that I need to debug. In the past, I would shut down Gunicorn and Nginx, go to my Django project directory and start the Django development server ("python ./manage.py runserver 0:8000"), and then restart Nginx. I could then insert set_trace() commands and do my debugging. When I fixed the problem I'd shut down Nginx and then restart Gunicorn and Nginx. I'm pretty sure this was working.
Recently, though, I've begun having problems. What happens now is that when I've stopped at a breakpoint, after a couple of minutes the web page that I've stopped on will change and display "404 Not Found" and if I take another step in the debugger, I'll see this error:
- Broken pipe from ('127.0.0.1', 43742)
This happens on my development, staging, and production servers which I'm accessing via their domain names, e.g. "web01.example.com" (not really example).
What is the correct way to debug my Django application on my remote servers?
Thanks.
I figured out the problem. First I observed that when I stopped at a breakpoint, the page always timed out after exactly one minute which suggested that the Nginx connection to the web server was timing out if the web server took more than 60 seconds to respond. I then found an Nginx proxy_read_timeout directive which defines this timeout. Then it was merely a matter of changing the length of the timeout in my Nginx config file:
# /etc/nginx/sites-enabled/example.conf
http {
server {
...
location #django {
...
# Set timeout to 1 hour
proxy_read_timeout 3600s;
...
}
...
}
}
Once you've made this change you need to reload Nginx, not restart it, in order to this change to take effect. Then you start Django as I indicated above and you can now debug your Django application without it timing out. Just be sure to remove the timeout setting when you're done debugging, reload Nginx again, and restart Gunicorn.
I'm just starting out with EC2, and I've pulled down a git repo that I started on my local machine and so I know that it works running the server from there, and it seems to works when I run my server from the EC2 instance I have running, but for some reason, when I visit the elastic IP address of that instance I get a page-not-found. Any idea on why that might be?
So, I've now started using nginx, and made a conf file following the instructions here: https://code.djangoproject.com/wiki/DjangoAndNginx that is as follows:
server {
listen 80;
server_name ec2-54-242-149-154.compute-1.amazonaws.com;
access_log /var/log/nginx/USBag.access.log;
error_log /var/log/nginx/USBag.error.log;
location /basicMap/ {
alias /home/www/ec2-54-242-149-154.compute-1.amazonaws.com/basicMap/;
expires 30d;
}
location / {
include fastcgi_params;
fastcgi_pass 127.0.0.1:8080;
}
}
basicMap is a place that I have already defined in my django app, and the linked ec2 ip is the one my server is running on. I am having a lot of difficulty finding documentation on how to proceed or how to determine if my conf file is correct or not. Using the standard python manage.py runserver doesn't work however. Advice on how to proceed?
There is a lot of info about setting up a production django server out there, and I'll give you my personal preferences below, but before all that let's backup and see if we can just get any response from the production server.
To start the development server on your EC2 instance run:
manage.py runserver 0.0.0.0:8000
That command will cause runserver to bind to all interfaces and serve files to the external world. You'll never want to do this outside of development, but it is a good way just to test if your django app is setup before complicating things. Now try hitting your EC2 instance and see if you get a response.
If that's still not working, make sure you allow incoming connections to the server's port (8000 in the command above, 80 once live). You could test that you have ports open using netcat (nc -l).
Once you are satisfied that you have your app setup, I'd recommend you use nginx as your front end webserver and gunicorn as your django webserver in production. You'll likely want to look into setting up a virtualenv, supervisord etc for your production setup (here is a tutorial: http://senko.net/en/django-nginx-gunicorn/), but all that depends on the specifics of your project.