According to the django-debug-toolbar my CPU Time is around 31000ms (on average). This is true for my own pages as well as for the admin. Here is the breakdown when loading http://127.0.0.1:8000/admin/:
Resource usage
User CPU time 500.219 msec
System CPU time 57.526 msec
Total CPU time 557.745 msec
Elapsed time 30236.380 msec
Context switches 11 voluntary, 1345 involuntary
Browser timing (Timing attribute / Milliseconds since navigation start (+length))
domainLookup 2 (+0)
connect 2 (+0)
request 7 (+30259)
response 30263 (+3)
domLoading 30279 (+1737)
domInteractive 31154
domContentLoadedEvent 31155 (+127)
loadEvent 32016 (+10)
As far as I understand the "request" step [7 (+30259)] is the biggest bottleneck here. But what does that tell me? The request panel just shows some variables, and no GET nor POST data.
The same code works fine hosted on pythonanywhere, locally I am running a MacBook Air (i5, 1.3 Ghz, 8GB RAM). The performance hasn't been this poor all the time. IIRC it happened "over night". One day I started the dev server and it was slow. Didn't change anything in the code or DB.
Is it right to assume that it could be an issue with my local machine?
EDIT:
I tried running ./manage.py runserver --noreload but the performance didn't improve. Also, starting the dev-server (using ./manage.py runserver) also takes around 40s and accessing the DB using postico takes around 1 minute. Starting the dev-sever while commenting out the database from django's settings makes load times normal.
Solved it.
This post pointed me in the right direction. Essentially, I ended up "reseting" my hostfile according to this post. My hostfile now looks like this:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
fe80::1%lo0 localhost
Does not, however, explain what caused the sudden "overnight" issue in the first place. My guess: renamed hostname. A couple of days ago my hostname ended up sounding something along the lines of android- (similar to this) which was apparently caused by me using Android's file share tool. Ended up renaming my hostname to my username (see instructions below).
Perform the following tasks to change the workstation hostname using
the scutil command. Open a terminal. Type the following command to
change the primary hostname of your Mac: This is your fully qualified
hostname, for example myMac.domain.com sudo scutil --set HostName Type
the following command to change the Bonjour hostname of your Mac: This
is the name usable on the local network, for example myMac.local. sudo
scutil --set LocalHostName Optional: If you also want to change the
computer name, type the following command: This is the user-friendly
computer name you see in Finder, for example myMac. sudo scutil --set
ComputerName Flush the DNS cache by typing: dscacheutil -flushcache
Restart your Mac.
from here
Didn't test the "renaming hostname" theory, though.
Related
GCP VM doesn't update the system datetime after resuming it from suspension.
It keeps the system date/time same as what it was while suspending. Due to this, my scripts to fetch gcloud resources is failing as with auth token expiry error.
As per the Google Documentation https://cloud.google.com/compute/docs/instances/managing-instances#linux_1,
NTP is already configured but for my VMs I get the "command not found" error for ntpq -p.
$ sudo timedatectl status
Local time: Wed 2020-08-05 15:31:34 EDT
Universal time: Wed 2020-08-05 19:31:34 UTC
RTC time: Wed 2020-08-05 19:31:34
Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no
gcloud auth activate-service-account in my script is failing with below error
(gcloud.compute.instances.describe) There was a problem refreshing your current auth tokens: invalid_grant: Invalid JWT: Token must be a short-lived token (60 minutes) and in a reasonable timeframe. Check your iat and exp values in the JWT claim.
OS - Windows/Linux
After resuming, the hardware clock of the VM instance is set correctly as it gets time from the hypervisor. You can check it with sudo hwclock.
The problem is with the time service of the operating system.
For Windows, it could take few minutes to sync system time with the time source. If you can't wait for the timesync cycle to complete, you can logon to Windows and force time synchronization manually:
net stop W32Time
net start W32Time
w32tm /resync /force
In Linux, NTP cannot handle a time offset of more that 1000 seconds (see http://doc.ntp.org/4.1.0/ntpd.htm. Therefore you have to force time synchronization manually. There are various ways to do that (some of them are deprecated, but still may work):
netdate timeserver1
ntpdate -u -s timeserver1
hwclock --hctosys
service ntp restart
systemctl restart ntp.service
If you run into this issue while using Google Cloud Platform, they replace netd and systemd-timesyncd with chronyd
I had to use systemctl start chrony to get my time in working order. Tried hwclock --hctosys, but it was ignoring time zones and thus setting the wrong time.
This happened because I was suspending every minute by accident. A permanent fix would be to modify the systemd definition and ask it keep retrying to start it.
The reason it stopped was this Can't synchronise: no selectable sources
While registering a host to the cluster of Ambari-server, I am getting the following error.
"Host checks were skipped on 1 hosts that failed to register."
I'm trying to install HDP 2.5 version on the instance of AWS.
I have tried to follow the documentation of Hortonworks.
https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-installation/content/set_the_hostname.html
I have added public ip address and public hostname to /etc/hosts file and change the name of host in /etc/hostname file on the server and on the host. Rebooted both, hostname got changed. Then I have stop iptables by
sudo service iptables stop
After doing everything, the host registration is still failing. Kindly help. I am stuck.
Background
From my experience with Ambari (Hortonworks) you have to explicitly setup your Hadoop nodes in each other's /etc/hosts file with the actual name/IPs that the Hadoop services will bind to. NOTE: hostnames should also be FQDN - fully qualified domain names.
For example if you're setting up the hosts as:
node01.mydom.com (10.0.0.2)
node02.mydom.com (10.0.0.3)
node03.mydom.com (10.0.0.4)
These entries should be in all 3 server's /etc/hosts and these should be the names used when referencing them within Ambari's installation/setup wizards.
If you do not pay special attention to this detail, Ambari's server will fail to find/manage any of the other node's that you're telling it to manage.
hostname of ambari-agents
The other item to look at is that the ambari-agent's and what hostnames they think they're going as.
$ ps -eaf|grep ambari_agent
root 3282 1 0 Jul30 ? 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start --expected-hostname=node01.mydom.com
root 3290 3282 1 Jul30 ? 08:24:29 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start --expected-hostname=node01.mydom.com
Debugging further
In the screen where you're attempting to register the other nodes as agents, there's a full log of what's happening and you can typically get the commands from this area and attempt to run them directly. I've done this on a number of occasions. The commands will often be python ... commands which you can then copy/paste from the logs and run on the Ambari server where you're attempting to run the install.
I have a problem with uwsgi everytime I restart the server when I have a code updates.
When I restart the uwsgi using "sudo restart accounting", there's a small gap between stop and start instance that results to downtime and stops all the current request.
When I try "sudo reload accounting", it works but my memory goes up (double). When I run the command "ps aux | grep accounting", it shows that I have 10 running processes (accounting.ini) instead of 5 and it freezes up my server when the memory hits the limit.
accounting.ini
I am running
Ubuntu 14.04
Django 1.9
nginx 1.4.6
uwsgi 2.0.12
This is how uwsgi does graceful reload. Keeps old processes until requests are served and creates new ones that will take over incoming requests.
Read Things that could go wrong
Do not forget, your workers/threads that are still running requests
could block the reload (for various reasons) for more seconds than
your proxy server could tolerate.
And this
Another important step of graceful reload is to avoid destroying
workers/threads that are still managing requests. Obviously requests
could be stuck, so you should have a timeout for running workers (in
uWSGI it is called the “worker’s mercy” and it has a default value of
60 seconds).
So i would recommend trying worker-reload-mercy
Default value is to wait 60 seconds, just lower it to something that your server can handle.
Tell me if it worked.
Uwsgi chain reload
This is another try to fix your issue. As you mentioned your uwsgi workers are restarting in a manner described below:
send SIGHUP signal to the master
Wait for running workers.
Close all of the file descriptors except the ones mapped to sockets.
Call exec() on itself.
One of the cons of this kind of reload might be stuck workers.
Additionaly you report that your server crashes when uwsgi maintains 10 proceses (5 old and 5 new ones).
I propose trying chain reload. DIrect quote from documentation explains this kind of reload best:
When triggered, it will restart one worker at time, and the following worker is not reloaded until the previous one is ready to accept new requests.
It means that you will not have 10 processes on your server but only 5.
Config that should work:
# your .ini file
lazy-apps = true
touch-chain-reload = /path/to/reloadFile
Some resources on chain reload and other kinds are in links below:
Chain reloading uwsgi docs
uWSGI graceful Python code deploy
I'm guessing there's a very simple solution to this, but I searched every forum and setup guide and can't figure it out:
I built a Django/CentOS-6.3 environment on my local server (using VirtualBox and Vagrant). When I startup my server in the vagrant terminal with 'python manage.py runserver [::]:8000' it starts up with no errors.
Validating models...
0 errors found
May 31, 2013 - 13:56:15
Django version 1.5.1, using settings 'mysitename.settings'
Development server is running at http://[::]:8000/
Quit the server with CONTROL-C.
However, when I try to navigate to 'http://127.0.0.1:8001' in my browser (I set up port forwarding from port 8000 to port 8001 in my Vagrantfile), the browser just hangs for 5 minutes until it times out, then it returns the message:
> The connection was reset
> The connection to the server was reset while the page was loading.
> ...
This is the exact same message I get from the browser even after I shut down my local server. My computer obviously recognizes this as a forwarded port, because any other port I try (such as 8000) instantly returns an error saying that it can't establish a connection to the server at 127.0.0.1:8000.
With regard to the server files, I have done many similar setups with Django/Ubuntu in the past and have never had any issues, but there must be something different about Django/CentOS that is causing this to happen (or maybe I made a mistake someone in one of my server files). I have followed guides for setting up Django & PostgreSQL on CentOS, too, but to no avail. I'll comment some of the files I have created/edited below.
If anyone has a solution, or even has advice on where to start looking for errors, I would very much appreciate it.
If your network is configured correctly and your django application with
python manage.py runserver 0.0.0.0:8000
and you still can't access your django app from the VM host there is almost certainly a firewall issue. The solution above is good if you are running iptables.
I deployed CentOS 7 on a virtualbox VM from a Windows 7 host. I didn't know that this distribution uses firewalld, not iptables to control access.
if
ps -ae | grep firewall
returns something like
602 ? 00:00:00 firewalld
your system is running firewalld, not iptables. They do not run together.
To correct you VM so you can access your django site from the host use the commands:
firewall-cmd --zone=public --add-port=8000/tcp --permanent
firewall-cmd --reload
Many thanks to pablo v for pointing this out in the post "Access django server on virtual Machine".
the host's "127.0.0.1" is not the same as the guest's "127.0.0.1". Per default the command
python manage.py runserver
listens only to the guest's localhost. You should be able to test it from within the vm (use "vagrant ssh" to login) and run
curl -I http://127.0.0.1:8000/
The host as a different IP. To access the development server from the host you have to start it without ip restriction:
python manage.py runserver 0.0.0.0:8000
Yes:
python manage.py runserver [::]:8000
should be the same. But that's IPv6 syntax AFAIK. Are you sure that the "manage.py runserver" command supports IPv6 by default? I've never used ipv6 addresses w/ django, but looking at the source (https://github.com/django/django/blob/master/django/core/management/commands/runserver.py) there seams to be a flag that the default to False ("--ipv6"). Perhaps that's the "real" problem?
Regards,
For a similar problem,
This command worked like a charm for me
python manage.py runserver [::]:8001
Check your iptables, and stop it. Ubuntu commonly does not open the iptables when it starts.
My setup is django 1.3 and the default mod_wsgi and apache packages for ubuntu 10.04. I tested one view of my app on my development VM (DEBUG and debugging toolbar off):
ab -n 200 -c 5 http://127.0.0.1/
and got 4 requests per second. This seemed slow so I simplified the queries, used indexes, etc. to the point where debugging toolbar tells me I have 4 queries taking 8ms. Running the same test, I only get 8 requests per second. The CPU seems to be at 100% the whole time. This seems quite slow for what is now a quite simple view, but it is just a low powered VM.
I decided to start up a large ec2 instance (4 cpu) to see what kind of performance I would get on that class of machine and was surprised to only get 13 requests per second. How can I change the configuration of apache/mod_wsgi to get more performance out of this class of machine?
I think I am using worker rather than prefork:
$ /usr/sbin/apache2 -l
Compiled in modules:
core.c
mod_log_config.c
mod_logio.c
worker.c
http_core.c
mod_so.c
My worker configuration is:
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
and my WSGI settings look like this:
WSGIScriptAlias / /home/blah/site/proj/wsgi.py
WSGIDaemonProcess blah user=blah group=blah processes=1 threads=10
WSGIProcessGroup blah
Thanks very much for your help!
NOTE: I tried the ab test from another instance and got the same result
Make sure keep-alive is off.
More processes and single-threaded I have seen better performance where CPU is the limiting factor; try processes=4 threads=1.
The best way to tweak mod_wsgi is to not use it :)
First: I don't think your problem is the web server: with mod_wsgi you can get hundreds requests/s. You can get better results with caching and with DB connection pooling. If you're using postgres, take a look at pgpool II: http://pgpool.projects.postgresql.org/ .
However, if you want to go deeper into wsgi web servers, read carefully this nice benchmark: http://nichol.as/benchmark-of-python-web-servers .
If you don't need asyncronous workers, gunicorn is a good choice. It's very easy to setup (you can run it with manage.py run_gunicorn) and it's pretty fast. If you want to be sure that mod_wsgi is not the cuprit, try it. If you want better performance go with gevent or uWSGI.
But the Web Server won't change your benchmark: you can go from 4 req/s to 4.01 req/s.