I want to profile my project to figure out what indices to create in db. I've installed django-debug-toolbar. The problem is that it is extremely slowing down page loading.
Moreover, couple minutes ago it was showing about 80 000 queries, now it takes about 48 seconds and it says that there are 2300 queries (which is according to Django Querycount correct). But when I turn django-debug-toolbar off, it works correctly (about 3 or 4 seconds).
Do you know what could cause problem? I've found that this was bug, but it was a couple years ago.
Related
OVERVIEW:
Hi, I have massive lag on my development server as long as I am offline. I'm using windows 10 pro. I have a website on my local environment at 127.0.0.1:8000/. I have a fairly high end laptop that can play high graphics games on it with no problems, and no lag, such as fortnite. But my local development server is super slow, and does not stop being slow unless I get online (even though I'm using the development server). This problem started back in may of this year, and back then I asked some other programmers who had no clue how to fix it. It had never been that way before, and I don't want to be online all the time...
EXAMPLE OF PROBLEM:
For a developed project while I'm offline: python manage.py runserver, paste 127.0.0.1:8000/ into the browser, wait ~10 seconds, then the background color shows up (no text yet), ~10 secs later some pics show up with some textless elements that have background colors, ~8 seconds after that the text shows up in portions of those elements, and ~10 secs after that the webfont and the rest of the page finally shows up.
For a fresh, brand spankin' new project made 5 seconds ago with nothing added to it: The same thing happens on the django success rocket page that pops up.
And everytime I do this, to even get the images/etc to show up, I have to click in the terminal and hit any button, like spacebar. Then every once in awhile the terminal decides to be a bro and shows a new line with a new GET or POST request (depending on what app I clicked on the page). Example:
[04/May/2018 19:21:09] "GET /blog/ HTTP/1.1" 200 59899
[04/May/2018 19:21:33] "POST /vote/ HTTP/1.1" 200 7648
[04/May/2018 19:21:33] "GET /vote/ HTTP/1.1" 200 7648
[04/May/2018 19:21:59] "GET /blog/ HTTP/1.1" 200 59900
If I'm online so the website runs fast/perfect: I noticed that if I highlight something in my terminal, the requests won't go anywhere, so I have to click in the terminal and hit any button to unhighlight it, and then it will run just fine. But when offline and nothing is highlighted, it's super slow.
THINGS I HAVE TRIED:
Full system scan with bitdefender (antivirus)... No problems found, and no improvements.
Defragged hard drive.... No improvements.
Went back to the previous lightning fast versions of my developed projects before ANY changes were made.... No improvements. Still super slow.
Ran the developed projects, and the brand spankin' new one on a different hard drive... No improvements.
Cleaned the registry with regseeker... MASSIVE IMPROVEMENTS so it will be lightning fast again, but it's very short lived (about 3-4 minutes).
Checked my CPU and RAM usage while requesting pages and clicking my apps... Nothing ever spikes anywhere even close to 100%. About 50% is max at all times with all my programs running. So I always have plenty available.
Installed Django-debug-toolbar. And my pages load fast according to it. Bigger pages take about 246ms, and smaller pages about 40ms. So the problem doesn't show up here, it only shows up in my terminal HTTP requests and on the browser window by taking forever to load.
Using python manage.py clearsessions on the database... No improvements.
A full clean install of windows 10 pro. Not the 1/2 assed install that lets you keep all your programs, but a full wipe, diaper change, and reinstall of everything. I actually did this 5 times (long story).
QUESTION:
Does anyone have any idea what's going on??? If you have no idea, do you know anywhere I can go to figure this out??? I would love to finally get rid of this bug. Back when this problem started I spent about 10-12 hrs a day for 2 weeks doing all the above, until I gave up (most of my problems were from #9). Having had some success asking quesitons here, I figured I'd ask.
Truly, the terminal gods hath smiled upon me this day!
I disabled my firewall, restarted my computer, and it works lightning fast again while offline.
If I didn't restart my computer after disabling that, it doesn't work. No idea why that is, but this bug was annoying, and I got sick of having to be online everytime I wanted to make an app.
Thanks to whoever suggested it might be something with my firewall, but later deleted their suggestion/answer.
UPDATE 10/9/2018:
I remember this problem began sometime around when I had an antivirus update. A couple days ago, I restarted my computer cause the antivirus had another update, and all of a sudden I don't get the lag anymore, even if keep the firewall on, but am still offline.
On my development machine (Dell XPS 8700), I can build my solution in 1 minute. Using VSO to build it takes much longer. Also, my build times over the past 10 days are increasing dramatically, even though I've only made minor changes to the source.
Here are the last 4 build times for the line in the log that says "Run MSBuild":
7:11
10:56
14:32
22:33
Does anybody know why this is happening?
With only 60 minutes of included build time per month, this is an issue for me.
Thanks.
We were having some issues with the build service a couple of weeks ago - when you posted the question. Is this problem still occurring?
I'm aware that a web application should render it's pages as fast as possible, with few database requests only in milliseconds. What are the guidelines about this response time (like Microsoft guidelines for UI or something like that)?
What is the absolute maximum time a webpage should respond under?
Are there any "limits" or general guidelines for this?
When should I put jobs into task queues (like Python celery for example)?
My concrete problem is that I have to parse a bunch of text files, which users submits. The average time these can be parsed are 2-3 seconds (response times are 3-4sec with database inserts) but if the file is very big, it takes 8sec to parse (10sec to respond).
Is it okay to let the user without feedback for this time? If not, what is the best way for handling these kinds of situations?
Is it even okay to put these in the request-response cycle ?
Is there any difference if I provide a REST API vs a website form ? Are these "allowed" slower to respond?
I think this is really hard to answer. Different guidelines exist.
When I was at university during interface / interaction design courses I learned that no user should be left with response times over 50 ms.
If that is exceeded, something like a loading icon etc should be displayed.
Also users are educated enough to expect right loading times form websites... So the user will accept 2 seconds loading time for a ticket booking page but not accept more than 300 ms from a search engine.
The limits I hear about during this days are 0.1 sec, 1 sec and 10 sec.
0.1 feels instantly to the user on websites
1 sec is slow but no interruption
10 sec is the maximum for the user to endure before loosing attention (for example light a smoke, check facebook feed in the meantime, etc...)
There is a nice article along with a lot of useful comments which I lately read which I would like to point you to:
http://www.nngroup.com/articles/response-times-3-important-limits/
I think it answers your questions well.
Please understand that this is all purely subjective but I think this is a very subjective topic...
Launching my second-ever Django site.
I've had problems in the past with Django's ORM (basically, the SQL it was generating just wasn't what I wanted and even using things like select_related() I couldn't wrangle it into what it should've been) -- I ended up just writing all my DB queries by hand in my views and using this function, taken from the Django docs, to turn the cursor's responses into usable dictionaries:
def dictfetchall(cursor, returnMultiDictAnyway=False):
"Returns all rows from a cursor as a dict"
desc = cursor.description
rows = [
dict(zip([col[0] for col in desc], row))
for row in cursor.fetchall()
]
if len(rows) == 1 and not returnMultiDictAnyway:
return rows[0]
return rows
I'm almost ready to launch my site but I'm finding pretty huge performance problems on the two different webservers I've tried hosting the app with.
Locally, it doesn't run blazingly fast, but I generally put this down to my machine in general being a little slow. I don't have the numbers to hand (will add later on) but the SQL times aren't crazily high and I've made the effort to optimise MySQL (adding missing indexes etc).
Here's the app, running on two different webhosts (using bit.ly to avoid Google spidering these URLs, sorry!):
http://bit.ly/10iEWYt (hosted on Dreamhost, using Passenger WSGI)
http://bit.ly/UZ9adS (hosted on WebFaction, also using WSGI)
At the moment I have Debug=False on both of those hosts (so there shouldn't be a loading penalty) and a file-based cache of 15 minutes for each one. On the Dreamhost one I have an experimental cronjob hitting the homepage every 15 minutes in an effort to see if this keeps the Python server alive -- this doesn't seem to have done much.
If you try those links you should see how long it takes for the server to respond as you click around, even including the cache (try going from the homepage to another page then back home).
I've tried this profiling middleware but not really sure how to interpret results (can add them to this post later on when I'm home) -- in any case, the functions/lines it pointed to were all inside Django's own code so I struggled to relate that to my own views etc.
Is it likely that the dictfetchall() method above could be an issue here? I use that to work with the results of every DB query on the site (~5-10 per page, most on the homepage). I do have a few included templates but nothing too crazy. I have a context processor for common things like showing album reviews, which I use all over the place. I'm stumped about what else could be causing this slowness.
Thanks, hope this is enough info to be helpful.
EDIT: okay, here's a profiling trace of the site homepage: http://pastebin.com/raw.php?i=c7kHNXAZ -- struggling to interpret it, to be honest.
Also, I looked at the Debug Toolbar stats: 8 SQL queries in 246ms (looking currently at further optimising these), but total time for render of 3235ms (locally). This is what's confusing me.
SQLite is a great little database, but I am having an issue with it on Windows. It can take up to 50 seconds to perform a query on a 100MB database the first time the application is launched. Subsequent loads take 10% of that time.
After some discussions on the SQLite mailing list, I am told
"The bug is in Windows. It aggressively pre-caches big database files
-- reads in big chunks of the files -- to make it look as if programs
like Outlook are better than they really are. Unfortunately although
this speeds up some programs it makes others act jerky because they
have no control over how much is read when they ask for just a few
bytes of file."
This problem is compounded because there is no way to get progress information while all this is happening from SQLite, so my users think something is broken. (I could display a dummy progress report, but that is really cheesy for a sharp tool.)
I believe there is a way to turn the pre-caching off globally, but is there some way around this programmatically?
I don't know how to fix the caching problem, but 50 seconds sounds extreme. If the query itself takes 10% of that, that means 45 seconds to load a 100mb file. Even if Windows does read in the entire file in one go, that shouldn't take more than a couple of seconds given normal harddrive speeds.
Is the file very fragmented or something?
It sounds to me like there's more than just precaching at play here.
I'm too having the same problem with my first query. The problem returns after not querying the database for a long time. It seems to be a memory caching problem. My software runs 24/7 and every once in a while the user performs the SELECT query. I am also performing the query on a database of the same size.