I am using Django + memcached and have a (hopefully) simple question.
My database is updated once a day. My pages are set to time out after 24 hours.
Is there a way to generate all the pages of the site into the cache once each day, just after the database is updated, in advance of any users coming to them?
I'd like the first user of the day to see the fast-loading cached version, not the slow-loading non-cached version.
I guess I could do this by scraping the site, but is there a neater way?
I think this is going to depend on how you have your urls.py set up.
If your urls are all either
A.) Straight out of the urls.py
or
B.) Predictable based on your database
If so, you might be able to use django_extensions show_urls
by doing python manage.py show_urls it outputs a list of all the urls.
From there, you can just capture them in a list and loop over them while hitting each one with a requests.get(some_url)
The output will tell if a variable is needed. If so, just replace it with the correct variable(s) and your done.
Related
I'm reviving an old django 1.2 app. most of the steps have been taken.
I have views in my django app that will reference a simple dictionary of only 1300ish key-value pairs.
Basically the view will query the dictionary a few hunderd to a few thousand times for user supplied values.The dictionary data may change twice a year or so.
fwiw: django served by gunicorn, db=postgres, apache as proxy, no redis available yet on the server
I thought of a few options here:
a table in the database that will be queried and let caching do its
job (at the expense of a few hundred sql queries)
Simply define the dictionary in the settings file (ugly, and how many time is it read? Every time you do an 'from django.conf import settings'?
This was the situation how it was coded in the django 1.2 predecessor of this app many years ago
read a tab delimited file using Pandas in the django settings and make this available. the advantage is that I can do some pandas magic in the view. (How efficient is this, will the file be read many times for different users or just once during server startup?)
prepopulate a redis cache from a file as part of the startup process (complicates things on the server side and we want it to be simple, but its fast.
List items in a tab delimited file and read it in in the view (my least popular option since it seems to be rather slow)
What are your thoughts on this? Any other options?
Let me give a few - simple to more involved
Hold it in memory
Basic flat file
Sqlite file
Redis
DB
I wouldn't bring redis in for 1300 kv pairs that don't even get mutated all that much
I would put a file alongside the code that gets slurped in memory at startup or do a single sql query and grab the entire thing at startup and keep it in memory to use throughout the application
i'm a total noob in django and just wondering if it's possible for an admin doing a same thing at the same time ? the only thing i get after looking at the django documentation is that it is possible to have two admins, but is it possible for the admins to do a task in the same databases at the same time ?
thanks for any help
You didn't made it clear that what do you actually want but:
If by admin you mean a superuser then yes you can have as many admins as you want.
Admins can change anything in database at the same time, but if you mean changing a specific row of a specific table at the same time, its not possible because of these reasons:
Its kinda impossible to save something at the same time. when both admins tries to save anything, the last request will be saved (the first one will be saved too but it changes to the last request)
and if there is any important data in database, you should block any other accesses to that row till the first user has done his job and saved the changes. (imagine a ticket reservation website which has to block any other users to be allowed to order the same ticket number till user finishes the order or cancel it.)
Also if you mean 2 different django projects using a single database, then its another yes. Basically they are like 2 different admins and all above conditions works for them too.
I will have a sidebar that appears on almost every page of my web app. The sidebar will have some drop-downs, which will consist of a total of say, 1000 different options, which are pulled from the db. Rather than doing the query to get these choices on every page load, I think it makes more sense to just do the query once, say in my config.py and store them in a variable that my views have access to. Is this OK? Is there a better way to accomplish this?
You could do that, but then you'd need to restart your Flask server every time you wanted to update the sidebar.
I'd use some other form of caching here. You could use Flask-Cache and memoize your query results. You can pick a nice long cache timeout, and then clear the cached result whenever you update the sidebar.
I have a multi-database Django Project (Python 2.7, Django 1.4 ) and I need that the database used in every single transaction triggered in a request be selected through the url from that request.
Ex:
url='db1.myproject.com'
Would trigger the use of 'db1' in every transaction, like this:
MyModel.objects.using('db1').all()
That is where my problem resides.
Its virtualy impossible include in every call of 'Model.objects' the '.using(dbname)' function (this project already have 2 dozen of models, and its build with scaling in mind, so it could have hundreds of models, and almost everymodel has (or might have) a implementation a of post_save/post_delete that acts as database triggers).
Is there a way of telling django that he should change the default database for this or that request ?
I already have a Custom QuerySet and a Custom Manager, so, if i could tell THEM the database should be this or that could do the trick to (i dont find anywhere how to get request data inside them).
Any help would be apreciated.
I think my answer to a similar question here will do what you need:
https://stackoverflow.com/a/21382032/202168
...although I'm a bit worried you may have chosen an unconventional or 'wrong' way of doing database load balancing, maybe this article can help with some better ideas:
http://engineering.hackerearth.com/2013/10/07/scaling-database-with-django-and-haproxy/
Hi I have some problems that has been bothering me for a week. I am running Selenium testing scripts on my dev machine, and in my test I would call simple script to delete accounts by their sub domain names:
for a in Account.objects.filter(domain = sub_domain):
a.delete()
The problem is that the query to find all such accounts are not returning correct results after the first time it is run (I use this query to clean up the database before each test). When I set a break point at this point, I can see the query return 0 records, even though in the database it has one record. I also set up mysql query log to see the actual query Django sent to mysql, and the query looks good, and will return correct result if I copy and paste to mysql command shell.
What am I missing? Why Django model query does not give me the correct result? MySQL is using InnoDB engine in case that makes any difference.
Transactions. Do a COMMIT in the shell.
This is a recurring problem, so I'm doing a shameless plug with a question in which I described details of the problem:
How do I deal with this race condition in django?