I have a django view with this function to get the data for a template:
def get_context_data(self, **kwargs):
context = super(MyView, self).get_context_data(**kwargs)
context['extra_data'] = a_long_running_function()
return context
The extra_data is displayed in a table. As the above function indicates, the page takes a long time to load due to calculation of extra_data.
So how can I show the page straight away, and then update the tablewhen extra_data is computed?
I understand how I can use celery to make a_long_running_function execute asynchronously, but I dont know how to then make the page (which is now loaded, but missing data for the table), get that data and update automatically?
If you plan in going ahead with celery, you will need 2 views:
1.viewA that loads the main page (without the extra_data - maybe a spinning gif animation in it's place in the HTML, to convey to the user that there is still data to be loaded in the page). This view will also start the celery task (but will not wait for it to complete). It would look similar to:
def viewA(request):
task = a_long_running_function.delay()
return render_to_response('viewA.html', {'task_id': task.id})
2.viewB that will be accessed via AJAX after the user's browser loads viewA (it's purpose will be to provide the extra_data which was not loaded by viewA). It would look similar to:
def viewB(request, task_id):
extra_data = a_long_running_function.AsyncResult(task_id)
if extra_data.ready():
return render_to_response('viewB.html', {'extra_data': extra_data.get()})
return HttpResponse('')
Once the user's browser finishes loading viewA, you will need a bit of javascript to start running AJAX requests every X seconds/minutes to viewB to attempt to retrieve the celery task result (based on the celery task id that is available). Once the AJAX request successfully retrieves the task result from viewB, it can make it visible to the user.
Anybody interested in asynchronous updating a template using AJAX can use django-async-include (GitHub repository).
This project makes it easy changing an static block inclusion to a asynchronous one. That's perfect for inclusion of computational-heavy template block.
Disclaimer: I'm the developer of this project.
Related
I have a Django .save() execution that loops at n times.
My concern is how to guard against network errors during saving, as some entries could be saved while others won't and there could be no telling.
What is the best way to make sure that the execution is completed?
Here's a sample of my code
# SAVE DEBIT ENTRIES
for i in range(len(debit_journals)):
# UPDATE JOURNAL RECORD
debit_journals[i].approval_no = journal_transaction_id
debit_journals[i].approval_status = 'approved'
debit_journals[i].save()
Either use bulk_create / bulk_update to execute a single DB query, or use transaction.atomic as decorator for your function so that any error on save will rollback your database before your function was run.
Try something like below (I suppose your model name is DebitJournal and debit_journals is a list).
for debit_journal in debit_journals:
debit_journal.approval_no = journal_transaction_id
debit_journal.approval_status = 'approved'
DebitJournal.objects.bulk_update(debit_journals, ["approval_no", "approval_status"])
If debit_journals is a QuerySet you can also try
debit_journals.update(approval_no=journal_transaction_id, approval_status='approved').
It depends of what you call a network error, if it's between the user and the django application or between the django application and the database. If it's only between the user and the app, note that if the request has been sent correctly even if the user lose the connection afterward the objects will be created. So a user might not have the request response, but objects will still be created.
If it's between the database and the django application some objects might still be created before the error.
Usually if you want a "All or Nothing" behaviour you should use manual transaction as described there: https://docs.djangoproject.com/en/4.1/topics/db/transactions/
Note that if the creation is really long you might hit the request timeout. If the creation takes more than a few seconds you should consider making it a background task. The request is only there to create the task.
See Python task queue alternatives and frameworks for 3rd party solutions.
I am making a Django application for an experiment and it will offer the possibility to the user to be able to download really big image files to a specific directory in the server.
I have a PoC of that working, where the user sees a button clicks on it and the process to download begins. I have put the download process inside views.py which means it occurs upon visiting a specified URL.
The code responsible for the download part similar to the following:
with open(loc, "wb") as f:
response = s.get(pos, stream=True)
total_length = response.headers.get('content-length')
if total_length is None:
f.write(response.content)
else:
dl = 0
total_length = int(total_length)
for data in response.iter_content(chunk_size=4096):
dl += len(data)
f.write(data)
The problem is that for some images which are > 500MB the process takes quite some time and there are several issues, like if the user closes the app, or clicks away then the process stops.
How can I make it in such a way so the user visits that page and clicks that button and the download process begins on the background without binding the user to stay to that page.
Also maybe creating another endpoint which could update the user about the status of the download process.
Please feel free to share any ideas or thoughts.
I'm setting up an application working with Django and Keras. It takes a lot of time to take my uploaded document, prepare, train and to show my results. That's why I'm trying to split my code into different phases. But I don't know how to trigger a function using my created object before. Do you know how to connect a button to trigger a process via using created object?
Thanks in advance!
To trigger an action from a template view you need to set up an endpoint and use needed class/func there. You can call it with ajax if that's you want.
path('action/', app.views.action, 'action')
def action(request):
my_util_class.my_function()
return HttpResponse('')
Button
I am working on a legacy Django application that has a lot of third party dependencies, one of which is the storage backend for file uploads. I was recently tasked with replacing our legacy third party cloud storage vendor with a newer cloud storage vendor (Box).
The cloud storage is implemented as a custom storage backend and used as the "storage" parameter in FileFields in models throughout the app. I'm basically trying to figure out what exactly happens in storage if you have a FileField in a model and you create a ModelForm based on that model, then you call "save" on the form.
It seems that a lot of stuff is going on and some of it is causing connection problems with the cloud storage API.
I tried reading the Django source to follow it and got all the way down to where the model is deciding if it should do an update (by doing an "exists" check in storage) or an insert.
Once it decides to do an insert, I noticed a call to my cloud storage backend occurs to upload the file (presumably non blocking?) as the insert sql is being generated.
Somewhere in here, connections to the cloud storage begin to hang and become unresponsive. At least, all I see in logs is
INFO requests.packages.urllib3.connectionpool - Starting new HTTPS connection (1): upload.box.com"
and no further info or response.
Unlike the previous cloud storage, the new one issues JWT's per session instead of having a static auth token that you simply pass every time. If I do not use Django's ModelForm with its magical "save" method, but call methods directly on the models with the FileFields, I do not encounter the connection problem - I get responses just fine from the cloud storage API.
So, I'm thinking there must be some kind of concurrency issue when calling "save" on a form that affects a model with a FileField?? I'm a little stumped. The code is involved, so it is hard to copy here, but basically it comes down to:
class CustomStorage:
def __init__(self):
set up storage API client,
authenticate client instance, etc
def _save(self, name):
call storage API methods to upload file
** includes a retry loop with file renaming
algorithm to avoid name conflicts, as
cloud API does not allow duplicate file
names
def exists(self, name):
call storage API methods to determine if file name conflict exists
def _open(self, name):
call storage API methods to download file
custom_storage = CustomStorage()
class ExampleModel(models.Model):
name = models.CharField(maxlength=255)
file_ref = FPFileField(upload_to="uploads", storage=custom_storage) #because we also have a dependency on FilePicker, now called FileStack
class ExampleModelForm(ModelForm):
file_ref = CustomFilePickerField()
class Meta:
model = ExampleModel
fields = ('name', 'file_obj')
form = ExampleModelForm()
model = form.save() # --> connection problem with
# cloud storage API starts here
# if I were to call ExampleModel.objects.create(...),
# the storage upload process would work fine
Is there some gotcha I'm not aware of that Django experts would know about implementing custom storage backends for Django based on cloud storage APIs?
Turns out that the problem was our usage of a deprecated version of a FilePicker field for the uploaded document model field, combined with an old bug in a stale version of Requests. Occasionally, this field would return a Django file instance wrapped around a cStringIO.StringIO object instead of a vanilla file object. This ended up running into a bug in the Requests library which caused stalled responses when chunking for a multi-part POST when the payload is a StringIO instance. Because upgrading is not an option, I solved the issue by detecting if the the underlying Django FileField File is not a file object and re-wrapping it in a ContentFile instance and seek-ing to 0, in order to play nice with the older version of Requests. If anyone knows of a better alternative, by all means, please let me know.
I am using django as a framework to build a content management system for a site with a blog.
Each blog post will have a route that contains a unique identifier for the blog post. These blog posts can be scheduled and have an expiry date. This means that the routes have to be dynamic.
The entire site needs to be cached and we have redis set up as a back end cache. We currently cache rendered pages against out static routes, but need to find a way of caching pages against the dynamic routes (and invalidating them when the blog posts expire.)
I could use a cron job but it isn't appropriate because...
a) New blog posts go live rarely and not periodically
b) Users can schedule posts to the minute. This means that a cron job would have to run every minute which seems like overkill!
I've just found the django-cacheops library, which seems to do exactly what I need (schedule the invalidation of our cache and invalidate them via signals). Is this compatible with our existing setup and how easy is the setup?
I assume this is a pretty common problem - does anyone have any better ideas than the above?
I can't comment on django-cacheops because I've never used it, but Redis provides a really easy way to do this using the EXPIRE command:
Set a timeout on key. After the timeout has expired, the key will automatically be deleted.
Usage:
SET some_key "some_value"
EXPIRE some_key 10
The key some_key will now automatically be cleaned/deleted by Redis in 10 seconds. If you need to delete blog posts' cache knowing when they should be deleted from the outset, this should serve your needs perfectly.
Cacheops invalidate cache when a post is changed, that's its primary use. But you can also expire by timeout:
from cacheops import cached_as, cached_view_as
# A queryset
post = Post.objects.cache(timeout=your_timeout).get(pk=post_pk)
# A function
#cached_as(Post.objects.filter(pk=post_pk), timeout=your_timeout)
def get_post_data(...):
...
# A view
#cached_view_as(Post, timeout=your_timeout)
def post(request, ...):
...
However, there is currently no way you can specify timeout depending on cached object.