I have a big form on my site. When the users fill it out and submit it, most of the data just gets dumped to the database, and then they get redirected to a new page. However, I'd also like to use the data to query another site, and then parse the results. That might take a bit longer. It's not essential that the user sees these results right away, so I was wondering if it's possible to asynchronously call a function that will handle this, and then return an HttpResponse from my view like usual without making them wait?
If so... how? Any particular libraries I should look at?
User RabbitMQ and Celery with django. If you are deployed on EC2, also look at SQS
You create a message from the request-response cycle and an alternative process or a cron keeps checking off the messages.
Related
I have a python+django project and want to implement following functionality: by certain trigger (at certain time or by manual admin action) show message to all active users (i.e. to all sessions).
Webpush seems unnecessary here as far as django has nice built-in messages subframework.
I guess I may develop such functionality manually: make sql table session_messages, each time take snapshot of all sessions and show message, perform some checks to assure that message is shown only once and only to active users, etc. etc.
Question: maybe there is some nice little library that already does it? Or even maybe django messages are able to do it "from the box"?
I've googled a bit but found only packages for webpush integrations.
Thanks :)
You must implement a software architecture based on django channels, redis or rabbitmq and signals
Basically you must open a socket at the moment the user logs in, add the authenticated user to a group, and when you trigger the event with the signals send a message to the group
I have a django application that deploys the model logic and data handling through the administration.
I also have in the same project a python file (scriptcl.py) that makes use of the model data to perform heavy calculations that take some time, per example 5 secs, to be processed.
I have migrated the project to the cloud and now I need an API to call this file (scriptcl.py) passing parameters, process the computation accordingly to the parameters and data of the DB (maintained in the admin) and then respond back.
All examples of the django DRF that I've seen so far only contain authentication and data handling (Create, Read, Update, Delete).
Could anyone suggest an idea to approach this?
In my opinion correct approach would be using Celery to perform this calculations asynchronous.
Write a class which inherits from DRF APIView which handles authentication, write whatever logic you want or call whichever function, Get the final result and send back the JsonReposen. But as you mentioned if the Api takes more time to respond. Then you might have to think of some thing else. Like giving back a request_id and hit that server with the request_id every 5seconds to get the data or something like that.
Just to give a feedback to this, the approach that I took was to build another API using flask and normal python scripts.
I also used sqlalchemy to access the database and retrieve the necessary data.
I was wondering what is the best way to implement certain features that don't require templates. IE My application template can have multiple outlets, one of them being a notification service, and it would be constantly checking for new notifications and so on.
However, lets say there is a service for if someone logs into the app from a different browser, it automatically logs you out from the previous one. Basically the ember app would constantly have to be checking for these events to be happening on the server. But what if I had a lot of similar services/tasks that constantly work in the background of the client. How would I implement something like that?
Should it all be in one parent resource/route (maybe the application route), constantly (reloading the model) getting data from the server, waiting for the server to tell the client to log out or what not. Or would it be able to have something like a BackgroundService, that would not be a route, but would basically mimic one by constantly going thru the store to adapter to server to get data.
I know I could simply put a recursive function in the application route's model/aftermodel/beforemodel to be doing this, but I'm not sure if its proper and safe. I also don't know the app would react if this would be a simple ajax call, instead of using ember data. I know ember data does not have to be used, but I'm just wondering how proper/safe this is.
Good question, In the app I'm developing I have also several tasks which just run in the background. I mostly use Ember initializers for this, because you can create as many initializers as you want and separate all background tasks nicely (with the use of an initializer you can even add an order to the tasks in which they need to be started). I'm using ajax requests, but it should also be doable with ember-data. Although this depends off course on what you want to do with the task.
Keep in mind though that when having a lot of background tasks it might slow down your app a bit (because of all the traffic to and from the server). So don't refresh too often.
I've been trying to learn Django, but I'm still pretty much a web dev newbie, so please bear with me. Maybe something is just fundamentally wrong with this question...
For example, lets say some data exists in a JSON stream that is updated constantly. I'm trying to capture bits of that data and store it in my database, and it's displayed when I visit my Django built page. I guess there's two ways to do this:
In my views.py, it checks the data source, updates the database, and displays the information through a html file. This just seems like it's not the right way to do it. The source would be polled every time the page is viewed.
I would think the correct way to do it is have an application on the server that polls the data source every 1 minute or whatever and updates the database. The views.py only displays information from the database.
Am I even thinking about this correctly? I haven't found any information/examples on how to write the application that would sit on the server and constantly update the database.
Thanks!!
The second way is the right way to go about this, and the application that you would write to poll the json stream does not have to be written with django.
If you want to use the same models for the application, you can implement it as a custom management command, then run the command using cron at an interval. The command would poll the stream, update the database. Your view would then read the database and display the data.
If you want to do this in "realtime" (I use the word realtime here loosely), the server that is hosting the json stream should allow for "push" or a socket connection that will remain open.
One of my view functions is a very long processing job and clearly needs to be handled differently.
Instead of making the user wait for long time, it would be best if I were able to lunch the processing job which would email the results, and without waiting for completion notify the user that their request is being processed and let them browse on.
I know I can use os.fork, but I was wondering if there is a 'right way' in terms of Django. Perhaps I can return the HTTP response, and than go on with this job somehow?
There are a couple of solutions to this problem, and the best one depends a bit on how heavy your workload will be.
If you have a light workload you can use the approach used by django-mailer which is to define a "jobs" model, save new jobs into the database, then have cron run a stand-alone script every so often to process the jobs stored in the database (deleting them when done). You can use something like django-chronograph to manage the job scheduling easier
If you need help understanding how to write a script to process the job see James Bennett's article Standalone Django Scripts for help.
If you have a very high workload, meaning you'll need more than a single server to process the jobs, then you want to use a real distribute task queue. There is a lot of competition here so I can't really detail all the options, but a good one to use with for Django apps is celery.
Why not simply start a thread to do the processing and then go on to send the response?
Before you select a solution, you need to determine how the process will be run. I.e is it the same process for every single user, the data is the same and can be scheduled regularly? or does each user request something and the results are slightly different ?
As an example, if the data will be the same for every single user and can be run on a schedule you could use cron.
See: http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/
or
http://docs.djangoproject.com/en/dev/howto/custom-management-commands/
However if the requests will be adhoc and you need something scalable that can handle high load and is asynchronous: what you are actually looking for is a message queing system. Your view will add a request to the queue which will then get acted upon.
There are a few options to implement this in Django:
Django Queue service is purely django & python and simple, though the last commit was in April and it seems the project has been abandoned.
http://code.google.com/p/django-queue-service/
The second option if you need something that scales, is distributed and makes use of open source message queing servers: celery is what you need
http://ask.github.com/celery/introduction.html
http://github.com/ask/celery/tree