Where to use Celery in Django? - django

I'm reading this tutorial of Django Celery and I'm wondering since Django is a web development framework, where Celery will be actually used to queue the tasks in web development ?
Give your suggestions using examples .

Basically, anything that doesn't have to happen in the typical request-response web application cycle.
The goal is generally to improve performance by doing those tasks in the background.
A few examples:
Sending notification emails.
Updating or invalidating non-critical caches
Calling an external web service

Related

Design question for flask real-time messages

My question relates primarily to website design. As a novice programmer, I would like to have a tab in my flask website dedicated to serving real-time messages (likely using pika and a remote rabbitmq server). The reason for this is to serve the visitors real-time information without them having to refresh on the site.
I'm using Blueprints, and the reading i've done shows that the integration of flask-socketio and blueprints is far from elementary, so i'm wondering if there are easier options to push updates out to the end user. Should i create websockets directly? Is this easier in a flask blueprint environment? Should i use long polling? What might be the preferred route here?

running slack events adapter within a flask app

The new slack events api with slack events adapter (https://github.com/slackapi/python-slack-events-api) is definitely powerful and abstracts out a lot of unnecessary code including handling http retries and so on. Therefore, I am considering switching to the adapter model from the normal events api.
However, the adapter is a flask app on its own. My current event listener was a flask blueprint. I will therefore have to run 2 flask apps now to support the events adapter. Although, running 2 flask apps is possible, sharing data across the 2 apps becomes difficult and leads to unnecessary complication.
Therefore, I wanted to know from you guys, if there is an easy and elegant way to run multiple flask apps which use same configurations, need similar installations and share data and models across.
I figured it out. Its fairly straightforward. I just went through their github code and I can replicate the rest of the adapter code without starting the server which kinda resolves the problem.

Consume kafka messages from django app

I'm designing a django based web application capable of serving via Web sockets data that need to be consumed from Kafka topic.
At this time, I came up with a solution splitted in two components: one component which consumes from kafka, perform some basic operations over retrieved data, and send the result to the django app using an http request.
After request have been received, a message is written over a specific django channel.
Is there a better architecture to address this kind of scenario? Should I enclose all the Kafka part in a "while True" loop in a celery async task? Should I spawn a new process when django starts? If so, can I still use the django signals to send the data via web socket?
Thanks,
Fb
yes, you can use your django code/repository and build separate app/program to deal with kafka queue and database through django ORM
just add at begin of this program code like
sys.path.append(os.getcwd())
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<your_app>.settings")
django.setup()
and then you can use your models in this program, like
from <your_app>.models.timeslots import TimeSlotReserve
also good idea is to add some multithreading to this separate app

Is django server based on blocking code

I read somewhere that Django is a blocking code. Does that mean when I deploy my code, the server will be able to serve only one request at a time? Do I need to use some other framework like tornado to solve this purpose? Is Django only meant for development and debugging purpose? If it does not solve the purpose of deployment why not use node.js or some other framework.
It means each thread or process can only serve one request at a time. But your server is normally configured to spin up multiple processes.

How to use Tornado work with Django? Is it a good solution?

I've a huge django project and have to use Instagram API and its subscriptions model to work. For the subscriptions, my server has to be very responsive and be ready to work asynchronously to set up a hook so as to receive notifications once the user posts. Or that's what the documentation suggests. Now will it be a good thing to use Tornado there? Just for that small part or can I do it using Django in an effective way? if so, how?
You can use the WSGI container on top of Tornado to host any WSGI application, including Django, however, when you do that the WSGI application is still running as a blocking application and will not magically be running as an asynchronous application. So, when Django is handling a request there is no ability to handle another request at the same time within Django. The solution at that point is not much different to running a single threaded WSGI server and you would need to have multiple Tornado instances to handle concurrent requests.
So all really depends on what you mean by asynchronous. You certainly can't make use of Tornado's direct asynchronous programming API in Django. Thus there isn't really any great benefit from using Tornado with Django via the WSGI interface.
As I understand you are talking about this paragraph in Instagram docs
You should build your system to accept multiple update objects per payload - though often there will be only one included. Also, you should acknowledge the POST within a 2 second timeout--if you need to do more processing of the received information, you can do so in an asynchronous task.
That's another type of "asynchronous" that Tornado provides.
I think Django + Celery will suite better for this.
Your application will work in this way:
You receive JSON-data from Instagram
Create a celery-task, e.g. instagram_process.delay(request.raw_post_data) or instagram_process.delay(request.body) according to your Django version
Response to Instagram with 200 status code
In instagram_process task you do all your procession - parse JSON, store it do database and anything else you need.
If you want to check X-Hub-Signature you can either do it between steps 1 and 2, or pass this header to the task and verify the signature at step 4.
You can use tornado.wsgi to integrate Tornado with other WSGI compliant frameworks. Check out this demo project for details:
https://github.com/bdarnell/django-tornado-demo