I'm designing a django based web application capable of serving via Web sockets data that need to be consumed from Kafka topic.
At this time, I came up with a solution splitted in two components: one component which consumes from kafka, perform some basic operations over retrieved data, and send the result to the django app using an http request.
After request have been received, a message is written over a specific django channel.
Is there a better architecture to address this kind of scenario? Should I enclose all the Kafka part in a "while True" loop in a celery async task? Should I spawn a new process when django starts? If so, can I still use the django signals to send the data via web socket?
Thanks,
Fb
yes, you can use your django code/repository and build separate app/program to deal with kafka queue and database through django ORM
just add at begin of this program code like
sys.path.append(os.getcwd())
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<your_app>.settings")
django.setup()
and then you can use your models in this program, like
from <your_app>.models.timeslots import TimeSlotReserve
also good idea is to add some multithreading to this separate app
Related
The new slack events api with slack events adapter (https://github.com/slackapi/python-slack-events-api) is definitely powerful and abstracts out a lot of unnecessary code including handling http retries and so on. Therefore, I am considering switching to the adapter model from the normal events api.
However, the adapter is a flask app on its own. My current event listener was a flask blueprint. I will therefore have to run 2 flask apps now to support the events adapter. Although, running 2 flask apps is possible, sharing data across the 2 apps becomes difficult and leads to unnecessary complication.
Therefore, I wanted to know from you guys, if there is an easy and elegant way to run multiple flask apps which use same configurations, need similar installations and share data and models across.
I figured it out. Its fairly straightforward. I just went through their github code and I can replicate the rest of the adapter code without starting the server which kinda resolves the problem.
I'm using django as backend. While reading stuff about meteor, i found django-ddp.
I searched a lot, but I didn't get what django-ddp is for.
I understood that you can use it to connect meteor to your django backend, but what is the use case?
How does the client connect to django and/or meteor? Does meteor have to run on the same server? How are the http requests handled?
Maybe a tiny example would help me to get this.
For me important: Can i use this to combine benefits of django and meteor?
Django DDP provides a Meteor compatible, realtime, latency compensated backend framework for Django (Python) models. It can also serve your Meteor frontend code (HTML/JS/CSS/...) allowing you to avoid using Meteor (and node.js) on the server, whilst serving regular Django views at the same time.
Django is a respected web framework with a powerful object relational mapper (ORM), with support for schema migrations included by default. Django DDP is efficient and secure, using gevent to handle HTTP requests and manage concurrency at the process level, and multiple processes (across multiple hosts) to allow scaling-out to serve many clients simultaneously. WebSockets are handled using gevent-websocket. Combining these aspects with the realtime, latency compensated benefits of Meteor does indeed give you the advantages of both (unless you prefer to run node.js on your backend servers).
If Django DDP is used to serve your Meteor app then the client (browser) will connect to Django DDP automatically. Otherwise, you can connect your Meteor app to Django DDP and use the Django DDP connection like this:
if(Meteor.isClient) {
Django = DDP.connect('http://ddp.example.com/');
Tasks = new Mongo.Collection('myapp.Tasks', {connection: Django});
Django.subscribe('Tasks', {
onReady: function(error, result) {
// Log each matching Task to the browser console in a table
console.table(Tasks.find().fetch());
}
});
}
If you're serving your Meteor app from Django DDP then drop the DDP.connect line and omit the second parameter to new Mongo.Collection.
You might find the Todos example app a useful place to start. It includes a full working example of how to write both the Meteor client app and the Django DDP server app.
Disclaimer: I'm the author of Django DDP - sorry if parts of my answer sound like marketing guff but I'm just trying to answer the first part of the question.
I am making a website with django now and I want to implement a live notification feature like the one on facebook or SE.
I did some research and it seems although there's two options: ajax long polling and websockets, the latter is the way to go.
However, as you know the go to plugin for websocket 'socket.io' turns out to be a node.js plugin and the django port only seems to support python 2 and the project seems pretty much dead. I am using python 2.7 as my project interpreter but I want to future proof myself so that if I upgrade to python3 later, I don't find myself not being able to use this functionality.
So my question is this:
Is there a straight forward and future ready way to implement websocket which will be used to send live notifications and chats in django env?
Django itself is build in blocking manner, i.e. with synchronous approach. So, you cannot open persistent websocket with django app, as it will block entire django thread.
If you want to enable notification/chat within django project environment, i would recommend to use centrifuge. It is written in python, but async (non-blocking) framework is used: tornado.
But, you don't need to even know how it works, as it provides simple REST api to communicate with it.
Simplified workflow, check docs for more details:
Start centrifuge at same server, as your django project (or on another but with low latency between them)
Your front-end will open websocket with centrifuge, not with django project.
When you need to send notification, send it to centrifuge from django via REST api, and centrifuge will deliver it to needed clients!
I've already tried it and it works!
Django doesn't provide what you're looking for out of the box. You'll have to use a third party library. One that works across frameworks is Pusher.
I think you must go for Firebase it gives you awesome synchronization and any how you are going to use chat on frontend so its does not have to do anything with django environment so you can update you backend asynchron in callback with firbase. Also firebase with AngularJS provides you really really awesome three way binding.
I'm reading this tutorial of Django Celery and I'm wondering since Django is a web development framework, where Celery will be actually used to queue the tasks in web development ?
Give your suggestions using examples .
Basically, anything that doesn't have to happen in the typical request-response web application cycle.
The goal is generally to improve performance by doing those tasks in the background.
A few examples:
Sending notification emails.
Updating or invalidating non-critical caches
Calling an external web service
I've a huge django project and have to use Instagram API and its subscriptions model to work. For the subscriptions, my server has to be very responsive and be ready to work asynchronously to set up a hook so as to receive notifications once the user posts. Or that's what the documentation suggests. Now will it be a good thing to use Tornado there? Just for that small part or can I do it using Django in an effective way? if so, how?
You can use the WSGI container on top of Tornado to host any WSGI application, including Django, however, when you do that the WSGI application is still running as a blocking application and will not magically be running as an asynchronous application. So, when Django is handling a request there is no ability to handle another request at the same time within Django. The solution at that point is not much different to running a single threaded WSGI server and you would need to have multiple Tornado instances to handle concurrent requests.
So all really depends on what you mean by asynchronous. You certainly can't make use of Tornado's direct asynchronous programming API in Django. Thus there isn't really any great benefit from using Tornado with Django via the WSGI interface.
As I understand you are talking about this paragraph in Instagram docs
You should build your system to accept multiple update objects per payload - though often there will be only one included. Also, you should acknowledge the POST within a 2 second timeout--if you need to do more processing of the received information, you can do so in an asynchronous task.
That's another type of "asynchronous" that Tornado provides.
I think Django + Celery will suite better for this.
Your application will work in this way:
You receive JSON-data from Instagram
Create a celery-task, e.g. instagram_process.delay(request.raw_post_data) or instagram_process.delay(request.body) according to your Django version
Response to Instagram with 200 status code
In instagram_process task you do all your procession - parse JSON, store it do database and anything else you need.
If you want to check X-Hub-Signature you can either do it between steps 1 and 2, or pass this header to the task and verify the signature at step 4.
You can use tornado.wsgi to integrate Tornado with other WSGI compliant frameworks. Check out this demo project for details:
https://github.com/bdarnell/django-tornado-demo