I'm new to Django and Channels and so far I couldn't find any solution to the issue that I face:
I need to communicate with an external WebSocket, to process received data and then sent it to some Channels groups or maybe start some Celery tasks based on that output.
As I've understood it's not a good practice to put that logic inside Consumer. What is the right way of doing this in Django?
Thanks
It'S probably not at all best practice to do it in Django in first place. Django is a Web Framework, that processes individual http requests. Connecting to websockets for potentially longer running process should happen in another component of your architecture.
Related
I know the question I am going to ask is a bit duplicate. But, I am still asking as I want to know the latest technologies and I am a bit lost after researching for a few hours.
I have a Raspberry Pi logging real-time temperature and humidity. Now, I am writing a flask app to push these data to clients who (subject to rights) will be able to observe continuously without refreshing the dashboard/page.
What will be the best option to make an efficient system, keeping in mind that there will be multiple sensors in the future? The options I find:
Ajax
WebSocket
Framework e.g. bokeh or dash
MQTT
Please give me your opinions.
Good choice if you want to write your backend using Python is:
Server: Flask with Sokcet.IO + InfluxDB for real-time data storing
Frontend: Some JS framework or pure Js + websocket
UPD (this message is too long to post it to comments):
https://www.smashingmagazine.com/2018/02/sse-websockets-data-flow-http2/
The thing is that I'm not talking that websocket is the right solution for all possible cases/problems and should be used everywhere. Obviously it's depends on your needs and your project architecture. I think that article can help you to make a choice: if your app architecture requires full-duplex browser-server connection - you can use websocket for this and that will work for you, but if your frontend requires only one-way data send direction - from server to browser - you can use SSE, as the article says about SSE: "our main flow of data is from the server to the client and in much fewer occasions from the client to the server". To sum it up, you need to think about your application architecture and about how data needs to be sent between browser and server to choose right technology. Also, if you don't want to use neither websocket nor SSE - you can use ajax to pull data from server and that will also work for you.
I'm working on a long request to a django app (nginx reverse proxy, mysql db, celery-rabbitMQ-redis set) and have some doubts about the solution i should apply :
Functionning : One functionality of the app allows users to migrate thousands of objects from one system to another. Each migration is logged into a db, and the users are provided the possibility to get in a csv format the history of the migration : which objects have been migrated, which status (success, errors, ...)
To get the history, a get request is sent to a django view, which returns, after serialization and rendering into csv, the download response.
Problem : the serialisation and rendering processes, for a large set of objects (e.g. 160 000) are quite long and the request times out.
Some solutions I was thinking about/found thanks to pervious search are :
Increasing the amount of time before timeout : easy, but I saw everywhere that this is a global nginx setting and would affect every requests on the server.
Using an asynchronous task handled by celery : the concept would be to make an initial request to the server, which would launch the serializing and rendering task with celery, and give a special httpresponse to the client. Then the client would regularly ask the server if the job is done, and the server would deliver the history at the end of processing. I like this one but I'm not sure about how to technically implement that.
Creating and temporarily storing the csv file on the server, and give the user a way to access it & to download it. I'm not a big fan of that one.
So my question is : has anyone already faced a similar question ? Do you have advises for the technical implementation of the solution (#2), or a better solution to propose me ?
Thqnks !
Clearly you should use Celery + RabbitMQ/REDIS. If you look at the docs it´s not that hard to setup.
The first question is whether to use RabbitMQ or Redis. There are many SO questions about this with good information about pros/cons.
The implementation in django is really simple. You can just wrap django functions with celery tasks (with #task attribute) and it´ll become async, so this is the easy part.
The problem I see in your project is that the server who is handling http traffic is the same server running the long process. That can affect performance and user experience even if celery is running on the background. Of course that depends on how much traffic you are expecting on that machine and how many migrations can run at the same time.
One of the things you setup on Celery is the number of workers (concurrent processing units) available. So the number of cores in your machine will matter.
If you need to handle http calls quickly I would suggest to delegate the migration process to another machine. Celery/REDIS can be configured that way. Let´s say you´ve got 2 servers. One would handle only normal django calls (no celery) and trigger celery tasks on the other server (the one who actually runs the migration process). Both servers can connect to the same database.
But this is just an infrastructure optimization and you may not need it.
I hope this answers your question. If you have specific Celery issues it would be better to create another question.
Hello Stackoverflowers,
We're developing an online board-game (think online monopoly) site using Python for the backend.
We use Django for the non realtime stuff (authenticating, player profiles, ranking...). The chat server is implemented using socket.io and Tornodo. The game server part is what caused us problems.
We currently (that could change) also use Tornado and socket.io, each Tornado instance is located at a gameX.site.com address on a (maybe) different server and host several games simultaneously (much like a chat server in fact, except that messages would not go to all users but only to the ones involved in the same game).
What cause us trouble is how to we update the Django instance (game log, score, and so on) as games progress. Also we'd like to use Django for authentication as each player would ask the Django server to join the game and be given a disposable id/password couple just for it. Obviously we would have to communicate those to the game server in some way.
At first the chosen solution was to use something like Redis as a bidirectional message queue, Django would post is/password to Redis and the Tornado would then querying Redis on incoming connection. Also a Django cron would run every minute or so to deal with the waiting message. But we fear that frequently and possibly long running cron would impede the main site since the PostgreSQL database is hosted on the same server as Django (and some Game server may also run on the same machine).
We could alternatively wait for a player to request a ranking updated to process the past games results but we fear such an indefinite delay will skew the overall ranking (and experience) and would possibly cause data loss.
We could use Celery/RabbitMQ to update the Main database using Django ORM out of the Tornado processes, but would it be possible to use the same solution to communicate the temporary id/password to the game server ? It doesn't look like you can post a message to Celery and retrieve it on an other side.
Thank for your insight.
I have been trying to solve this for 2 weeks and I have not been able to reach a solution.
Here is what I am trying to do:
I need a web application in which users can upload a video; the video is going to be transformed using opencv's python API. Since I have Python's API for opencv I decided to create the webapp using Django. Everything is fine to that point.
The problem is that the video transformation is a very long process so I was trying to implement some real time capabilities in order to show the user the video as it is transformed, in other words, I transform a frame and show it to the user inmediatly. I am trying to do this with CoffeScript and io sockets following some examples; however I havent been successful.
My question is; what would be the right approach to add real time capabilities to a Django application ?
I'd recommend using a non-django service to handle the websockets. Setting up websockets properly is tricky on both the client and server side. Look at pusher.com for a free/cheap solution that will just work and save you a whole lot of hassle.
The initial request to start rendering should kick off the long-lived process, and return with an ID which is used to listen to the websocket for updates.
Once you have your websockets set up, you can send messages to the client about each finished frame. Personally I wouldn't try to push the whole frame down the websocket, but rather just send a message saying the frame is done with a URL to get the frame. Then normal HTTP with its caching and browser niceties moves the big data.
You're definitely not choosing the easy path. The easy path is to have your long-lived render task update the render state in the database, and have the client poll that. Extra server load, but a lot simpler.
Django itself really is focused on doing one kind of web interface, which is following the HTTP Request/Response pattern. To maintain a persistent connection with clients, which socket.io really makes dead simple, you need to diverge a bit from a normal Django installation.
This article discusses the issue of doing real-time with Django, with the help of Orbited and Twisted. It's rather old, and it relies on Comet, which is not the preferred way of doing real-time these days.
You might benefit a lot by going for Socket.io on the client, and something like Tornado (wiki) + Tornado client for Socket.io. But, if you really want to stick with Django for the web development (which Tornado also provide), you would need to make the two work together internally, each handling their particular use case.
Finally, this other article discusses how to make Django work with gevent (an coroutine-based networking library for Python) and Socket.io, which might well be your best option otherwise.
Don't hesitate to post questions/comments as they pop up!
What is a good way to implement server to client callbacks (events) with a web service?
We already have a SOAP-based API exposed by mod_perl running on Apache, but it is one way client to server. We need to notify the client of certain changes on the server by other clients.
What is a good way to make this work? Available modules for perl?
One thing we thought of was to make a long running web request that would finally respond when an event occurred, but this would seem to keep too many mod_perl processes running if many clients were connected.
One way is the one you mentioned, long-running polls.
The other is websockets. There are many Perl frameworks which are able to deal with websockets requests, one of which is Mojolicious.
See the docs at Mojolicious::Lite#WebSockets for some examples.
I would suggest in your case running the websockets webapp separately from the mod_perl server, and "just" using Apache to reverse-proxy the mojolicious app.
Have you actually tried your first approach ? How many max concurrent clients would that be? A few dozens should be manageable by apache. Just guessing.
Have you tried this low-tech method "Watching long processes through CGI", or is it what you wanted to get rid of in the first place?