What should i use to implement comet on django?
All things i've found on google seems outdated. Some people point to orbited.org or hookbox.org, but both of them are just dead now. How people solve this problem nowadays?
I would check out Pusher which is a third party service that will allow you to push events to your browser with a drop-dead simple API. They have a free sandbox plan that comes with 100k messages per day and 20 connections.
Alternatively, you could run APE on your server to push events down.
Django isn't really designed for long polling and comet.
Related
We use gunicorn with django and django-telegrambot. We also have a MQTT client in an own app. When some MQTT messages arrive we send Telegram messages and the other way around. The Problem is now that when we use gunicorn with multiple workers, we have multiple MQTT Clients, so that when a MQTT message arrives we will send multiple times the same Telegram message.
When we use gunicorns preload with workers, we only have one MQTT client, but then all processes share the same Telegram TCP connection and we get wired SSL errors. As an alternative we could use only use on process and multiple threads, but then sometimes MQTT and Telegram messages gets not processed (idk why).
Is there a way to get this running?
Instead of using webhooks one could use botpolling, but django-telegrambot says:
Polling mode by management command (an easy to way to run bot in local machine, not recommended in production!)
I'm not familiar with the django-telegrambot library, so I can't judge why the authors chose to make this statement (maybe ask on the GitHub repository …). However, both polling and webhooks are officially supported by Telegram (see here). IMHO both have pros and cons. Webhooks may have a slight performance benefit over polling, but also require more work to set up. Polling requires you to continuously fetch for updates, which can be seen as downside. OTOH with webhooks you have to have a webserver running. For small to medium sized bots (in terms of usernumber), polling should be fine - I'm using polling without problems for my (rather small) bots.
Please take this with a grain of salt as I'm far from being an expert on networking topics.
I know the question I am going to ask is a bit duplicate. But, I am still asking as I want to know the latest technologies and I am a bit lost after researching for a few hours.
I have a Raspberry Pi logging real-time temperature and humidity. Now, I am writing a flask app to push these data to clients who (subject to rights) will be able to observe continuously without refreshing the dashboard/page.
What will be the best option to make an efficient system, keeping in mind that there will be multiple sensors in the future? The options I find:
Ajax
WebSocket
Framework e.g. bokeh or dash
MQTT
Please give me your opinions.
Good choice if you want to write your backend using Python is:
Server: Flask with Sokcet.IO + InfluxDB for real-time data storing
Frontend: Some JS framework or pure Js + websocket
UPD (this message is too long to post it to comments):
https://www.smashingmagazine.com/2018/02/sse-websockets-data-flow-http2/
The thing is that I'm not talking that websocket is the right solution for all possible cases/problems and should be used everywhere. Obviously it's depends on your needs and your project architecture. I think that article can help you to make a choice: if your app architecture requires full-duplex browser-server connection - you can use websocket for this and that will work for you, but if your frontend requires only one-way data send direction - from server to browser - you can use SSE, as the article says about SSE: "our main flow of data is from the server to the client and in much fewer occasions from the client to the server". To sum it up, you need to think about your application architecture and about how data needs to be sent between browser and server to choose right technology. Also, if you don't want to use neither websocket nor SSE - you can use ajax to pull data from server and that will also work for you.
In my website, I'd like to create a public API that would allow clients (unknown people) to interact with my services. A classic REST API would work well in that case.
However, I need to be able to send events to the clients too. These events are not related to client HTTP requests. I saw "webhooks" are a way to deal with this. If I understood well, with webhooks, my service would send HTTP POST requests to a URL specified by the client, with event data inside this request.
I think websocket can be used too as a solution for this full-duplex communication need.
What I want to know, is which method would be the simplest for clients to implement to talk to my services? Simplicity is the key point here.
The hard thing is that my clients can use various technologies (full websites with HTTP servers, iOS/Android apps without server, etc.)
What are implications for clients if I use REST API + webhooks? Websockets? etc?
How to make a choice?
Hope it's clear (but not sure). Thanks :)
I would consider webhooks a simpler solution. And yes, you understood it well, that with webhooks, a developer using your API would register a URL where your backend would POST event data. It's a common pattern that's used in APIs.
A great benefit of using a webhooks design is that a client/server connection does not need to stay open. After all, if events occur infrequently (i.e. only a few times per hour, or per day) or keeping a consistent connection open is a challenge, establishing a connection only when it's needed is rather efficient.
The challenge of using webhooks for you, the API provider, is designing an evented backend system that deals with change of state detection and reliable webhook calling mechanisms (i.e. dealing with webhook receiver URLs that are unresponsive or throw errors).
The challenge of using webhooks on the developer end is that they need to stand up a reliable web server that listens for the event POST data from your server.
Realtime APIs (i.e. based on Websockets, Bayeux/CometD) are really swell because that live connection means that new connections do not have to be established, which is particularly useful with very chatty sessions. Additionally, there are a lot of projects and companies out there that have taken care of the heavy lifting on the server and client with fully-baked libraries. One of those is Fanout.io which makes pushing messages between the client/server possible with just a few lines of code, utilizing XMPP, Bayeux, and Websockets when possible.
(I am not affiliated with Fanout, but I have used it)
So, to sum it up, webhooks are simple mostly because you are already familiar with the architecture needed to implement them, and the pattern is a well traveled one. If you are leaning toward a persistent connection approach, I would look at tools/platforms like Fanout because it takes care of the heavy lifting (i.e. subscribe/publish, concurrent connection scale, client/server libraries).
I have been trying to solve this for 2 weeks and I have not been able to reach a solution.
Here is what I am trying to do:
I need a web application in which users can upload a video; the video is going to be transformed using opencv's python API. Since I have Python's API for opencv I decided to create the webapp using Django. Everything is fine to that point.
The problem is that the video transformation is a very long process so I was trying to implement some real time capabilities in order to show the user the video as it is transformed, in other words, I transform a frame and show it to the user inmediatly. I am trying to do this with CoffeScript and io sockets following some examples; however I havent been successful.
My question is; what would be the right approach to add real time capabilities to a Django application ?
I'd recommend using a non-django service to handle the websockets. Setting up websockets properly is tricky on both the client and server side. Look at pusher.com for a free/cheap solution that will just work and save you a whole lot of hassle.
The initial request to start rendering should kick off the long-lived process, and return with an ID which is used to listen to the websocket for updates.
Once you have your websockets set up, you can send messages to the client about each finished frame. Personally I wouldn't try to push the whole frame down the websocket, but rather just send a message saying the frame is done with a URL to get the frame. Then normal HTTP with its caching and browser niceties moves the big data.
You're definitely not choosing the easy path. The easy path is to have your long-lived render task update the render state in the database, and have the client poll that. Extra server load, but a lot simpler.
Django itself really is focused on doing one kind of web interface, which is following the HTTP Request/Response pattern. To maintain a persistent connection with clients, which socket.io really makes dead simple, you need to diverge a bit from a normal Django installation.
This article discusses the issue of doing real-time with Django, with the help of Orbited and Twisted. It's rather old, and it relies on Comet, which is not the preferred way of doing real-time these days.
You might benefit a lot by going for Socket.io on the client, and something like Tornado (wiki) + Tornado client for Socket.io. But, if you really want to stick with Django for the web development (which Tornado also provide), you would need to make the two work together internally, each handling their particular use case.
Finally, this other article discusses how to make Django work with gevent (an coroutine-based networking library for Python) and Socket.io, which might well be your best option otherwise.
Don't hesitate to post questions/comments as they pop up!
What is a good way to implement server to client callbacks (events) with a web service?
We already have a SOAP-based API exposed by mod_perl running on Apache, but it is one way client to server. We need to notify the client of certain changes on the server by other clients.
What is a good way to make this work? Available modules for perl?
One thing we thought of was to make a long running web request that would finally respond when an event occurred, but this would seem to keep too many mod_perl processes running if many clients were connected.
One way is the one you mentioned, long-running polls.
The other is websockets. There are many Perl frameworks which are able to deal with websockets requests, one of which is Mojolicious.
See the docs at Mojolicious::Lite#WebSockets for some examples.
I would suggest in your case running the websockets webapp separately from the mod_perl server, and "just" using Apache to reverse-proxy the mojolicious app.
Have you actually tried your first approach ? How many max concurrent clients would that be? A few dozens should be manageable by apache. Just guessing.
Have you tried this low-tech method "Watching long processes through CGI", or is it what you wanted to get rid of in the first place?