I am new to AKKA (I am using java). I am not able to figure out how to transfer a zip file to remote actors. Should i use akka-http?
I tried going through akka-streams but i could not understand how to transfer a binary file to remote actors.
Indeed using akka-http could be a nice way to decouple the various systems from each other.
akka-streams is mainly used to stream data within a system, and connects to 'external' systems with sources and sinks. This of course works best when those sources and sinks also support streaming and backpressure. The akka-http client and server indeed support streaming and backpressure, so they could be a good candidate ;).
Related
I want to build a live streaming app.
My thought process:
Get the Video/Audio data from the
navigator.mediaDevices.getUserMedia(constraints); [client-streamer]
create rooms using sockets(Socket.IO or WebSockets from flask) [backend]
Send the data in 1 to the room members using sockets.
display the media on the client-side.
Is that correct? How should I do it?
how do I broadcast data to specific room members and not to everyone? (flask)
How to consistently send data from the streamer -> server -> room members. the stream is given from 1 is an object, where is the data?
any other better ideas will be great! thanks.
I need to implement the server-side by myself without help from libraries that will do the work for me.
Implementing a streaming platform is not trivial. Unfortunately, it is not as simple as emitting chunks received from the MediaRecorder with onndatavailable and forwarding them to users using a WebSocket server - this is not scalable nor efficient nor reliable.
Below are some strategies you can try for different types of scenarios:
P2P: If you want to have simple peer-to-peer streaming, you can use WebRTC to achieve that with a simple socket.io server for signaling purposes.
Conference: Here things start to get more complicated. You will need a media server if you want to be somewhat scalable. One approach is to route your stream to the users using an SFU or MCU. This will take care of forwarding/processing media to different peers efficiently.
Broadcast: Here things are also non-trivial. Common WebRTC-based architectures include ingesting the WebRTC stream and forward that to an HLS server which will let your stream chunks available for clients through a CDN, or perform RTP forwarding of the WebRTC stream, convert it to RTMP using something like FFmpeg and deliver it through Youtube Live or Twitch to leverage from their infrastructure.
Be aware that the last 2 items are resource-intensive and will certainly not be cheap to maintain.
Below are some open source projects that could help you along the way:
Janus
MediaSoup
AntMedia
Jitsi
Good luck!
Explaining all this is far beyond the scope of a Stack Overflow answer.
Here are a few hints:
You need to use the MediaRecorder API to capture compressed data from your gUM (getUserMedia) stream. MediaRecorder support is inconsistent between makes and models of browser. though.
It kicks a Blob into its onndatavailable handler every so often.
They're compressed as a webm data stream.
You can push those Blobs to a server with socket.io, and the server can turn around and push them to whatever clients you want to.
Playing the webm on the clients is tricky. You may, on some makes and models of browsers, be able to feed the webm stream to the Media Source API using appendBuffer(). But some browsers cannot consume the webm streams.
These webm streams are useless to a player without all their Blob data in order. You can't just start sending a new client the Blobs of the stream when they sign in; you have to restart the MediaRecorder.
(You may be able to make it work without a MediaRecorder restart if you send the first few k bytes of the stream to each new client before sending the current Blob. Extracting those bytes is an intricate programming job involving the ebml package to parse the webm stream and extract the prologue. I have not proven this concept.)
Because getting all this to work -- originator -- server -- viewer is such a pain in the xxx neck, you may want to investigate using something like mediasoup instead. It uses WebRTC transport rather than socket.io, and works cross-platform.
I am new to Apache Kafka. Kafka documentation (https://kafka.apache.org/documentation.html#introduction) mentions that it can be used with C++. But i am not sure how to do that. My application continuously generates image files and need to be transferred to another machine. I felt i could use stream api of Kafka but not sure how to stream image files.
Yes, you can use Apache Kafka with C/C++.
By far the most popular C/C++ client is https://github.com/edenhill/librdkafka. You can use the client to read data from Kafka and write data back to Kafka.
Further documentation on librdkafka is available at http://docs.confluent.io/current/clients/index.html (the author of librdkafka, Magnus Edenill, works at Confluent).
I have some framework doing specific task in C++ and a django-based web app. The idea is to launch this framework, receive some data from it, send some data or request and check it's status in some period.
I'm looking for the best way of communication. Both apps run on the same server. I was wondering if a json server in C++ is a good idea. Django would send a request to this server, and server would parse it, and delegate a worker thread to complete the task. Almost all data that need to be send is string-like. Other data will be stored in database so there is no problem with that.
Is JSON a good idea? Maybe you know some better mechanism for local communication between C++ and django?
If your requirement is guaranteed to always have the C++ application on the same machine as the Django web application, include the C++ code by converting it into a shared library and wrapping python around it. Just like this Calling C/C++ from python?
JSON and other serializations make sense if you are going to do remote calls and the code needs to communicate across machines.
JSON seems like a fair enough choice for data serialization - it's good at handling strings and there's existing libraries for encoding/decoding JSON in both python & C++.
However, I think your bigger problem is likely to be the transport protocol that you use for transferring JSON between your client and server. Here's some options:
You could build an HTTP server into your C++ application (which I think might be what you mean by "JSON server" in your question), which would work fine, though might be a bit of a pain to implement unless you get a hold of a library to handle the hard work for you.
Another option might be to use the 0MQ library to send JSON (or otherwise) messages between your client and server. I think this would probably be a lot easier than implementing a full HTTP server, and 0MQ has some interprocess communication code that would likely be a lot faster than sending things over the network.
A third option would just be to run your C++ as a standalone application and pass the data in to it via stdin or command line parameters. This is probably the simplest way to do things, though may not be the most flexble. If you were to go this way, you might be better off just building a Python/C++ binding as suggested by ablm.
Alternatively you could attempt to build some sort of job queue based on redis or something other database system. The idea being that your django application puts some JSON describing the job into the job queue, and then the C++ application could periodically poll the queue, and use a seperate redis entry to pass the results back to the client. This could have the advantage that you could reasonably easily have several "workers" reading from the job queue with minimal effort.
There's almost definitely some other ways to go about it, but those are the ones that immediately spring to mind.
I have a program that performs compute-heavy local processing. The compute packages have 16MiB input buffers and generate between 10KB and 16MiB of output. This all happens in a multi-threaded environment.
I have a number of users who are compute-bound. I'm interested in adding support for peer-to-peer multiprocessing. That is, clients would be run on multiple computers in the organization. The master would find available clients, send them 16MiB buffers, and then get the results.
The master will have a list of available clients. It will then send a message to each client that it is free. The clients will start a thread for each core that they have. Each thread will ask the master for a work buffer, will compute the results, and will then send the results to the master. The master will incorporate the results from each client into the final output.
A typical run involves crunching between 100 and 50,000 16MiB buffers. The output is typically in the 100MB to 10GB range. There are no data dependencies between the buffers.
I'm looking for the best way to arrange the communications between the client and the server. My plan is to use some kind of RPC. I do not want to embed a web server in either the client or the server. Instead, my plan is to simply receive connections on a TCP socket and have some kind of basic RPC.
Here are the options I'm considering:
I can role my own client/server system and use protocol buffers to put the whole thing together.
I could use one of the existing RPC systems for potocol buffers.
I could embed a web server and use XMLRPC.
The implementation language is C++. The program compiles for Linux, MacOS, and with mingw for Windows.
Thanks.
We found that 0MQ met our criteria.
I would like to create a connection between two applications. Should I be using Client-Server or is there another way of efficiently communicating between one another? Is there any premade C++ networking client server libraries which are easy to use/reuse and implement?
Application #1 <---> (Client) <---> (Server) <---> Application #2
Thanks!
Client / server is a generic architecture pattern (much like factory, delegation, inheritance, bridge are design patterns). What you probably want is a library to eliminate the tedium of packing and unpacking your data in a format that can be sent over the wire. I strongly recommend you take a look at the protocol buffers library, which is used extensively at Google and released as open source. It will automatically encode / decode data, and it makes it possible for programs written in different languages to send and receive messages of the same type with all the dirty work done for you automatically. Protobuf only deals with encoding, not actually sending and receiving. For that, you can use primitive sockets (strongly recommend against that) or the Boost.Asio asynchronous I/O library.
I should add that you seem to be confused about the meaning of client and server, since in your diagram you have the application talking to a client which talks to a server which talks to another application. This is wrong. Your application is the client (or the server). Client / server is simply a role that your application takes on during the communication. An application is considered to be a client when it initiates a connection or a request, while an application is considered to be a server when it waits for and processes incoming requests. Client / server are simply terms to describe application behavior.
If you know the applications will be running on the same machine, you can use sockets, message queues, pipes, or shared memory. Which option you choose depends on a lot of factors.
There is a ton of example code for any of these strategies as well as libraries that will abstract away a lot of the details.
If they are running on different machines, you will want to communicate through sockets.
There's a tutorial here, with decent code samples.