I am new to Apache Kafka. Kafka documentation (https://kafka.apache.org/documentation.html#introduction) mentions that it can be used with C++. But i am not sure how to do that. My application continuously generates image files and need to be transferred to another machine. I felt i could use stream api of Kafka but not sure how to stream image files.
Yes, you can use Apache Kafka with C/C++.
By far the most popular C/C++ client is https://github.com/edenhill/librdkafka. You can use the client to read data from Kafka and write data back to Kafka.
Further documentation on librdkafka is available at http://docs.confluent.io/current/clients/index.html (the author of librdkafka, Magnus Edenill, works at Confluent).
Related
I'am currently evaluating Apache Kafka for the use as a middleware in a microservice environment. Not only as a Message Queue but also for aggregating other data sources. Kafka seems to be the perfect fit. The services in majority are based on the Play Framework, so Akka Stream Kafka seems to be the natural choice to interact with Kafka.
I prototyped a small App with a Consumer and a Publisher, communicating via JSON and that was pretty straight forward. But when it comes to unit testing I become a little helpless. Is it possible to run the tests in a lightweight fashion and not with a running Kafka Cluster or an embedded server (check here)? I also found this project which looked promising, but I was not able to test my Consumer with it. Isn't that the right tool? I'am a little confused.
Not sure if your question is still relevant, but have you had a look at the Alpakka Kafka testkit?
I am new to AKKA (I am using java). I am not able to figure out how to transfer a zip file to remote actors. Should i use akka-http?
I tried going through akka-streams but i could not understand how to transfer a binary file to remote actors.
Indeed using akka-http could be a nice way to decouple the various systems from each other.
akka-streams is mainly used to stream data within a system, and connects to 'external' systems with sources and sinks. This of course works best when those sources and sinks also support streaming and backpressure. The akka-http client and server indeed support streaming and backpressure, so they could be a good candidate ;).
I am trying to stream a data through web service and planning to consume it into kafka. Streaming data would be of size 4 MB, at max it can goes upto 10 MB. Data source SDK is written onto .Net and Apache Kafka does not provide DLL for its consumer and producer. Its very typical to write Kafka producer and consumer in .Net and we can't use github Kafka producer.
My Questions are -
Is web service is good option for real time streaming?
Is web service able to stream upto 10MB of data without impacting the performance of web server and data ingestion?
Is there any better approach to solve this issue?
answer with authentic source will really helps me.
Thanks...
I think, the used of webservices is a good option to push stream data if you would like to decouple your front-ends either from kafka or an API written into a specific language. But that depends of your use-case.
You should have a look at the open-source Kafka REST Proxy provided by the Confluent Distribution (http://docs.confluent.io/2.0.0/kafka-rest/docs/index.html).
It will allow you to produce/consumer messsage though webservice.
If you expect to push messages with a max size of 10MB don't forget to increase the following kafka properties (as by default kafka is tuned for 1MB messages).
max.message.bytes
replica.fetch.max.bytes
fetch.message.max.bytes (consumer config)
Log files of my application keep accumulating on a server.I want to dump them into HDFS through KAFKA.I want the Kafka producer to read the log files,send them to Kafka broker and then move those files to another folder.Can the Kafka producer read log files ? Also, is it possible to have the copying logic in Kafka producer ?
Kafka maintains feeds of messages in categories called topics.
We'll call processes that publish messages to a Kafka topic producers.
We'll call processes that subscribe to topics and process the feed of published messages consumers..
Kafka is run as a cluster comprised of one or more servers each of which is called a broker.
So, at a high level, producers send messages over the network to the Kafka cluster which in turn serves them up to consumers like this:
So this is not a suitable for your application where you want to injest log files. Instead you can try flume.
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
As you know, Apache Kafka is publish-subscribe messaging system. you can send message from your application. To send message from your application you can use kafka clients or kafka rest api.
In short, you can read your log with your application and can send these logs to kafka topics.
To handle these logs, you can use apache storm. You can find many integrated solution for these purposes. And by using storm you can
add any logic your stream processing.
You can read many useful detailed information about storm kafka integration.
Also to put your processed logs to hdfs, you can easily integrate your storm with hadoop. You can check this repo for it.
Kafka was developed to support high volume event streams such as real-time log aggregation. From the kafka documentation
Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption
Also I got this little piece of information from this nice article which almost similar to your use-case
Today, Kafka has been used in production at LinkedIn for a number of projects. There are both offline and online usage. In the offline case, we use Kafka to feed all activity events to our data warehouse and Hadoop, from which we then run various batch analysis
I am implementing a client/server application where video streaming occurs between two computers (in one direction). I would like to have the server publish an SDP file when it starts streaming. The client would then be able to download this SDP file and use it to get the stream. In order to implement this it seems I need to include a RTSP server in my server application.
I am planning to use either libVLC or GStreamer for the client. Both are able to get incoming video streams using the info from an SDP file.
Server-side I don't really know where to start. Can anyone recommend a good C++ library that would allow me to create a small RTSP server?
Use Live555 LGPL library or for fun, read the RFC and implement :-)
Libcurl's library offers a simple example that can be usefull for the server side..
Take a look at: https://curl.haxx.se/libcurl/c/rtsp.html