C# Publisher is publishing continuos marketdata messages in custom protobuff format over the socket using "writeDelimitedTo" API. I have to read all messages in C++ and desearialize it. Below is my code. Since C++ don't have "parseDelimitedFrom", so have coded something like below after going through multiple suggestions in this forum.
Now my question is - Refering to the code below, If the first message size is less than 1024 then in the first iteration, i will have full stream of the 1st message and part of the stream from the 2nd message. After deserializing first message, How can i read remaining streams of the second message from socket and merge it with the stream which i read in the previous iteration ?
EDIT: Support for "delimited" format is now part of the official protobuf library. The post below predates it being added.
I've written optimally-efficient versions of parseDelimitedFrom and writeDelimitedTo in C++ here (the read and write methods of Uncompressed):
https://github.com/capnproto/capnproto/blob/06a7136708955d91f8ddc1fa3d54e620eacba13e/c%2B%2B/src/benchmark/protobuf-common.h#L101
Feel free to copy.
These implementations read from / write to a ZeroCopyInputStream / ZeroCopyOutputStream.(Hmm, for some reason my write is declared to use FileOutputStream, but you should be able to just change that to ZeroCopyOutputStream.)
So, you'll need to create a ZeroCopyInputStream which reads from your StreamSocket, then pass it to my read().
It looks like StreamSocket is a classic copying-read interface. You should therefore use CopyingInputStreamAdaptor as your ZeroCopyInputStream, wrapping an implementation of CopyingInputStream which reads from your StreamSocket.
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.io.zero_copy_stream_impl_lite#CopyingInputStreamAdaptor
Related
So... I'm dealing with a system that has input data coming in buffers (i.e. NOT a file). I want to determine which decoder to create to decompress an audio stream (MP3, WAV, OGG, ...) So obviously I do not know the input format.
I found out that I could determine the format using the av_probe_input_format[23]() functions. That part works great, I get a format pointer that matches the files that I use as input.
AVInputFormat * format(av_probe_input_format3(&pd, true, &score));
I can print the format->name and format->long_name and these are the correct type (so the detection is working as expected).
Now, I'm trying to understand how to convert that AVInputFormat * into a AVCodec * so I can call avcodec_alloc_context3(codec) to create the actual audio decoder.
I found a couple of functions, which I used like so:
AVCodecID const codec_id(av_codec_get_id(format->codec_tag, format->raw_codec_id));
AVCodec * codec(avcodec_find_decoder(codec_id));
Problem 1. the raw_codec_id field is marked as "private" (should not access/use anywhere in your client's code).
Problem 2. the first function always returns AV_CODEC_ID_NONE (0) so of course the second call fails each time.
Am I doing something wrong? Is there is way to instead create a generic decode that will automatically detect the type of audio I have as input? (that is, would that be the only way to make that work?)
Okay, so the fact is that trying to use these functions directly is pretty much futile. The problem I have with the design is that it forces me to actually have a callback and that callback forces me to have a thread (i.e. I have to somehow feed data from a stream, not a file or such!)
So I can use the avformat_open_input() as mentioned by Gyan, only I have to have my own AVIOContext. I was hoping I could just call functions with my incoming data and avoid the pipeline concept. The issue here is some background processes could be servers that use fork() and thus you need to be really careful (i.e. fork() is not friendly with threads).
I'm stuck and I need help.
I'm trying to write the correct code for sending back an image file so the web browser can render it. It can send back text/html just fine, but image/* is not working.
You can see the code and the URL is shown below.
https://github.com/MagnusTiberius/iocphttpd/blob/master/iocphttpl/SocketCompletionPortServer.cpp
What the browser is receiving is just a few bytes of image data.
I tried vector, std::string and const char* to set the values of WSABUF, but still the same few bytes are sent over.
Please let know what is the missing piece to make this one work.
Thanks in advance.
Here's your problem:
PerIoData->LPBuffer = _strdup(str.c_str());
The _strdup function only copies up until the first null, so it cannot be used to copy binary data. Consider using malloc and memcpy if you don't want to use the C++ library.
The alternate implementation (in the false branch) is also incorrect, because it saves the data in an object (vc) that goes out of scope before the I/O is completed. You could instead do something like
vector<char> * vc = new vector<char>;
In my project I want to send possibly large and frequent XML data to a custom server written in c++. I don't want to use Apache and CGI because the data is too frequent to be starting a CGI process for every request. I would prefer if the data was recieved directly in the c++ code that will process the data and send a reply.
I started out by using libmicrohttpd for the c++ server but now I believe it won't be possible because it doesn't give access to the raw POST data. I started looking for another library but I can't seem to find a c++ library that does this. Can anyone suggest a c++ http server library that has access to the raw post data?
Here is the code I intended to start with. It is one of the example files provided in the source code of libmicrohttpd. Post Example from libmicrohttpd library
Edit:
A little more context.
From what I understand to access the post data in libmicrohttpd you create MHD_PostProcessor function that gets called incrementally as the post data is received in chunks. But in the example below it only shows how to get post data in the form of key value pairs. But I can't see how to get the data from a post.
The example implements the MHD_PostProcessor as post_iterator. See the definition of
static int post_iterator(void *cls,
enum MHD_ValueKind kind,
const char *key,
const char *filename,
const char *content_type,
const char *transfer_encoding,
const char *data, uint64_t off, size_t size) {
...
in the example posted above. You will see it only shows how to iterate the key value pairs.
MHD does give you access to the raw POST data, just grab it from "upload_data" directly instead of passing it to the MHD_PostProcessor. MHD will give you the uploaded POST stream incrementally by calling your main request processing callback repeatedly with more and more POST data being given to you raw, unprocessed in "upload_data".
I am currently evaluating Protocol Buffers for use in a project (no code written as of yet). One of the things I'm unclear on is how you would read part of an encoded message, for example say I have a common header:
message Header {
required uint16 msg_type = 1;
required uint16 length = 2;
}
And say I deliver multiple different messages to a queue. How would the consumer work out how much data to read per message and what message type is should be constructed as?
There should be no need for a Header message here; the most common approach is to follow the "streaming" advice from here. Within that, you could either treat it as a sequence of identical union type messages, or (my preference) when writing, instead of just writing a length-prefix before each, include a varint that indicates the message type then the length (as a varint). The number that indicates the message type is some arbitrary map you invent, so 1 = Foo, 2 = Bar, 3 = Blap, etc). If you left-shift the message-type by 3 bits then "or" 2, then it will also be a well-formed protobuf stream itself, 100% identical to a repeated YourUnionType.
Basically, this is exactly the same as this answer, but instead of being field 1 each time, the number varies per message-type. Most implementations have a reader/writer API that make it possible to read and write raw varints, and to length-restrict the reader API. Some implementations have helper mechanisms to support streams of heterogeneous messages directly (basically, doing all the above for you).
In a recent project, I used Protocol Buffers like this:
We had one 'container' message that included all the actual messages as optional members:
message ContainerMessage {
optional Message1 message_1 = 1;
optional Message2 message_2 = 2;
//...
optional MessageN message_N = N;
}
Inside an application, you could just use ContainerMessage as a discriminated union of the real Messages.
Between applications, we serialized/deserialized the ContainerMessage and sent the serialized content, prefixed with a simple header containing the length of the serialized content.
That will depend on the protocol you are using.
Note that e.g. a lot of protocols go via serial interfaces, where you might have extra lines telling when a message starts and stops.
Often, messages will have there length at a fixed offset after the message start.
In other cases, you might need to parse the message element by element to find out how much of the message is left. So a string embedded in the message may be of fixed length, or have the length at the beginning, or might have \0 as end marker.
Mostly, when you store messages in a queue for further processing, you will want to add some more information to make your life easier - like when you just have an extra signal telling you when the message stops, you might store the message internally with its length.
I am trying to send 1kb of data from a "server" to a "client", but I just can't get it right.
There are a few things that I NEED to do in this:
1) Need to use boost-asio sockets to transfer the data
2) Need to serialize a type I created (Packet) that will contain the data as a string or char*
Here is what is going on:
First, I get 1kb of data from a sample text file on the server. I get this and put it into the Packet type that I created. I have defined the data field in Packet to hold this data as a std::string. (I tried char* but it didnt work well - see next paragraph).
Second I serialize it using boost text_oarchive . I have no problems serializing the Packet type if it just contains a string, but what I really want is a way to serialize it with the data type being a char array (so that it works better with the socket below)
Third, I send it over a boost asio socket. Here I have a problem because I can't find a way to send a std::string over the socket connection. Everything I see as examples and in the documentation need a buffer using some type of char* and not a string.
its just a headache. can you help?
Everything I see as examples and in the documentation need a buffer
using some type of char* and not a string
That is correct, though it's quite simple to do using Boost.Serialization and Boost.Asio. You can serialize using a text_oarchive to a boost::asio::streambuf then send the resulting stream buffer contents using a socket.
See this question and my answer to that question for a more complete example.