Joining fragments of messages together - c++

I need to write a function that can receive fragments of different messages, and then piece them together. The fragments are in the form of a class msg, which holds information of
int message_id
int no_of_fragments
int fragment_id
string msg_fragment
The function needs to do the following
Check received message - if no_of_fragments == 1 then the message has not been fragmented and function can stop here
If no_of_fragments > 1 then message is fragmented
get message_id and fragment_id
collect all fragments e.g. for message_id=111 with no_of_fragments=6, the system should ensure that fragments_id 1-6 have been collected
piece fragments together
What is the best way for doing this? I thought a map might be useful (with the message_id serving as key, pointing to a container that would hold the fragments) but would appreciate any suggestions.
Thank you!

I would use a map of vectors. Each time you receive a new message ID, use that as your map key. Then allocate a vector to hold the fragments based on the number of fragments specified in the first fragment received (doesn't have to be in order). You'll also need to hold the count, so it's easy to know when you've received the last fragment, so probably a map of message_id to a struct of count and the vector of fragments.
My c++ is rusty:
struct message_parts {
int fragments_expected; // init to no_of_fragments
int fragments_received; // init to 0 (you'll bump it as soon as you add the fragment to the vector)
vector<fragment *> fragments; <-- initialize size to no_of_fragments
}
std::map<int, message_parts> partial_messages
When you insert a fragment, put it directly into the location in the fragments vector based on the fragment_id - 1 (since they are zero-indexed). This way you'll always have them in the right order, no matter the order they come in.
After you add a fragment, check to see if fragments_received == fragments expected, and then you can piece it together and deal with the data.
This gives constant time first-fragment detection and allocation, constant time fragment insertion, constant time complete-message-received detection, and linear time message reconstruction (can't do any better than this).
This solution requires no special casing for non-fragmented data.
Don't forget to delete the fragments once you've reassembled them into the complete message.

Related

Correct use of memcpy

I have some problems with a project I'm doing. Basically I'm just using memcpy the wrong way. I know the theroy of pointer/arrays/references and should know how to do that, nevertheless I've spend two days now without any progress. I'll try to give a short code overview and maybe someone sees a fault! I would be very thankful.
The Setup: I'm using an ATSAM3x Microcontroller together with a uC for signal aquisition. I receive the data over SPI.
I have an Interrupt receiving the data whenever the uC has data available. The data is then stored in a buffer (int32_t buffer[1024 or 2048]). There is a counter that counts from 0 to the buffer size-1 and determines the place where the data point is stored. Currently I receive a test signal that is internally generated by the uC
//ch1: receive 24 bit data in 8 bit chunks -> store in an int32_t
ch1=ch1|(SPI.transfer(PIN_CS, 0x00, SPI_CONTINUE)<<24)>>8;
ch1=ch1|(SPI.transfer(PIN_CS, 0x00, SPI_CONTINUE)<<16)>>8;
ch1=ch1|(SPI.transfer(PIN_CS, 0x00, SPI_CONTINUE)<<8)>>8;
if(Not Important){
_ch1Buffer[_ch1SampleCount] = ch1;
_ch1SampleCount++;
if(_ch1SampleCount>SAMPLE_BUFFER_SIZE-1) _ch1SampleCount=0;
}
This ISR is active all the time. Since I need raw data for signal processing and the buffer is changed by the ISR whenever a new data point is available, i want to copy parts of the buffer into a temporary "storage".
To do so, I have another, global counter wich is incremented within the ISR. In the mainloop, whenever the counter reaches a certain size, i call a method get some of the buffer data (about 30 samples).
The method aquires the current position in the buffer:
'int ch1Pos = _ch1SampleCount;'
and then, depending on that position I try to use memcpy to get my samples. Depending on the position in the buffer, there has to be a "wrap-around" to get the full set of samples:
if(ch1Pos>=(RAW_BLOCK_SIZE-1)){
memcpy(&ch1[0],&_ch1Buffer[ch1Pos-(RAW_BLOCK_SIZE-1)] , RAW_BLOCK_SIZE*sizeof(int32_t));
}else{
memcpy(&ch1[RAW_BLOCK_SIZE-1 - ch1Pos],&_ch1Buffer[0],(ch1Pos)*sizeof(int32_t));
memcpy(&ch1[0],&_ch1Buffer[SAMPLE_BUFFER_SIZE-1-(RAW_BLOCK_SIZE- ch1Pos)],(RAW_BLOCK_SIZE-ch1Pos)*sizeof(int32_t));
}
_ch1Buffer is the buffer containing the raw data
SAMPLE_BUFFER_SIZE is the size of that buffer
ch1 is the array wich is supposed to hold the set of samples
RAW_BLOCK_SIZE is the size of that array
ch1Pos is the position of the last data point written to the buffer from the ISR at the time where this method is called
Technically I'm aware of the requirements, but apparently thats not enough ;-).
I know, that the data received by the SPI interface is "correct". The problem is, that this is not the case for the extracted samples. There are a lot of spikes in the data that indicate that I've been reading something I wasn't supposed to read. I've changed the memcpy commands that often, that I completly lost the overview. The code sample above is one version of many's, and while you're reading this I'm sure I've changed everything again.
I would appreciate every hint!
Thanks & Greetings!
EDIT
I've written down everything (again) on a sheet of paper and tested some constellations. This is the updated Code for the memcpy part:
if(ch1Pos>=(RAW_BLOCK_SIZE-1)){
memcpy(&ch1[0],&_ch1Buffer[ch1Pos-(RAW_BLOCK_SIZE-1)] , RAW_BLOCK_SIZE*sizeof(int32_t));
}else{
memcpy(&ch1[RAW_BLOCK_SIZE-1-ch1Pos],&_ch1Buffer[0],(ch1Pos+1)*sizeof(int32_t));
memcpy(&ch1[0],&_ch1Buffer[SAMPLE_BUFFER_SIZE-(RAW_BLOCK_SIZE-1-ch1Pos)],(RAW_BLOCK_SIZE-1-ch1Pos)*sizeof(int32_t));
}
}
This already made it a lot better. From all the changes, everything kinda got messed up. Now there is just one Error there. There is a periodical spike. I'll try to get more information, but I think it is a wrong access while wrapping around.
I've changed the if(_ch1SampleCount>SAMPLE_BUFFER_SIZE-1) _ch1SampleCount=0; to if(_ch1SampleCount>=SAMPLE_BUFFER_SIZE) _ch1SampleCount=0;.
EDIT II
To answer the Questions of #David Schwartz :
SPI.transfer returns a single byte
The buffer is initialised once at startup: memset(_ch1Buffer,0,sizeof(int32_t)*SAMPLE_BUFFER_SIZE);
EDIT III
Sorry for the frequent updates, the comment section is getting too big.
I managed to get rid of a bunch of zero values at the beginning of the stream by decreasing ch1Pos: 'int ch1Pos = _ch1SampleCount;' Now there is just one periodic "spike" (wrong value). It must be something with the splitted memcpy command. I'll continue looking. If anyone has an idea ... :-)

Optimising Network Bandwidth Use: Reusing strings

I'll try to be as brief and to the point as possible..
I currently develop a small, live "MMO" and would like to reduce the bandwidth use of host -> client.
The most costly of this is the use of strings, that, for the most part, are constantly repeated and sent to the clients. They differ only slightly, due to their nature, be it names or values, but the rest is repeated.
I have an idea that I'm sure has already been done, but can't really find any example of it.
Basically, these repeated strings, could be assigned a "stringcode" , then the variables, and then sent to the client. The clients already have enough information to fill in the blanks, so to speak, based on id numbers assigned to each player
So for example, in the case of names of a player..
"Supanova transfers 75 crew to Angus" // 35 bytes
becomes..
1 23 75 8 // only 6 bytes needed
where ..
1, is the "stringcode"
23, is the player ID of Supanova, (the client has the name already for this player
75, is the integer variable.
8, is the other player ID of Angus
In theory, I could find all of the strings sent to the client, and store them on the client, this is what I wish to avoid, as it's a painful process.
OR, create a dictionary of sorts, at runtime, assigning a "stringcode" value to each new string, the amount of variables, and quite possibly the order of the types would be needed. The host will send the packet contain just the "stringcode" and variables, if the client doesn't yet know the context of the string for that "stringcode", request the string, and then it can piece it together.
Not sure if I'm explaining this all too well, it's getting late and it sounds so much better in my head.....
You may be onto something with your "stringcode" idea. It's similar to what's done in some compression algorithms: build a dictionary to represent common strings with short IDs.
However, regarding your actual example:
"Supanova transfers 75 crew to Angus" // 35 bytes
Don't think of this as a "string," think of it as a "command" or "message." Then the message is a struct or class like this:
enum class MessageType : uint8_t { Transfer, /*...*/ };
struct BaseMessage { // message header
MessageType messageType;
} __attribute__((packed));
struct TransferMessage : BaseMessage {
PlayerId playerIdFrom;
PlayerId playerIdTo;
int32_t amount;
} __attribute__((packed));
If PlayerId is four bytes, this entire thing is still 13 bytes (16 if not packed). But PlayerId only needs to be large if it is an "absolute" ID in the entire world. You could also use a "relative" ID within the current game, in which case a single byte is enough for 256 players per game. In that case you can get away with 7 bytes total.
There's nothing novel in the above: it's very typical network message formatting. Be mindful of endianness. You might consider using a suitable library such as Cap'N'Proto.

How to determine length of buffer at client side

I have a server sending a multi-dimensional character array
char buff1[][3] = { {0xff,0xfd,0x18} , {0xff,0xfd,0x1e} , {0xff,0xfd,21} }
In this case the buff1 carries 3 messages (each having 3 characters). There could be multiple instances of buffers on server side with messages of variable length (Note : each message will always have 3 characters). viz
char buff2[][3] = { {0xff,0xfd,0x20},{0xff,0xfd,0x27}}
How should I store the size of these buffers on client side while compiling the code.
The server should send information about the length (and any other structure) of the message with the message as part of the message.
An easy way to do that is to send the number of bytes in the message first, then the bytes in the message. Often you also want to send the version of the protocol (so you can detect mismatches) and maybe even a message id header (so you can send more than one kind of message).
If blazing fast performance isn't the goal (and you are talking over a network interface, which tends to be slower than computers: parsing may be cheap enough that you don't care), using a higher level protocol or format is sometimes a good idea (json, xml, whatever). This also helps with debugging problems, because instead of debugging your custom protocol, you get to debug the higher level format.
Alternatively, you can send some sign that the sequence has terminated. If there is a value that is never a valid sequence element (such as 0,0,0), you could send that to say "no more data". Or you could send each element with a header saying if it is the last element, or the header could say that this element doesn't exist and the last element was the previous one.

avoiding collisions when collapsing infinity lock-free buffer to circular-buffer

I'm solving two feeds arbitrate problem of FAST protocol.
Please don't worry if you not familar with it, my question is pretty general actually. But i'm adding problem description for those who interested (you can skip it).
Data in all UDP Feeds are disseminated in two identical feeds (A and B) on two different multicast IPs. It is strongly recommended that client receive and process both feeds because of possible UDP packet loss. Processing two identical feeds allows one to statistically decrease the probability of packet loss.
It is not specified in what particular feed (A or B) the message appears for the first time. To arbitrate these feeds one should use the message sequence number found in Preamble or in tag 34-MsgSeqNum. Utilization of the Preamble allows one to determine message sequence number without decoding of FAST message.
Processing messages from feeds A and B should be performed using the following algorithm:
Listen feeds A and B
Process messages according to their sequence numbers.
Ignore a message if one with the same sequence number was already processed before.
If the gap in sequence number appears, this indicates packet loss in both feeds (A and B). Client should initiate one of the Recovery process. But first of all client should wait a reasonable time, perhaps the lost packet will come a bit later due to packet reordering. UDP protocol can’t guarantee the delivery of packets in a sequence.
// tcp recover algorithm further
I wrote such very simple class. It preallocates all required classes and then first thread that receive particular seqNum can process it. Another thread will drop it later:
class MsgQueue
{
public:
MsgQueue();
~MsgQueue(void);
bool Lock(uint32_t msgSeqNum);
Msg& Get(uint32_t msgSeqNum);
void Commit(uint32_t msgSeqNum);
private:
void Process();
static const int QUEUE_LENGTH = 1000000;
// 0 - available for use; 1 - processing; 2 - ready
std::atomic<uint16_t> status[QUEUE_LENGTH];
Msg updates[QUEUE_LENGTH];
};
Implementation:
MsgQueue::MsgQueue()
{
memset(status, 0, sizeof(status));
}
MsgQueue::~MsgQueue(void)
{
}
// For the same msgSeqNum should return true to only one thread
bool MsgQueue::Lock(uint32_t msgSeqNum)
{
uint16_t expected = 0;
return status[msgSeqNum].compare_exchange_strong(expected, 1);
}
void MsgQueue::Commit(uint32_t msgSeqNum)
{
status[msgSeqNum] = 2;
Process();
}
// this method probably should be combined with "Lock" but please ignore! :)
Msg& MsgQueue::Get(uint32_t msgSeqNum)
{
return updates[msgSeqNum];
}
void MsgQueue::Process()
{
// ready packets must be processed,
}
Usage:
if (!msgQueue.Lock(seq)) {
return;
}
Msg msg = msgQueue.Get(seq);
msg.Ticker = "HP"
msg.Bid = 100;
msg.Offer = 101;
msgQueue.Commit(seq);
This works fine if we assume that QUEUE_LENGTH is infinity. Because in this case one msgSeqNum = one updates array item.
But I have to make buffer circular because it is not possible to store entire history (many millions of packets) and there are no reason to do so. Actually I need to buffer enough packets to reconstruct the session, and once session is reconstructed i can drop them.
But having circular buffer significantly complicates algorithm. For example assume that we have circular buffer of length 1000. And at the same time we try to process seqNum = 10 000 and seqNum = 11 000 (this is VERY unlikely but still possible). Both these packets will map to the array updates at index 0 and so collision occur. In such case buffer should 'drop' old packets and process new packets.
It's trivial to implement what I want using locks but writing lock-free code on circular-buffer that used from different threads is really complicated. So I welcome any suggestions and advice how to do that. Thanks!
I don't believe you can use a ring buffer. A hashed index can be used in the status[] array. Ie, hash = seq % 1000. The issue is that the sequence number is dictated by the network and you have no control over it's ordering. You wish to lock based on this sequence number. Your array doesn't need to be infinite, just the range of the sequence number; but that is probably larger than practical.
I am not sure what is happening when the sequence number is locked. Does this mean another thread is processing it? If so, you must maintain a sub-list for hash collisions to resolve the particular sequence number.
You may also consider an array size as a power of 2. For example, 1024 will allow hash = seq & 1023; which should be quite efficient.

Reading packets off a fragmented byte stream with ranges?

I believe I sort of know ranges, but I have no real idea for when and where to use them, or how. I fail to "get" ranges. Consider this example:
Let's assume we have a network handler, that we have no control over, that calls our callback function in some thread whenever there's some new data for us:
void receivedData(ubyte[] data)
This stream of data contains packets with the layout
{
ushort size;
ubyte[size] body;
}
However, the network handler doesn't know about this, so a call to dataReceived() may contain one or two partial packets, one or several complete packets, or a combination. For simplicity, let's assume there can be no corrupt packets and that we'll receive a data.length == 0 when the connection is closed.
What we'd like now is some beautiful D code that turns all this chaos into appropriate calls to
void receivedPacket(ubyte[] body)
I can surely think of "brute-force" ways of accomplishing this. But here's perhaps where my confusion comes in: Can ranges play a role in this? Can we wrap up receivedData() into a nice range-thingy? How? Or is this just not the kind of problems where you'd use ranges? Why not?
(If it would make more sense for using ranges, feel free to redefine the example.)
what I'd do is
ubyte[1024] buffer=void;//temp buffer set the size as needed...
ushort filledPart;//length of the part of the buffer containing partial packet
union{ushort nextPacketLength=0;ubyte[2] packetLengtharr;}//length of the next packet
void receivedData(ubyte[] data){
if(!data.length)return;
if(nextPacketLength){
dataPart = min(nextPacketLength-filledPart.length,data.length);
buffer[filledPart..nextPacketLength] = data[0..dataPart];
filledPart += dataPart;
if(filledPart == nextPacketLength){
receivedPacket(buffer[0..nextPacketLength]);//the call
filledPart=nextPacketLength=0;//reset state
receivedData(datadataPart..$]);//recurse
}
} else{
packetLengtharr[]=data[0..2];//read length of next packet
if(nextPacketLength<data.length){//full paket in data -> avoid unnecessary copies
receivedPacket(data[2..2+nextPacketLength]);
receivedData(data[2+nextPacketLength..$]);//recurse
}else
receivedData(data[2..$]);//recurse to use the copy code above
}
}
it's recursive with 3 possible paths:
data is empty -> no action
nextPacketLength is set to a value != 0 -> copy as much data into the buffer as possible and if the packet is complete call the callback and reset filledPart and nextPacketLength and recurse with rest of data
nextPacketLength ==0 read the packet length (here with a union) if full packet available call callback then recurse with rest of data
there's only 1 bug still in there when data only hold the first byte of length but I'll let you figure that out (and I'm too lazy to do it now)