I am trying to send data from a vector over a TCP socket.
I'm working with a vector that I fill with values from 0 to 4999, and then send it to the socket.
Client side, I'm receiving the data into a vector, then I copy its data to another vector until I received all the data from the server.
The issue I'm facing is that when I receive my data, sometimes I will get all of it, and sometimes I will only receive the correct data from 0 to 1625 and then I get trash data until the end (please see the image below). I even received for example from 0 to 2600 correct data, then from 2601 to 3500 it's trash and finally from 3501 to 4999 it's correct again.
(left column is line number and right column is the data).
This is the server side :
vector<double> values2;
for(int i=0; i<5000; i++)
values2.push_back(i);
skt.sendmsg(&values2[0], values2.size()*sizeof(double));
The function sendmsg :
void Socket::sendmsg(const void *buf, size_t len){
int bytes=-1;
bytes = send(m_csock, buf, len, MSG_CONFIRM);
cout << "Bytes sent: " << bytes << endl;
}
Client side :
vector<double> final;
vector<double> msgrcvd(4096);
do{
bytes += recv(sock, &msgrcvd[0], msgrcvd.size()*sizeof(double), 0);
cout << "Bytes received: " << bytes << endl;
//Get rid of the trailing zeros
while(!msgrcvd.empty() && msgrcvd[msgrcvd.size() - 1] == 0){
msgrcvd.pop_back();
}
//Insert buffer content into final vector
final.insert(final.end(), msgrcvd.begin(), msgrcvd.end());
}while(bytes < sizeof(double)*5000);
//Write the received data in a txt file
for(int i=0; i<final.size(); i++)
myfile << final[i] << endl;
myfile.close();
The outputs of the bytes are correct, the server outputs 40 000 when sending the data and the client also outputs 40 000 when receiving the data.
Removing the trailing zeros and then inserting the content of the buffer into a new vector is not very efficient, but I don't think it's the issue. If you have any clues on how to make it more efficient, it would be great!
I don't really know if the issue is when I send the data or when I receive it, and also I don't really get why sometimes (rarely), I get all the data.
recv receives bytes, and doesn't necessarily wait for all the data that was sent. So you can be receiving part of a double.
Your code works if you receive complete double values, but will fail when you receive part of a value. You should receive your data in a char buffer, then unpack it into doubles. (Possibly converting endianness if the server and client are different.)
#include <cstring> // For memcpy
std::array<char, 1024> msgbuf;
double d;
char data[sizeof(double)];
int carryover = 0;
do {
int b = recv(sock, &msgbuf[carryover], msgbuf.size() * sizeof(msgbuf[0]) - carryover, 0);
bytes += b;
b += carryover;
const char *mp = &msgbuf[0];
while (b >= sizeof(double)) {
char *bp = data;
for (int i = 0; i < sizeof(double); ++i) {
*bp++ = *mp++;
}
std::memcpy(&d, data, sizeof(double));
final.push_back(d);
b -= sizeof(double);
}
carryover = b % sizeof(double);
// Take care of the extra bytes. Copy them down to the start of the buffer
for (int j = 0; j < carryover; ++j) {
msgbuf[j] = *mp++;
}
} while (bytes < sizeof(double) * 5000);
This uses type punning from What's a proper way of type-punning a float to an int and vice-versa? to convert the received binary data to a double, and assumes the endianness of the client and server are the same.
Incidentally, how does the receiver know how many values it is receiving? You have a mix of hard coded values (5000) and dynamic values (.size()) in your server code.
Note: code not compiled or tested
TL/DR:
Never-ever send raw data via a network socket and expect them properly received/unpacked on other side.
Detailed answer:
Network is built on top of various protocols, and this is for a reason. Once you send something, there is no warranty you counterparty is on the same OS and same software version. There is no standard how primitive types should be coded on byte level. There is no restriction how much intermittent nodes could be involved into the data delivery, and each of your send() may traverse via different routes. So, you have to formalize the way you send the data, then other party can be sure what is proper way to retrieve them from the socket.
Simplest solution: use a header before your data. So, you plan to send 5000 doubles? Then send a DWORD first, which contains 40000 inside (5k elements, 8 bytes each -> 40k) and push all your 5k doubles right after that. Then, your counterparty should read 4 bytes from the socket first, interpret it as DWORD and understand how much bytes should come then.
Next step: you may want to send not only doubles, but ints and strings as well. That way, you have to expand your header so it can indicate
Total size of further data (so called payload size)
Kind of the data (array of doubles, string, single int etc)
Advanced solution:
Take a look on ready-to-go solutions:
ProtoBuf https://developers.google.com/protocol-buffers/docs/cpptutorial
Boost.Serialization https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc/index.html
Apache Thrift https://thrift.apache.org
YAS https://github.com/niXman/yas
Happy coding!
Related
I am writing a code to capture serial readings from the Arduino to C++
Is there a way to capture the readings line by line and then store it into an array? I have read another post similar to mine, but I am still unable to apply it.
Any help is greatly appreciated, thank you.
Environment setup:
Arduino UNO
ADXL 335 accelerometer
Ubuntu 16.04
C++
[Updated] applied solution from Bart
Cpp file
The reason why I added the "for-loop with print and break" is to analyze the array contents.
#include <stdio.h>
#include <string.h>
#include <iostream>
#include <unistd.h>
using namespace std;
char serialPortFilename[] = "/dev/ttyACM0";
int main()
{
char readBuffer[1024];
FILE *serPort = fopen(serialPortFilename, "r");
if (serPort == NULL)
{
printf("ERROR");
return 0;
}
while(1)
{
usleep(1000); //sync up Linux and Arduino
memset(readBuffer, 0, 1024);
fread(readBuffer, sizeof(char),1024,serPort);
for(int i=0; i<1024; i++){
printf("%c",readBuffer[i]);
}
break;
}
return 0;
}
Ino file
Fetching data from the Accelerometer
#include <stdio.h>
const int xPin = A0;
const int yPin = A1;
const int zPin = A2;
void setup() {
Serial.begin(9600);
}
void loop() {
int x = 0, y = 0, z = 0;
x = analogRead(xPin);
y = analogRead(yPin);
z = analogRead(zPin);
char buffer[16];
int n;
n = sprintf(buffer,"<%d,%d,%d>",x,y,z);
Serial.write(buffer);
}
Results
Running the code for three times
Click Here
The ideal outputs should be
<a,b,c><a,b,c><a,b,c>...
but right now, some of the outputs has the values inside "corrupted" (please see the fourth line from the top).
Even if use the start and end markers to determine a correct dataset, the data within the set is still wrong. I suspect the issue lies with the char array from C++, due to it being unsynchronized with Arduino. Else I need to send by Bytes from Arduino (not really sure how)
When dealing with two programs running on different processors they will never start sending/receiving at the same time. What you likely see is not that the results are merged wrong it is more likely the reading program started and stopped half way through the data.
When sending data over a line it is best that you:
On the Arduino:
First frame the data.
Send the frame.
On Linux:
Read in data in a buffer.
Search the buffer for a complete frame and deframe.
1. Framing the data
With framing the data I mean that you need a structure which you can recognize and validate on the receiving side. For example you could add the characters STX and ETX as control characters around your data. When the length of your data varies it is also required to send this.
In the following example we take that the data array is never longer than 255 bytes. This means that you can store the length in a single byte. Below you see pseudo code of how a frame could look like:
STX LENGTH DATA_ARRAY ETX
The total length of the bytes which will be send are thus the length of the data plus three.
2. Sending
Next you do not use println but Serial.write(buf, len) instead.
3. Receiving
On the receiving side you have a buffer in which all data received will be appended.
4. Deframing
Next each time new data has been added search for an STX character, assume the next character is the length. Using the length +1 you should find a ETX. If so you have found a valid frame and you can use the data. Next remove it from the buffer.
for(uint32_t i = 0; i < (buffer.size() - 2); ++i)
{
if(STX == buffer[i])
{
uint8_t length = buffer[i+2];
if(buffer.size() > (i + length + 3) && (ETX == buffer[i + length + 2]))
{
// Do something with the data.
// Clear the buffer from every thing before i + length + 3
buffer.clear(0, i + length + 3);
// Break the loop as by clearing the data the current index becomes invalid.
break;
}
}
}
For an example also using a Cyclic Redundancy Check (CRC) see here
I am using 64-bit Ubuntu 16.04 LTS. Like I said, I am attempting to make a TCP socket connection to another device. The program starts by reading data from the socket to initialize the last_recorded_data variable (as seen below, towards the bottom of myStartProcedure()), and I know that this is working exactly as expected. Then, the rest of the program starts which is driven by callbacks. When I make UPDATE_BUFFER_MS something smaller like 8, it fails after a couple of seconds. A frequency of this value is the desired value, but if I make it larger for testing purposes (something like 500), then it works for a little bit longer, but also eventually fails the same way.
The failure is as follows: The device I'm attempting to read from consistently sends data every 8 milliseconds, and within this packet of data, the first few bytes are reserved for telling the client how large the packet is, in bytes. During normal operation, the received number of bytes and the size as described by these first few bytes are equal. However, the packet received directly before the read() call starts to block is always 24 bytes less than the expected size, but the packet says the data packet sent should still be the expected size. When the next attempt to get the data is made, the read() call blocks and upon timeout sets errno to be EAGAIN (Resource temporarily unavailable).
I tried communicating with this same device with a Python application and it is not experiencing the same issue. Furthermore, I tried this C++ application on another one of these devices and I'm seeing the same behavior, so I think it's a problem on my end. My code (simplified) is below. Please let me know if you see any obvious errors, thank you!!
#include <string>
#include <unistd.h>
#include <iostream>
#include <stdio.h>
#include <errno.h>
#include <sys/socket.h>
#include <stdlib.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#define COMM_DOMAIN AF_INET
#define PORT 8008
#define TIMEOUT_SECS 3
#define TIMEOUT_USECS 0
#define UPDATE_BUFFER_MS 8
#define PACKET_SIZE_BYTES_MAX 1200
//
// Global variables
//
// Socket file descriptor
int socket_conn;
// Tracks the timestamp of the last time data was recorded
// The data packet from the TCP connection is sent every UPDATE_BUFFER_MS milliseconds
unsigned long last_process_cycle_timestamp;
// The most recently heard data, cast to a double
double last_recorded_data;
// The number of bytes expected from a full packet
int full_packet_size;
// The minimum number of bytes needed from the packet, as I don't need all of the data
int min_required_packet_size;
// Helper to cast the packet data to a double
union PacketAsFloat
{
unsigned char byte_values[8];
double decimal_value;
};
// Simple struct to package the data read from the socket
struct SimpleDataStruct
{
// Whether or not the struct was properly populated
bool valid;
// Some data that we're interested in right now
double important_data;
//
// Other, irrelevant members removed for simplicity
//
};
// Procedure to read the next data packet
SimpleDataStruct readCurrentData()
{
SimpleDataStruct data;
data.valid = false;
unsigned char socket_data_buffer[PACKET_SIZE_BYTES_MAX] = {0};
int read_status = read(socket_conn, socket_data_buffer, PACKET_SIZE_BYTES_MAX);
if (read_status < min_required_packet_size)
{
return data;
}
//for (int i = 0; i < read_status - 1; i++)
//{
// std::cout << static_cast<int>(socket_data_buffer[i]) << ", ";
//}
//std::cout << static_cast<int>(socket_data_buffer[read_status - 1]) << std::endl;
PacketAsFloat packet_union;
for (int j = 0; j < 8; j++)
{
packet_union.byte_values[7 - j] = socket_data_buffer[j + 252];
}
data.important_data = packet_union.decimal_value;
data.valid = true;
return data;
}
// This acts as the main entry point
void myStartProcedure(std::string host)
{
//
// Code to determine the value for full_packet_size and min_required_packet_size (because it can vary) was removed
// Simplified version is below
//
full_packet_size = some_known_value;
min_required_packet_size = some_other_known_value;
//
// Create socket connection
//
if ((socket_conn = socket(COMM_DOMAIN, SOCK_STREAM, 0)) < 0)
{
std::cout << "socket_conn heard a bad value..." << std::endl;
return;
}
struct sockaddr_in socket_server_address;
memset(&socket_server_address, '0', sizeof(socket_server_address));
socket_server_address.sin_family = COMM_DOMAIN;
socket_server_address.sin_port = htons(PORT);
// Create and set timeout
struct timeval timeout_chars;
timeout_chars.tv_sec = TIMEOUT_SECS;
timeout_chars.tv_usec = TIMEOUT_USECS;
setsockopt(socket_conn, SOL_SOCKET, SO_RCVTIMEO, (const char*)&timeout_chars, sizeof(timeout_chars));
if (inet_pton(COMM_DOMAIN, host.c_str(), &socket_server_address.sin_addr) <= 0)
{
std::cout << "Invalid address heard..." << std::endl;
return;
}
if (connect(socket_conn, (struct sockaddr *)&socket_server_address, sizeof(socket_server_address)) < 0)
{
std::cout << "Failed to make connection to " << host << ":" << PORT << std::endl;
return;
}
else
{
std::cout << "Successfully brought up socket connection..." << std::endl;
}
// Sleep for half a second to let the networking setup properly
sleepMilli(500); // A sleep function I defined elsewhere
SimpleDataStruct initial = readCurrentData();
if (initial.valid)
{
last_recorded_data = initial.important_data;
}
else
{
// Error handling
return -1;
}
//
// Start the rest of the program, which is driven by callbacks
//
}
void updateRequestCallback()
{
unsigned long now_ns = currentTime(); // A function I defined elsewhere that gets the current system time in nanoseconds
if (now_ns - last_process_cycle_timestamp >= 1000000 * UPDATE_BUFFER_MS)
{
SimpleDataStruct current_data = readCurrentData();
if (current_data.valid)
{
last_recorded_data = current_data.important_data;
last_process_cycle_timestamp = now_ns;
}
else
{
// Error handling
std::cout << "ERROR setting updated data, SimpleDataStruct was invalid." << std:endl;
return;
}
}
}
EDIT #1
I should be receiving a certain number of bytes every time, and I would expect the return value of read() to be returning that value as well. However, I just tried changing the value of PACKET_SIZE_BYTES_MAX to be 2048, and the return value of read() is now 2048, when it should be the size of the packet that the device is sending back (NOT 2048). The Python application is also setting the max to be 2048 and its returning packet size is the correct/expected size...
Try commenting out the timeout setup. I never use that on my end and I don't experience the problem you're talking about.
// Create and set timeout
struct timeval timeout_chars;
timeout_chars.tv_sec = TIMEOUT_SECS;
timeout_chars.tv_usec = TIMEOUT_USECS;
setsockopt(socket_conn, SOL_SOCKET, SO_RCVTIMEO, (const char*)&timeout_chars, sizeof(timeout_chars));
To avoid blocking, you can setup the socket as a non-block socket and then use a select() or poll() to get more data. Both of these functions can use the timeout as presented above. However, with a non-blocking socket you must make sure that the read works as expected. In many cases you will get a partial read and have to wait (select() or poll()) again for more data. So the code would be a bit more complicated.
socket_conn = socket(COMM_DOMAIN, SOCK_STREAM | SOCK_NONBLOCK, 0);
If security is a potential issue, I would also set SOCK_CLOEXEC to prevent a child process from accessing the same socket.
std::vector<struct pollfd> fds;
struct pollfd fd;
fd.fd = socket_conn;
fd.events = POLLIN | POLLPRI | POLLRDHUP; // also POLLOUT for writing
fd.revents = 0; // probably useless... (kernel should clear those)
fds.push_back(fd);
int64_t timeout_chars = TIMEOUT_SECS * 1000 + TIMEOUT_USECS / 1000;
int const r = poll(&fds[0], fds.size(), timeout_chars);
if(r < 0) { ...handle error(s)... }
Another method, assuming the header size is well defined and never changes, is to read the header, then using the header information to read the rest of the data. In that case you can keep the blocking socket without any timeout. From your structures I have no idea what that could be. So... let's first define such a structure:
struct header
{
char sync[4]; // four bytes indicated a synchronization point
uint32_t size; // size of packet
... // some other info
};
I put a "sync" field. In TCP it is often that people will add such a field so if you lose track of your position you can seek to the next sync by reading one byte at a time. Frankly, with TCP, you should never get a transmission error like that. You may lose the connection, but never lose data from the stream (i.e. TCP is like a perfect FIFO over your network.) That being said, if you are working on a mission critical software, a sync and also a checksum would be very welcome.
Next we read() just the header. Now we know of the exact size of this packet, so we can use that specific size and read exactly that many bytes in our packet buffer:
struct header hdr;
read(socket_conn, &hdr, sizeof(hdr));
read(socket_conn, packet, hdr.size /* - sizeof(hdr) */);
Obviously, read() may return an error and the size in the header may be defined in big endian (so you need to swap the bytes on x86 processors). But that should get you going.
Also, if the size found in the header includes the number of bytes in the header, make sure to subtract that amount when reading the rest of the packet.
Also, the following is wrong:
memset(&socket_server_address, '0', sizeof(socket_server_address));
You meant to clear the structure with zeroes, not character zero. Although if it connects that means it probably doesn't matter much. Just use 0 instead of '0'.
As a test, I'm writing a series of byte arrays to a tcp socket from an Android application, and reading them in a C++ application.
Java
InetAddress address = InetAddress.getByName("192.168.0.2");
Socket socket = new Socket(address, 1300);
DataOutputStream out = new DataOutputStream(socket.getOutputStream())
...
if(count == 0) {
out.write(first, 0, first.length);
} else if(count == 1) {
out.write(second, 0, second.length);
}
C++
do {
iResult = recv(ClientSocket, recvbuf, 3, 0);
for (int i = 0; i < 3; i++) {
std::cout << (int)(signed char)recvbuf[i] << std::endl;
}
} while (iResult > 0);
As it stands, on the first receipt, recv[2] = -52, which I assume to be a junk value, as the output stream has not yet written the second byte array by the time I've received the first segment.
However, when I pause after the the ListenSocket has accepted the connection:
ClientSocket = accept(ListenSocket, NULL, NULL);
std::cin.ignore();
...giving the sender time to do both writes to the stream, recv[2] = 3, which is the first value of the second written byte array.
If I ultimately want to send and receive a constant stream of discrete arrays, how can I determine after I've received the last value of one array, whether the next value in the buffer is the first value of the next array or whether it's a junk value?
I've considered that udp is more suitable for sending a series of discrete data sets, but I need the reliability of tcp. I imagine that tcp is used in this way regularly, but it's not clear to me how to mitigate this issue.
EDIT:
In the actual application for which I'm writing this test, I do implement length prefixing. I don't think that's relevant though; even if I know I'm at the end of a data set, I need to know whether the next value on the buffer is junk or the beginning of the next set.
for (int i = 0; i < 3; i++)
The problem is here. It should be:
for (int i = 0; i < iResult; i++)
You're printing out data that you may not have received. This is the explanation of the 'junk value'.
You can't assume that recv() fills the buffer.
You must also check iResult for both -1 and zero before this loop, and take the appropriate actions, which are different in each case.
As you point out, TCP is stream-based, so there's no built-in way to say "here's a specific chunk of data". What you want to do is add your own "message framing". A simple way to do that is called "length prefixing". Where you first send the size of the data packet, and then the packet itself. Then the receiver will know when they've gotten all the data.
Sending side
send length of packet (as a known size -- say a 32-bit int)
send packet data
Receiving side
read length of packet
read that many bytes of data
process fully-received packet
Check out this article for more information: http://blog.stephencleary.com/2009/04/message-framing.html
I'm trying to establish a SerialPort connection which transfers 16 bit data packages at a rate of 10-20 kHz. Im programming this in C++/CLI. The sender just enters an infinte while-loop after recieving the letter "s" and constantly sends 2 bytes with the data.
A Problem with the sending side is very unlikely, since a more simple approach works perfectly but too slow (in this approach, the reciever sends always an "a" first, and then gets 1 package consisting of 2 bytes. It leads to a speed of around 500Hz).
Here is the important part of this working but slow approach:
public: SerialPort^ port;
in main:
Parity p = (Parity)Enum::Parse(Parity::typeid, "None");
StopBits s = (StopBits)Enum::Parse(StopBits::typeid, "1");
port = gcnew SerialPort("COM16",384000,p,8,s);
port->Open();
and then doing as often as wanted:
port->Write("a");
int i = port->ReadByte();
int j = port->ReadByte();
This is now the actual approach im working with:
static int values[1000000];
static int counter = 0;
void reader(void)
{
SerialPort^ port;
Parity p = (Parity)Enum::Parse(Parity::typeid, "None");
StopBits s = (StopBits)Enum::Parse(StopBits::typeid, "1");
port = gcnew SerialPort("COM16",384000,p,8,s);
port->Open();
unsigned int i = 0;
unsigned int j = 0;
port->Write("s"); //with this command, the sender starts to send constantly
while(true)
{
i = port->ReadByte();
j = port->ReadByte();
values[counter] = j + (i*256);
counter++;
}
}
in main:
Thread^ readThread = gcnew Thread(gcnew ThreadStart(reader));
readThread->Start();
The counter increases (much more) rapidly at a rate of 18472 packages/s, but the values are somehow wrong.
Here is an example:
The value should look like this, with the last 4 bits changing randomly (its a signal of an analogue-digital converter):
111111001100111
Here are some values of the threaded solution given in the code:
1110011001100111
1110011000100111
1110011000100111
1110011000100111
So it looks like the connection reads the data in the middle of the package (to be exact: 3 bits too late). What can i do? I want to avoid a solution where this error is fixed later in the code while reading the packages like this, because I don't know if the the shifting error gets worse when I edit the reading code later, which I will do most likely.
Thanks in advance,
Nikolas
PS: If this helps, here is the code of the sender-side (an AtMega168), written in C.
uint8_t activate = 0;
void uart_puti16(uint16_t val) //function that writes the data to serial port
{
while ( !( UCSR0A & (1<<UDRE0)) ) //wait until serial port is ready
nop(); // wait 1 cycle
UDR0 = val >> 8; //write first byte to sending register
while ( !( UCSR0A & (1<<UDRE0)) ) //wait until serial port is ready
nop(); // wait 1 cycle
UDR0 = val & 0xFF; //write second byte to sending register
}
in main:
while(1)
{
if(active == 1)
{
uart_puti16(read()); //read is the function that gives a 16bit data set
}
}
ISR(USART_RX_vect) //interrupt-handler for a recieved byte
{
if(UDR0 == 'a') //if only 1 single data package is requested
{
uart_puti16(read());
}
if(UDR0 == 's') //for activating constant sending
{
active = 1;
}
if(UDR0 == 'e') //for deactivating constant sending
{
active = 0;
}
}
At the given bit rate of 384,000 you should get 38,400 bytes of data (8 bits of real data plus 2 framing bits) per second, or 19,200 two-byte values per second.
How fast is counter increasing in both instances? I would expect any modern computer to keep up with that rate whether using events or directly polling.
You do not show your simpler approach which is stated to work. I suggest you post that.
Also, set a breakpoint at the line
values[counter] = j + (i*256);
There, inspect i and j. Share the values you see for those variables on the very first iteration through the loop.
This is a guess based entirely on reading the code at http://msdn.microsoft.com/en-us/library/system.io.ports.serialport.datareceived.aspx#Y228. With this caveat out of the way, here's my guess:
Your event handler is being called when data is available to read -- but you are only consuming two bytes of the available data. Your event handler may only be called every 1024 bytes. Or something similar. You might need to consume all the available data in the event handler for your program to continue as expected.
Try to re-write your handler to include a loop that reads until there is no more data available to consume.
I have a byte array like this:
lzo_bytep out; // my byte array
size_t uncompressedImageSize = 921600;
out = (lzo_bytep) malloc((uncompressedImageSize +
uncompressedImageSize / 16 + 64 + 3));
wrkmem = (lzo_voidp) malloc(LZO1X_1_MEM_COMPRESS);
// Now the byte array has 802270 bytes
r = lzo1x_1_compress(imageData, uncompressedImageSize,
out, &out_len, wrkmem);
How can I split it into smaller parts under 65,535 bytes (the byte array is one large image which I want to sent over UDP which has upper limit 65,535 bytes) and then join those small chunks back into a continuous array?
The problem with doing this is that the UDP packets can arrive out or order, or be dropped. Use TCP for this; that's what it's for.
You don't have to "split" the array. You just have to point into different parts of it.
Assuming you're using a typical UDP write() function, it takes several arguments. One of them is a pointer to the buffer and the other is the length.
If you want to get the first 65535 bytes, your buffer is at wrkmem and the length is 65535.
For the second 65535 bytes, your buffer is at wrkmem + 65535 and your length is 65535.
The third 65535 bytes, your buffer is at wrkmem + 2 * 65535 and your length is 65535.
Get it?
(That said, the other posters are correct. You should be using TCP).
On the other side, when you want to re-join the array, you must allocate enough memory for the whole thing, then use a copy function like memcpy() to copy the arriving chunks into their correct position. Remember that UDP may not deliver the pieces in order and may not deliver all of them.
You might wish to try a message based middleware like ØMQ and feed the entire compressed image as one message and have the middleware run asynchronously and manage redelivery at the fastest speed possible. It provides a BSD socket compatible API and so can be easy to migrate code over and allows you to easily swap between various underlying transport protocols as required.
Other message systems are available.
void my_free (void *data, void *hint)
{
free (data);
}
/* ... */
size_t uncompressedImageSize = 921600, compressedImageSize = 0;
size_t out_len = (uncompressedImageSize + uncompressedImageSize / 16 + 64 + 3);
lzo_bytep out = (lzo_bytep)malloc (out_len);
lzo_voidp wkrmem = (lzo_voidp)malloc (LZO1X_1_MEM_COMPRESS);
zmq_msg_t msg;
rc = lzo1x_1_compress (imageData, uncompressedImageSize,
out, &compressedImageSize, wrkmem);
assert (compressedImageSize > 0);
rc = zmq_msg_init_data (&msg, out, compressedImageSize, my_free, NULL);
assert (rc == 0);
/* Send the message to the socket */
rc = zmq_send (socket, &msg, 0);
assert (rc == 0);