I have a byte array like this:
lzo_bytep out; // my byte array
size_t uncompressedImageSize = 921600;
out = (lzo_bytep) malloc((uncompressedImageSize +
uncompressedImageSize / 16 + 64 + 3));
wrkmem = (lzo_voidp) malloc(LZO1X_1_MEM_COMPRESS);
// Now the byte array has 802270 bytes
r = lzo1x_1_compress(imageData, uncompressedImageSize,
out, &out_len, wrkmem);
How can I split it into smaller parts under 65,535 bytes (the byte array is one large image which I want to sent over UDP which has upper limit 65,535 bytes) and then join those small chunks back into a continuous array?
The problem with doing this is that the UDP packets can arrive out or order, or be dropped. Use TCP for this; that's what it's for.
You don't have to "split" the array. You just have to point into different parts of it.
Assuming you're using a typical UDP write() function, it takes several arguments. One of them is a pointer to the buffer and the other is the length.
If you want to get the first 65535 bytes, your buffer is at wrkmem and the length is 65535.
For the second 65535 bytes, your buffer is at wrkmem + 65535 and your length is 65535.
The third 65535 bytes, your buffer is at wrkmem + 2 * 65535 and your length is 65535.
Get it?
(That said, the other posters are correct. You should be using TCP).
On the other side, when you want to re-join the array, you must allocate enough memory for the whole thing, then use a copy function like memcpy() to copy the arriving chunks into their correct position. Remember that UDP may not deliver the pieces in order and may not deliver all of them.
You might wish to try a message based middleware like ØMQ and feed the entire compressed image as one message and have the middleware run asynchronously and manage redelivery at the fastest speed possible. It provides a BSD socket compatible API and so can be easy to migrate code over and allows you to easily swap between various underlying transport protocols as required.
Other message systems are available.
void my_free (void *data, void *hint)
{
free (data);
}
/* ... */
size_t uncompressedImageSize = 921600, compressedImageSize = 0;
size_t out_len = (uncompressedImageSize + uncompressedImageSize / 16 + 64 + 3);
lzo_bytep out = (lzo_bytep)malloc (out_len);
lzo_voidp wkrmem = (lzo_voidp)malloc (LZO1X_1_MEM_COMPRESS);
zmq_msg_t msg;
rc = lzo1x_1_compress (imageData, uncompressedImageSize,
out, &compressedImageSize, wrkmem);
assert (compressedImageSize > 0);
rc = zmq_msg_init_data (&msg, out, compressedImageSize, my_free, NULL);
assert (rc == 0);
/* Send the message to the socket */
rc = zmq_send (socket, &msg, 0);
assert (rc == 0);
Related
I am trying to send data from a vector over a TCP socket.
I'm working with a vector that I fill with values from 0 to 4999, and then send it to the socket.
Client side, I'm receiving the data into a vector, then I copy its data to another vector until I received all the data from the server.
The issue I'm facing is that when I receive my data, sometimes I will get all of it, and sometimes I will only receive the correct data from 0 to 1625 and then I get trash data until the end (please see the image below). I even received for example from 0 to 2600 correct data, then from 2601 to 3500 it's trash and finally from 3501 to 4999 it's correct again.
(left column is line number and right column is the data).
This is the server side :
vector<double> values2;
for(int i=0; i<5000; i++)
values2.push_back(i);
skt.sendmsg(&values2[0], values2.size()*sizeof(double));
The function sendmsg :
void Socket::sendmsg(const void *buf, size_t len){
int bytes=-1;
bytes = send(m_csock, buf, len, MSG_CONFIRM);
cout << "Bytes sent: " << bytes << endl;
}
Client side :
vector<double> final;
vector<double> msgrcvd(4096);
do{
bytes += recv(sock, &msgrcvd[0], msgrcvd.size()*sizeof(double), 0);
cout << "Bytes received: " << bytes << endl;
//Get rid of the trailing zeros
while(!msgrcvd.empty() && msgrcvd[msgrcvd.size() - 1] == 0){
msgrcvd.pop_back();
}
//Insert buffer content into final vector
final.insert(final.end(), msgrcvd.begin(), msgrcvd.end());
}while(bytes < sizeof(double)*5000);
//Write the received data in a txt file
for(int i=0; i<final.size(); i++)
myfile << final[i] << endl;
myfile.close();
The outputs of the bytes are correct, the server outputs 40 000 when sending the data and the client also outputs 40 000 when receiving the data.
Removing the trailing zeros and then inserting the content of the buffer into a new vector is not very efficient, but I don't think it's the issue. If you have any clues on how to make it more efficient, it would be great!
I don't really know if the issue is when I send the data or when I receive it, and also I don't really get why sometimes (rarely), I get all the data.
recv receives bytes, and doesn't necessarily wait for all the data that was sent. So you can be receiving part of a double.
Your code works if you receive complete double values, but will fail when you receive part of a value. You should receive your data in a char buffer, then unpack it into doubles. (Possibly converting endianness if the server and client are different.)
#include <cstring> // For memcpy
std::array<char, 1024> msgbuf;
double d;
char data[sizeof(double)];
int carryover = 0;
do {
int b = recv(sock, &msgbuf[carryover], msgbuf.size() * sizeof(msgbuf[0]) - carryover, 0);
bytes += b;
b += carryover;
const char *mp = &msgbuf[0];
while (b >= sizeof(double)) {
char *bp = data;
for (int i = 0; i < sizeof(double); ++i) {
*bp++ = *mp++;
}
std::memcpy(&d, data, sizeof(double));
final.push_back(d);
b -= sizeof(double);
}
carryover = b % sizeof(double);
// Take care of the extra bytes. Copy them down to the start of the buffer
for (int j = 0; j < carryover; ++j) {
msgbuf[j] = *mp++;
}
} while (bytes < sizeof(double) * 5000);
This uses type punning from What's a proper way of type-punning a float to an int and vice-versa? to convert the received binary data to a double, and assumes the endianness of the client and server are the same.
Incidentally, how does the receiver know how many values it is receiving? You have a mix of hard coded values (5000) and dynamic values (.size()) in your server code.
Note: code not compiled or tested
TL/DR:
Never-ever send raw data via a network socket and expect them properly received/unpacked on other side.
Detailed answer:
Network is built on top of various protocols, and this is for a reason. Once you send something, there is no warranty you counterparty is on the same OS and same software version. There is no standard how primitive types should be coded on byte level. There is no restriction how much intermittent nodes could be involved into the data delivery, and each of your send() may traverse via different routes. So, you have to formalize the way you send the data, then other party can be sure what is proper way to retrieve them from the socket.
Simplest solution: use a header before your data. So, you plan to send 5000 doubles? Then send a DWORD first, which contains 40000 inside (5k elements, 8 bytes each -> 40k) and push all your 5k doubles right after that. Then, your counterparty should read 4 bytes from the socket first, interpret it as DWORD and understand how much bytes should come then.
Next step: you may want to send not only doubles, but ints and strings as well. That way, you have to expand your header so it can indicate
Total size of further data (so called payload size)
Kind of the data (array of doubles, string, single int etc)
Advanced solution:
Take a look on ready-to-go solutions:
ProtoBuf https://developers.google.com/protocol-buffers/docs/cpptutorial
Boost.Serialization https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc/index.html
Apache Thrift https://thrift.apache.org
YAS https://github.com/niXman/yas
Happy coding!
i'm trying to modify headers of some GTP packets using the mbuf and mempool libraries, specifically i want to cut out all of ETH, IP, UDP ,GTP layers and obtain a (deep) copy of the packet's payload.
Here's the piece of code that should do the work:
void (const unsigned char* packet, size_t size)
{
auto outer_header_len = sizeof(ether_header) + sizeof(ip) + sizeof(udphdr) + sizeof(gtp); //length to cut
uint8_t byte_size = static_cast<uint8_t>(size);
struct rte_mempool* mbuf_pool;
struct rte_mbuf *mbuf_pkt = rte_pktmbuf_alloc(mbuf_pool);
mbuf_pkt->data_len = byte_size;
mbuf_pkt->pkt_len = byte_size;
rte_pktmbuf_append(mbuf_pkt, packet[byte_size]);
auto payload = rte_pktmbuf_adj(mbuf_pkt, outer_header_len);
}
This function is called from a loop which parses the stream of packets and passes packet and size at each iteration. Since there will be lots of calls, how can i make my code memory efficient and better? Any tips?
There are few issues with the code.
byte_size
uint8_t byte_size = static_cast<uint8_t>(size); I am not sure that is correct, since it basically limiting all packets to 256 bytes, which might not be always the case. So I would just deleted this line, since the size itself is a perfectly fine.
rte_pktmbuf_append()
rte_pktmbuf_append(mbuf_pkt, packet[byte_size]); this function increases the length of mbuf_pkt, but do not change the mbuf bytes themselves. So we have to use the pointer it returns to actually do the copy, i.e.
ptr = rte_pktmbuf_append(mbuf_pkt, size);
if (ptr != NULL)
rte_memcpy(ptr, packet + outer_header_len, size -outer_header_len);
Here are also the links to the rte_pktmbuf_append() and rte_memcpy() documentation.
Other Mbuf Length Manipulations
Since the rte_pktmbuf_append() increases the length of the packet, the following lines are redundant:
mbuf_pkt->data_len = byte_size;
mbuf_pkt->pkt_len = byte_size;
auto payload = rte_pktmbuf_adj(mbuf_pkt, outer_header_len);
Performance
We can copy data without the outer header by taking into account outer_header_len during the data copy as in the example before:
ptr = rte_pktmbuf_append(mbuf_pkt, size);
if (ptr != NULL)
rte_memcpy(ptr, packet + outer_header_len, size -outer_header_len);
A check if the outer_header_len is less than the packet length might be also required, unless there is such a check made earlier in the code.
As a test, I'm writing a series of byte arrays to a tcp socket from an Android application, and reading them in a C++ application.
Java
InetAddress address = InetAddress.getByName("192.168.0.2");
Socket socket = new Socket(address, 1300);
DataOutputStream out = new DataOutputStream(socket.getOutputStream())
...
if(count == 0) {
out.write(first, 0, first.length);
} else if(count == 1) {
out.write(second, 0, second.length);
}
C++
do {
iResult = recv(ClientSocket, recvbuf, 3, 0);
for (int i = 0; i < 3; i++) {
std::cout << (int)(signed char)recvbuf[i] << std::endl;
}
} while (iResult > 0);
As it stands, on the first receipt, recv[2] = -52, which I assume to be a junk value, as the output stream has not yet written the second byte array by the time I've received the first segment.
However, when I pause after the the ListenSocket has accepted the connection:
ClientSocket = accept(ListenSocket, NULL, NULL);
std::cin.ignore();
...giving the sender time to do both writes to the stream, recv[2] = 3, which is the first value of the second written byte array.
If I ultimately want to send and receive a constant stream of discrete arrays, how can I determine after I've received the last value of one array, whether the next value in the buffer is the first value of the next array or whether it's a junk value?
I've considered that udp is more suitable for sending a series of discrete data sets, but I need the reliability of tcp. I imagine that tcp is used in this way regularly, but it's not clear to me how to mitigate this issue.
EDIT:
In the actual application for which I'm writing this test, I do implement length prefixing. I don't think that's relevant though; even if I know I'm at the end of a data set, I need to know whether the next value on the buffer is junk or the beginning of the next set.
for (int i = 0; i < 3; i++)
The problem is here. It should be:
for (int i = 0; i < iResult; i++)
You're printing out data that you may not have received. This is the explanation of the 'junk value'.
You can't assume that recv() fills the buffer.
You must also check iResult for both -1 and zero before this loop, and take the appropriate actions, which are different in each case.
As you point out, TCP is stream-based, so there's no built-in way to say "here's a specific chunk of data". What you want to do is add your own "message framing". A simple way to do that is called "length prefixing". Where you first send the size of the data packet, and then the packet itself. Then the receiver will know when they've gotten all the data.
Sending side
send length of packet (as a known size -- say a 32-bit int)
send packet data
Receiving side
read length of packet
read that many bytes of data
process fully-received packet
Check out this article for more information: http://blog.stephencleary.com/2009/04/message-framing.html
I am sending the data from a file using the Libcurl post callback. The example here shows, how to send data as 1 byte per call from callback function.
I have changed the code so the file is read into chunks. That pretty much works fine.
the current sample code is:
if(sizeleft){
*( char *)ptr = readptr[0];
readptr++;
sizeleft--;
return 1;
}
This example sends the data as 1 byte.
but suppose i have to send it multiple bytes.I have tried to increment readptr by two each time and decreasing sizeleft by two and i return 2bytes at a time.
It didnt work like this and the data is corrupted.
I would appreciate if someone out there could help me out.
Thank you
It's difficult to tell from your question exactly what you're doing, exactly what you'd expect to happen and exactly what actually happens. However it looks like you're on the right track.
The documentation for CURLOPT_READFUNCTION states that size * nitems (a.k.a. size * nmemb in the example) is the upper limit of the number of bytes you may write into buffer, and the return value of your function is the actual number of bytes that you wrote. Returning zero means that you have written everything you wish to write, and your callback function won't be called any more.
If the value you return from your function does not equal the number of bytes that you actually wrote into the buffer, then you can expect corruption.
PS: Something like:
// copy as many bytes as we can, up to either:
// * The number of bytes we have remaining.
// or
// * The space available in ptr.
size_t maxBytes = size * nmemb;
size_t numBytes = std::min (maxBytes, sizeleft);
memcpy (ptr, readptr, numBytes);
readptr += numBytes;
sizeleft -= numBytes;
return numBytes;
I'm sending large data (well… 1Mb) via socket, but I don’t know why the send action is blocking the program and never ends. Small sends runs perfectly and I’m couldn’t found where is the problem here. Can anyone help me, please?
Thank you in advance for any help you can provide.
int liResult = 1;
int liConnection = 0;
int liSenderOption = 1;
struct addrinfo laiSenderAddrInfo;
struct addrinfo *laiResultSenderAddrInfo;
memset(&laiSenderAddrInfo,0,sizeof(laiSenderAddrInfo));
laiSenderAddrInfo.ai_socktype = SOCK_STREAM;
laiSenderAddrInfo.ai_flags = AI_PASSIVE;
liResult = getaddrinfo(_sIp.c_str(), _sPort.c_str(), &laiSenderAddrInfo, &laiResultSenderAddrInfo);
if (liResult > -1)
{
liConnection = socket(laiResultSenderAddrInfo->ai_family, SOCK_STREAM, laiResultSenderAddrInfo->ai_protocol);
liResult = liConnection;
if (liConnection > -1)
{
setsockopt(liConnection, SOL_SOCKET, SO_REUSEADDR, &liSenderOption, sizeof(liSenderOption));
liResult = connect(liConnection, laiResultSenderAddrInfo->ai_addr, laiResultSenderAddrInfo->ai_addrlen);
}
}
size_t lBufferSize = psText->length();
long lBytesSent = 1;
unsigned long lSummedBytesSent = 0;
while (lSummedBytesSent < lBufferSize and lBytesSent > 0)
{
lBytesSent = send(liConnection, psText->c_str() + lSummedBytesSent, lBufferSize - lSummedBytesSent, MSG_NOSIGNAL);
if (lBytesSent > 0)
{
lSummedBytesSent += lBytesSent;
}
}
Check the buffer size, you can do so by following this answer
How to find the socket buffer size of linux
In my case, the values are
Minimum = 4096 bytes ~ 4KB
Default = 16384 bytes ~ 16 KB
Maximum = 4022272 bytes ~ 3.835 MB
You can tweak the values net.core.rmem_max and net.core.wmem_max in /etc/sysctl.conf to increase the socket buffer size and reload with sysctl -p.
Source: http://www.runningunix.com/2008/02/increasing-socket-buffer-size-in-linux/
The send() call blocks until all of the data has been sent or buffered. If the program at the other end of the socket isn't reading and thus there is no flow of data, the write buffer at your end will fill up and send() will block. Chances are that when you tried to send a smaller amount of data it fit into the buffer.
See also this answer.
For TCP, the kernel has a fixed size buffer in which is stores unsent data. The size of this buffer is the current window size of the TCP session. Once this buffer is full any new send will be failed. This is a TCP flow control mechanism which prevents you from trying to send data faster than the receiver can consume the data while at the same time providing an automatic resend for lost data. The default window can be as small as 64K but can grow larger for high latency high bandwidth networks.
What you probably need to do is break the data up into smaller send blocks and then ensure you have a flow-off mechanism for when your send buffer is full.