Variable length Message over TCP Socket in C++ - c++

I was trying to send some length prefix data to the server, i have tried harder by using the code and solution posted by other people on stack overflow. Still looking how things are actually working using TCP.As i don't know so much(about network programming but i am aware about theoretical concepts). i am writing what i have tried so far.Based on that i have some questions.
As we are using Char Buffer[200] = "This is data" on client side to send(using send() function) string and char type data to server(receiving using recv()function). Till here its okay but what if i need to send some variable length message with their length information ? , How i can encode length info to the message ?
for example: 0C 54 68 69 73 20 69 73 20 64 61 74 61 07 46 72 6f 6d 20 6d 65
(length) T h i s i s d a t a (length) F r o m m e
How can i interpret these two message seperately from tcp stream at the sever side ?
i dont know how length information can be sent separetly ?. Or if someone can understand my test case(given below in edit) to verify length information of the string.
EDIT: It seems okay but i just need verify a bit about prefix length. I am sending 20 bytes("This is data From me") to server. My recevied length size is 4 bytes(i dont know whats inside, i need to verify does the 4 bytes of length i received contain 0000 0000 0000 0000 0000 00000 0001 0100). Thats it, so the way i thought to verify it by shifting length info by 2 bits right now it should look like ( 4 bytes of length i received contain 0000 0000 0000 0000 0000 00000 0000 0101) in this case i should get only 5 characters i.e "This ". Do you know how can i verify this at server side ?
Client Code
int bytesSent;
int bytesRecv = SOCKET_ERROR;
char sendbuf[200] = "This is data From me";
int nBytes = 200, nLeft, idx;
nLeft = nBytes;
idx = 0;
uint32_t varSize = strlen (sendbuf);
bytesSent = send(ConnectSocket,(char*)&varSize, 4, 0);
assert (bytesSent == sizeof (uint32_t));
std::cout<<"length information is in:"<<bytesSent<<"bytes"<<std::endl;
// code to make sure all data has been sent
while (nLeft > 0)
{
bytesSent = send(ConnectSocket, &sendbuf[idx], nLeft, 0);
if (bytesSent == SOCKET_ERROR)
{
std::cerr<<"send() error: " << WSAGetLastError() <<std::endl;
break;
}
nLeft -= bytesSent;
idx += bytesSent;
}
bytesSent = send(ConnectSocket, sendbuf, strlen(sendbuf), 0);
printf("Client: Bytes sent: %ld\n", bytesSent);
Server Code
uint32_t nlength;
int length_received = recv(m_socket,(char*)&nlength, 4, 0);
char *recvbuf = new char[nlength];
int byte_recived = recv(m_socket, recvbuf, nlength, 0);
Thanks

If you need to send variable-length data, you need to send the length of that data before the data itself.
In you code snippet above, you appear to be doing the opposite:
while (nLeft > 0)
{
bytesSent = send(ConnectSocket, &sendbuf[idx], nLeft, 0);
// [...]
}
bytesSent = send(ConnectSocket, sendbuf, strlen(sendbuf), 0);
Here you send the string first, and then the length. How is the client going to be able to interpret that? By the time the get the length, they have already pulled the string off the socket.
Instead, send the length first, and make sure you're explicit about the size of the size field:
const uint32_t varSize = strlen (sendbuf);
bytesSent = send(ConnectSocket, &varSize, sizeof (varSize), 0);
assert (bytesSent == sizeof (uint32_t));
while (nLeft > 0)
{
bytesSent = send(ConnectSocket, &sendbuf[idx], nLeft, 0);
// [...]
}
You might also consider not sending variable-length data at all. Fixed-width binary protocols are easier to parse on the receiving side in general. You could always send string data in a fixed-width field (say, 20 chars) and pad it out with spaces or \0's. This does waste some space on the wire, at least in theory. If you are smart about the sizes of the fixed-width fields and what you send in them, you can be economical with this space in many cases.

Related

storing the remaining char * and pre-appending it to original char * for processing

I am reading from a binary file.
char bigbuf[5000];
while (read(fd, bigbuf, 2) != 0) {
uint16_t packetSize = htons(*(uint16_t *)bigbuf);
read(fd, bigbuf + 2, packetSize - 2);
myParser.onUDPPacket(bigbuf, packetSize);
}
The packets are written in binary file are of 40 bytes, but inside onUDPPacket function I receive a packet of 61 bytes then on the second call I again receive a 60 byte packet. Now I have to write the function onUDPPacket such that on the first call it processes 40 byte of data from the 61 byte received and it has to append remaining 21 bytes in the starting of the next 60 byte of packet. How do I do this append thing ?
void Parser::onUDPPacket(const char *buf, size_t len)
{
}

Creating and sending raw data packets in C/C++

I know how my packet looks like. It has 6 header fields (1 byte each, each header has 8 fields) and then it has the payload (data).
I would like to build a raw packet in C or C++ (it should look the same I think).
Here's what I think I should do:
unsigned char packet[11];
packet[0] = (0x81); // first header with 8 fields
packet[1] = (0x8c); // second header with 8 fields
packet[2] = (0xfe);
packet[3] = (0x84);
packet[4] = (0x1d);
packet[5] = (0x79);
packet[6] = (0x96); // payload, the 'h' letter, masked
packet[7] = (0xe1); // 'e'
packet[8] = (0x71); // 'l'
packet[9] = (0x15); // 'l'
packet[10] = (0x91);// 'o'
Where, for instance, 0x81 is the first byte (I simply converted every field (bit) of my first header to hex).
And then, simply, I want to send it to server: send(sockfd, packet, sizeof(packet), 0) to send it.
Receiving and printing the response:
unsigned char buffer[1024];
if ((recv(sockfd, buffer, len, 0)) == 0)
{
if (errno != 0)
{
exit(1);
}
}
int i;
for(i = 0; i<len; i++)
printf("%x ", buffer[i]);
Am I right?
Other than mishandling the return value from recv, your code looks okay.
if ((recv(sockfd, buffer, len, 0)) == 0)
{
if (errno != 0)
{
exit(1);
}
}
A zero return indicates normal close of the connection. There's no reason to check errno if it returns zero.
A return value of -1 indicates an error. In that case, it does make sense to check errno.
A value greater than zero indicates that number of bytes have been received. Be aware that it is perfectly normal for recv to return fewer bytes than you asked it for. If you want to receive exactly some number of bytes, you must call recv in a loop.
TCP is a byte-stream protocol and has no idea where your "packets" (really, messages) begin and end.
Your code will not appear to be error-prone!
But a good practice would be:
const std::uint32_t BUFFER_SIZE = 11;
std::vector<std::uint8_t> buffer;
buffer.reserve(BUFFER_SIZE)
buffer = {0x81,0x8c.....};
send( sockfd,
reinterpret_cast <const char*> ( buffer.data() ),
static_cast <int> ( buffer.size() ),
0
);
Doing so, your code gets more optimized, and avoids possible leaks, using the std vectors.
May also benefit from taking a look at ZeroMQ, as an example of a ready-made, high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications.

How to send and receive large amounts of data in udp c++

I am trying to send and receive large amounts of data at once in udp c++, with the following code. I can send at once just 16000 bits, char. How can one send/receive millions of bytes of data without closing the socket?
//sends the data contained in aliceBuf, which is, char of size 16000.
if (sendto(aliceSocket, aliceBuf, strlen(aliceBuf), 0, (struct sockaddr *)&bobAddr, sizeof (bobAddr)) == -1) {
perror("sendto");
exit(1);
}
// receiver code: it is receiving just 16000 char.
recvlen = recvfrom(aliceSocket, aliceBuf1, receiveBuffer, 0, (struct sockaddr*)&bobAddr, &bobAddrSize);
if (recvlen >= 0) {
aliceBuf1[recvlen] = 0; /* expect a printable string - terminate it */
}
You can send a large amount of data in one go, but the question you have to ask yourself is: How will the receiver know how much data to expect?
I normally handle these cases by either encoding the length explicitly by prefixing the data with the length and then the receiver loops until that amount of data has arrived, or by having some sort of end of data marker like 'C' strings or more implicitly like json data and the receiver loops looking for something in the data itself.
You wil have to add a protocol on top of UDP, just as if you were using TCP. I'm sorry that you have to do some work, but that's just how things are. Some of the datagrams may get lost, so you may have to add a layer for that too. 1M bits is ~twice as large as the largest possible UDP datagram anyway, so even if you reconfigure your network stack to allow larger datagrams, you will still hit the 64k limit, so requiring a protocol.
I did with loopiing like this:
int totalGoLength= no of blocks you want to send
int dataLengthOneGo = length of data in one block you want to send
//start loop
int iii=1 ;
while (iii <= totalGoLength){ //send by dividing into packets
////--SEND/by converting to char * for less memory occupation----
// theString has the string data to send
std::string part(theString.substr(0, dataLengthOneGo));
char * s4;
s4 = new char[part.size() + 1];
memcpy(s4, part.c_str(), part.size() + 1);
if (sendto(aliceSocket, s4, strlen(s4), 0, (struct sockaddr *)&bobAddr, sizeof (bobAddr)) == -1) {
perror("sendto");
exit(1);
}
delete [] s4;
////----------------------Receiving------------
// receive buffer should have sufficient memory allocation
char *aliceBuf1;
aliceBuf1 = new char[receiveBuffer];
recvlen = recvfrom(aliceSocket, aliceBuf1, receiveBuffer, 0, (struct sockaddr *)&bobAddr, &bobAddrSize);
if (recvlen >= 0) {
aliceBuf1[recvlen] = 0; /* expect a printable string - terminate it */
//convert char to string
string s1(aliceBuf1);
//erase the white space
s1.erase(remove_if(s1.begin(), s1.end(), isspace), s1.end());
//convert string into integer vector
std::vector<int> ints;
ints.reserve(s1.size());
std::transform(std::begin(s1), std::end(s1), std::back_inserter(ints), [](char c) {
return c - '0'; });
}
delete[] aliceBuf1;
justCopy=ints;
KeepData.insert(KeepData.end(),justCopy .begin(), justCopy.end());
justCopy.erase(justCopy.begin(), justCopy.end()); //erase for next time
ints.erase(ints.begin(), ints.end()); //erase for next time
theString.erase(theString.begin(), theString.begin() + dataLengthOneGo);//keep the remaining
iii=iii+1;
}//end of the while

c++ winsock - recv() returns incorrect and strange buffer

I am writing a simple game named "TicTacToe". I also wanted to create a network function using WinSock. While connection between two users goes without errors, I can't receive any correct data. After receiving it, the buffer is filled with a strange characters like "☺$0/A", etc., while it should receive "Hello!".
Here's my code:
const char buf[] = "Hello!";
char rbuf[16];
int bytes = 0;
memset(buf, 0, sizeof(buf));
// sending
send(sox, buf, sizeof(buf), 0);
// recv
bytes = recv(sox, rbuf, 16, 0)
rbuf[bytes] = '\0';
cout << rbuf << endl;
Well, for one, you're not checking the return values of your send/recv calls, also you can't expect it to always receive 16 bytes. In order to ensure you get what you want to receive, you might have to call recv multiple times. Here's an example solution to that problem:
int FixedRecv(SOCKET sox, char* rbuf, int length)
{
int ref, len = 0;
do {
ref = recv(sox, rbuf + len, length - len, 0);
if(ref == 0) return 0;
if(SOCKET_ERROR != ref)
len += ref;
else return SOCKET_ERROR;
} while(len < length);
return len;
}
Same will also apply to your calls to send but I just want you to get a general idea.
Hope that helps, uses a little bit of pointer arithmetic, but nothing
too extensive.
Also, if you're using nonblocking sockets, you will want to check for WSAEWOULDBLOCK with WSAGetLastError.

Data through Sockets in C++

I am currently working on a project that uses the network. I have to send a struct
struct Header
{
uint32_t magic;
uint32_t checksum;
uint32_t timestamp;
uint16_t commandId;
uint16_t dataSize;
};
struct Packet
{
struct Header header;
char data[128];
};
I'm trying to send the struct Packet from one socket to another using TCP. I've tried to send my struct like that
send(socket, &my_struct, sizeof(my_struct), 0);
but it is not working so I've tried to serialize my struct into a char*
unsigned char *Serialization::serialize_uint32(unsigned char *buffer, uint32_t arg)
{
buffer[3] = (arg >> 24);
buffer[2] = (arg >> 16);
buffer[1] = (arg >> 8);
buffer[0] = (arg);
return (buffer + sizeof(uint32_t));
}
unsigned char *Serialization::serialize_uint16(unsigned char *buffer, uint16_t arg)
{
buffer[1] = (arg >> 8);
buffer[0] = (arg);
return (buffer + sizeof(uint16_t));
}
unsigned char *Serialization::deserialize_uint32(unsigned char *buffer, uint32_t *arg)
{
memcpy((char*)arg, buffer, sizeof(uint32_t));
return (buffer + sizeof(uint32_t));
}
unsigned char *Serialization::deserialize_uint16(unsigned char *buffer, uint16_t *arg)
{
memcpy((char*)arg, buffer, sizeof(uint16_t));
return (buffer + sizeof(uint16_t));
}
even when a client symply send a struct Header data is corrupt when I read it server side
Why is the data corrupt ?
Client sending loop
TcpSocket tcp;
Packet p;
std::stringstream ss;
int cpt = 0;
int ret = 0;
char *serialized;
tcp.connectSocket("127.0.0.1", 4242);
while (getchar())
{
ss.str("");
ss.clear();
ss << cpt++;
p.header.magic = 0;
p.header.checksum = 1;
p.header.timestamp = 2;
p.header.commandId = 3;
p.header.dataSize = ss.str().length();
memset(p.data, 0, 128);
memcpy(p.data, ss.str().c_str(), ss.str().length());
serialized = new char[sizeof(Header) + ss.str().length()];
bzero(serialized, sizeof(Header) + ss.str().length());
Serialization::serialize_packet(serialized, p);
hexDump("serialized", serialized+1, sizeof(Header) + ss.str().length());
ret = tcp.write(serialized+1, sizeof(Header) + ss.str().length());
}
server recv loop: (fonction call by select() )
buff = new char[bav];
socket->read(buff, bav);
hexdump("buff", buff, bav);
socket->read() :
int TcpSocket::read(char *buff, int len)
{
int ret;
ret = recv(this->_socket, buff, len, 0);
return (ret);
}
when I run those programs :
./server
[Server] new connexion :: [5]
recv returns : 17
buff serialized:
0000 00 00 00 00 14 00 00 00 1c 00 00 00 1a 00 00 00 ................
0010 1b
./client
serialized data:
0000 00 00 00 00 00 00 01 00 00 00 02 00 03 00 01 30 ...............0
0010 00
send returns : 17
So, this is wrong, and it will definitely cause errors.
buff = new char[bav];
socket->read(buff, bav);
hexdump("buff", buff, bav);
socket->read() :
int TcpSocket::read(char *buff, int len)
{
return recv(this->_socket, buff, len, 0);
}
The return value from recv() must not be ignored.
From man 2 recv:
RETURN VALUES
These calls return the number of bytes received, or -1 if an error
occurred.
For TCP sockets, the return value 0 means the peer has closed its half
side of the connection.
So, how many bytes did you receive? It's impossible to tell, if you discard the result from recv(). Maybe recv() failed, you'd never find out if you don't check the return value. Maybe it only filled up part of your buffer. You have to check the return code from recv(). This is the number one error people make when writing programs that use TCP.
You will need to alter your code to handle the following cases:
The recv() call may completely fill your buffer.
The recv() call may partially fill your buffer.
The recv() call may return 0, indicating that the sender has shut down the connection.
The recv() call may indicate EINTR because it was interrupted by a system call.
The recv() call may indicate ECONNRESET because the sender has closed the connection suddenly or has disappeared.
The recv() call may encounter some other error.
Remember: when using TCP, just because you send() 16 bytes doesn't mean that the other peer will recv() 16 bytes — it may be broken up into chunks. TCP is a stream protocol. Unlike UDP, adjacent chunks of data can be arbitrarily joined or split.
You need to mask only the low 8 bits each time:
buffer[3] = (arg >> 24) & 0xff;
buffer[2] = (arg >> 16) & 0xff;
buffer[1] = (arg >> 8) & 0xff;
buffer[0] = (arg) & 0xff;
Do the same when you deserialize
If I were you I would not reinvent the wheel. There are a lot of well documented and tested libraries / protocols out there for exactly the purpose you are looking for. A small list which just comes to my mind:
boost serialization
Google Protocol Buffers
JSON
XML
CORBA