I am trying to send and receive large amounts of data at once in udp c++, with the following code. I can send at once just 16000 bits, char. How can one send/receive millions of bytes of data without closing the socket?
//sends the data contained in aliceBuf, which is, char of size 16000.
if (sendto(aliceSocket, aliceBuf, strlen(aliceBuf), 0, (struct sockaddr *)&bobAddr, sizeof (bobAddr)) == -1) {
perror("sendto");
exit(1);
}
// receiver code: it is receiving just 16000 char.
recvlen = recvfrom(aliceSocket, aliceBuf1, receiveBuffer, 0, (struct sockaddr*)&bobAddr, &bobAddrSize);
if (recvlen >= 0) {
aliceBuf1[recvlen] = 0; /* expect a printable string - terminate it */
}
You can send a large amount of data in one go, but the question you have to ask yourself is: How will the receiver know how much data to expect?
I normally handle these cases by either encoding the length explicitly by prefixing the data with the length and then the receiver loops until that amount of data has arrived, or by having some sort of end of data marker like 'C' strings or more implicitly like json data and the receiver loops looking for something in the data itself.
You wil have to add a protocol on top of UDP, just as if you were using TCP. I'm sorry that you have to do some work, but that's just how things are. Some of the datagrams may get lost, so you may have to add a layer for that too. 1M bits is ~twice as large as the largest possible UDP datagram anyway, so even if you reconfigure your network stack to allow larger datagrams, you will still hit the 64k limit, so requiring a protocol.
I did with loopiing like this:
int totalGoLength= no of blocks you want to send
int dataLengthOneGo = length of data in one block you want to send
//start loop
int iii=1 ;
while (iii <= totalGoLength){ //send by dividing into packets
////--SEND/by converting to char * for less memory occupation----
// theString has the string data to send
std::string part(theString.substr(0, dataLengthOneGo));
char * s4;
s4 = new char[part.size() + 1];
memcpy(s4, part.c_str(), part.size() + 1);
if (sendto(aliceSocket, s4, strlen(s4), 0, (struct sockaddr *)&bobAddr, sizeof (bobAddr)) == -1) {
perror("sendto");
exit(1);
}
delete [] s4;
////----------------------Receiving------------
// receive buffer should have sufficient memory allocation
char *aliceBuf1;
aliceBuf1 = new char[receiveBuffer];
recvlen = recvfrom(aliceSocket, aliceBuf1, receiveBuffer, 0, (struct sockaddr *)&bobAddr, &bobAddrSize);
if (recvlen >= 0) {
aliceBuf1[recvlen] = 0; /* expect a printable string - terminate it */
//convert char to string
string s1(aliceBuf1);
//erase the white space
s1.erase(remove_if(s1.begin(), s1.end(), isspace), s1.end());
//convert string into integer vector
std::vector<int> ints;
ints.reserve(s1.size());
std::transform(std::begin(s1), std::end(s1), std::back_inserter(ints), [](char c) {
return c - '0'; });
}
delete[] aliceBuf1;
justCopy=ints;
KeepData.insert(KeepData.end(),justCopy .begin(), justCopy.end());
justCopy.erase(justCopy.begin(), justCopy.end()); //erase for next time
ints.erase(ints.begin(), ints.end()); //erase for next time
theString.erase(theString.begin(), theString.begin() + dataLengthOneGo);//keep the remaining
iii=iii+1;
}//end of the while
Related
I want to send a string from a server to a client. To account for the case that the string is bigger than the buffer that the client reads into, I send a byte count to the client before the actual string. Then on the client side, I split the read operation into multiple recv() calls if needed.
Server:
// Send byte count to client so it knows how much data to expect
strcpy(buffer, std::to_string(string_size).c_str());
int result{ send(socket, buffer, buffer_size, 0) };
// Send actual data
unsigned int bytes_to_send{ sizeof(s) };
while (bytes_to_send > 0) {
send_bytes = std::min(buffer_size, bytes_to_send);
strcpy(buffer, s.substr(0, send_bytes).c_str());
result = send(socket, buffer, send_bytes, 0);
if (s.size() > send_bytes) {
s = s.substr(send_bytes, s.size() - send_bytes);
}
bytes_to_send -= send_bytes;
return_int += send_bytes;
(In the actual code, I check result to see whether send() succeeded)
Client:
result = recv(socket, buffer, buffer_size, 0);
count = std::stoi(buffer);
while(count > 0){
read_bytes = std::min(count, buffer_size);
result = recv(socket, buffer, read_bytes, 0);
if(result < 1 ) {
// throw exception
}else{
return_string += buffer;
}
count -= read_bytes;
What I expected:
Client blocks on first recv() call until server sends byte count
Client proceeds until second recv() call, then blocks until server sends string
What actually happens:
Client blocks on first recv() call until server sends byte count
Client does not block on second recv() call. recv() return value is equal to read_bytes, but does not fill the buffer with anything
Thanks in advance for any help with this
There are quite a few problems with your code.
strcpy(buffer, std::to_string(string_size).c_str()); is a buffer overflow waiting to happen.
send(socket, buffer, buffer_size, 0) - you did not set buffer_size to the number of bytes copied into buffer by strcpy(). It looks like you are sending the entire buffer without regard to how many bytes it is actually holding.
sizeof(s) does not give you the number of characters in a std::string. It gives you the byte size of the std::string class itself, which is not the same thing. You need to use the string's size() or length() method to get the proper number of characters.
TCP is a byte stream, there is no 1-to-1 relationship between send() and recv(). Both can report fewer bytes were processed than requested, but you are not accounting for that properly.
You are completely ignoring the return value of send(), assuming that all requested bytes have been sent in full on each call, when in actuality fewer bytes may have been sent, or none at all. On the 1st call, you are not accounting for the possibility that the entire size string may not be sent in 1 call and may need to be looped. Inside of the loop, you are completely ignoring the return value of send() for purposes of looping.
Likewise, you are ignoring the return value of recv() on the 1st call, assuming the entire string size is received in 1 call, without regard to how many characters are actually in the string size. Inside the loop, you are paying attention to the return value of recv() for purposes of looping, except that you are not accounting for the possibility that recv() returns 0 on a graceful disconnect. You are not breaking your loop if that happens.
It is good that you want to send the string's size before sending its content. However, you are sending the size as another variable-length string, which really defeats the purpose of sending a size at all. You should send the size using a fixed-length integer in binary form instead.
Try something more like this instead:
void send_raw(int socket, const void *buffer, size_t buffer_size)
{
const char *ptr = (const char*) buffer;
while (buffer_size > 0) {
int sent = send(socket, ptr, buffer_size, 0);
if (sent < 0) {
// throw exception
}
ptr += sent;
buffer_size -= sent;
}
}
void send_u32(int socket, uint32_t value)
{
value = htonl(value);
send_raw(socket, &value, sizeof(value));
}
...
// Send byte count to client so it knows how much data to expect
send_u32(socket, s.size());
// Send actual data
send_raw(socket, s.c_str(), s.size());
void recv_raw(int socket, void *buffer, size_t buffer_size)
{
char *ptr = (char*) buffer;
while (buffer_size > 0) {
int recvd = recv(socket, ptr, buffer_size, 0);
if (recvd <= 0) {
// throw exception
}
ptr += recvd;
buffer_size -= recvd;
}
}
uint32_t recv_u32(int socket)
{
uint32_t value;
recv_raw(socket, &value, sizeof(value));
return ntohl(value);
}
...
size_t count = recv_u32(socket);
return_string.resize(count);
recv_raw(socket, &return_string[0], count);
However, if you really want to stick with sending the string size as another string, you need to delimit it, such as sending its null terminator, eg:
// Send byte count to client so it knows how much data to expect
std::string size_buffer = std::to_string(s.size());
send_raw(socket, size_buffer.c_str(), size_buffer.size()+1);
// Send actual data
send_raw(socket, s.c_str(), s.size());
std::string size_buffer;
char ch;
do {
recv_raw(socket, &ch, 1);
if (ch == '\0') break;
size_buffer += ch;
}
while (true);
size_t count = std::stoi(size_buffer);
return_string.resize(count);
recv_raw(socket, &return_string[0], count);
You can add MSG_WAITALL to the recv() flags and do only one read.
I know how my packet looks like. It has 6 header fields (1 byte each, each header has 8 fields) and then it has the payload (data).
I would like to build a raw packet in C or C++ (it should look the same I think).
Here's what I think I should do:
unsigned char packet[11];
packet[0] = (0x81); // first header with 8 fields
packet[1] = (0x8c); // second header with 8 fields
packet[2] = (0xfe);
packet[3] = (0x84);
packet[4] = (0x1d);
packet[5] = (0x79);
packet[6] = (0x96); // payload, the 'h' letter, masked
packet[7] = (0xe1); // 'e'
packet[8] = (0x71); // 'l'
packet[9] = (0x15); // 'l'
packet[10] = (0x91);// 'o'
Where, for instance, 0x81 is the first byte (I simply converted every field (bit) of my first header to hex).
And then, simply, I want to send it to server: send(sockfd, packet, sizeof(packet), 0) to send it.
Receiving and printing the response:
unsigned char buffer[1024];
if ((recv(sockfd, buffer, len, 0)) == 0)
{
if (errno != 0)
{
exit(1);
}
}
int i;
for(i = 0; i<len; i++)
printf("%x ", buffer[i]);
Am I right?
Other than mishandling the return value from recv, your code looks okay.
if ((recv(sockfd, buffer, len, 0)) == 0)
{
if (errno != 0)
{
exit(1);
}
}
A zero return indicates normal close of the connection. There's no reason to check errno if it returns zero.
A return value of -1 indicates an error. In that case, it does make sense to check errno.
A value greater than zero indicates that number of bytes have been received. Be aware that it is perfectly normal for recv to return fewer bytes than you asked it for. If you want to receive exactly some number of bytes, you must call recv in a loop.
TCP is a byte-stream protocol and has no idea where your "packets" (really, messages) begin and end.
Your code will not appear to be error-prone!
But a good practice would be:
const std::uint32_t BUFFER_SIZE = 11;
std::vector<std::uint8_t> buffer;
buffer.reserve(BUFFER_SIZE)
buffer = {0x81,0x8c.....};
send( sockfd,
reinterpret_cast <const char*> ( buffer.data() ),
static_cast <int> ( buffer.size() ),
0
);
Doing so, your code gets more optimized, and avoids possible leaks, using the std vectors.
May also benefit from taking a look at ZeroMQ, as an example of a ready-made, high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications.
I'm writing a console application using c++ which using sockets and send an HTTP GET request to a server but the response is an html file bigger than 1000000 infact my buffer: char buffer[1000000]; is too small.
I need to receive bigger data from the server than the size of buffer.
I use this code but what is the way to receive a bigger response? I'm a beginner in this programming area so please help me with code and explenations thanks:
char buffer[1000000];
int nDataLength;
while ((nDataLength = recv(Socket, buffer, 1000000, 0)) > 0) {
int i = 0;
while (buffer[i] >= 32 || buffer[i] == '\n' || buffer[i] == '\r') {
myString += buffer[i];
i += 1;
}
}
cout << myString << "\n";
You need to use a smaller fixed length buffer when reading from the socket, and then append the received data to a dynamically growing buffer (like a std::string, or a file) on each loop iteration. recv() tells you how many bytes were actually received, do not access more than that many bytes when accessing the buffer.
char buffer[1024];
std::string myString;
int nDataLength;
while ((nDataLength = recv(Socket, buffer, sizeof(buffer), 0)) > 0) {
myString.append(buffer, nDataLength);
}
std::cout << myString << "\n";
recv return value is total size of receved data.
so you can know total data size, if your buffer is smaller than total data size there is 2 solutions. I guess...
1. allocate buffer on the heap. using like new, allcoc etc.
2. store received data to data structure(like circular queue, queue) while tatal data size is zero(recv function return)
I prefer to use 2nd solution.
Googling about recv function , socket programming sample codes.
That'll helpfull.
i developed client server program using c++,so i want to receive more than 500kb , my client message is terminated with "!" ,so i want to receive until my last byte(!) receive ,
this is my code it doesn't work.what is wrong with it.
do
{
int num = recv(*csock, buf, bytesLeft,0);
if (num == 0)
{
break;
}
else if (num < 0 && errno != EINTR)
{
fprintf(stderr, "Exit %d\n", __LINE__);
exit(1);
}
else if (num > 0)
{
numRd += num;
buf += num;
bytesLeft -= num;
fprintf(stderr, "read %d bytes - remaining = %d\n", num, bytesLeft);
}
}
while (bytesLeft != 0);
fprintf(stderr, "read total of %d bytes\n", numRd);
While I'm not sure exactly what your problem is because of the wording of your question, you generally can't use strcat to append raw buffers received over the network unless you know specifically they will be NULL-terminated, and even then, that's not really "safe" in the event you get an unexpected data transmission. The assumption with c-strings is that they are NULL-terminated, but a raw network buffer may not be, and using strcat will cause you to over-run the input buffer should it not be NULL-terminated. Instead of strcat, use a known fixed-size buffer of size N bytes for receiving the data into, and increment a temporary pointer through the buffer until you reach the end of the buffer or the end of the packet transmission. That way you will always read from the network up to N bytes and no more, and prevent buffer over-run situations from occuring.
For instance, you can do the following (this is not the fastest or more efficient solution because of all the copying, but it works):
unsigned char buf[10000]; //10Kb fixed-size buffer
unsigned char buffer[MAXRECV]; //temporary buffer
unsigned char* temp_buf = buf;
unsigned char* end_buf = buf + sizeof(buf);
do
{
iByteCount = recv(GetSocketId(), buffer,MAXRECV,0);
if ( iByteCount > 0 )
{
//make sure we're not about to go over the end of the buffer
if (!((temp_buf + iByteCount) <= end_buf))
break;
fprintf(stderr, "Bytes received: %d\n",iByteCount);
memcpy(temp_buf, buffer, iByteCount);
temp_buf += iByteCount;
}
else if ( iByteCount == 0 )
{
if(temp_buf != buf)
{
//do process with received data
}
else
{
fprintf(stderr, "receive failed");
break;
}
}
else
{
fprintf(stderr, "recv failed: ");
break;
}
} while(iByteCount > 0 && temp_ptr < end_buf); //check for end of buffer
Do you need all 1MB+ of data in one contiguous byte buffer? If so, and you stick with that protocol that has a terminating '!' and does not have a header that includes the length, then you ar stuck with memcpy() and realloc() a lot or some other buffer type like std::vector which, really just does the same thing.
If you don't need all those bytes in one string, you can store them in some other way, eg. a vector of *buffer, and so avoid copying.
Assuming you are using a blocking socket (which is the default mode for sockets), then recv() will block waiting for the full MAXRECV number of bytes to arrive. If the client sends less than that number of bytes, recv() will block waiting for data that does not arrive.
To work around that, you need to either:
1) call recv() with a 1-byte buffer, calling recv() until you encounter your ! byte.
2) call select() before calling recv() to detect when the socket actually has data to read, then call ioctlsocket(FIONREAD) to determine how many bytes can actually be read with recv() without blocking, then have recv() read that number of bytes.
My goal is create an app client server, written in C++.
When the server read an input from the client, should process the string and give an output.
Basically, I have a simply echo server that send the same message.
But if the user types a special string (like "quit"), the program have to do something else.
My problem is that this one dont happend, because the comparison between strings is not working... I dunno why!
Here a simple code:
while(1) {
int num = recv(client,buffer,BUFSIZE,0);
if (num < 1) break;
send(client, ">> ", 3, 0);
send(client, buffer, num, 0);
char hello[6] ="hello";
if(strcmp(hello,buffer)==0) {
send(client, "hello dude! ", 12, 0);
}
buffer[num] = '\0';
if (buffer[num-1] == '\n')
buffer[num-1] = '\0';
std::cout << buffer;
strcpy(buffer, "");
}
Why the comparison is not working?
I have tried many solutions...but all failed :(
Your data in buf may not be NULL-terminated, because buf contains random data if not initialized. You only know the content of the first num bytes. Therefore you also have to check how much data you've received before comparing the strings:
const char hello[6] ="hello";
size_t hello_sz = sizeof hello - 1;
if(num == hello_sz && memcmp(hello, buffer, hello_sz) == 0) { ...
As a side note, this protocol will be fragile unless you delimit your messages, so in the event of fragmented reads (receive "hel" on first read, "lo" on the second) you can tell where one message starts and another one ends.
strcmp requires null terminated strings. The buffer you read to might have non-null characters after the received message.
Either right before the read do:
ZeroMemory(buffer, BUFSIZE); //or your compiler defined equivalent
Or right after the read
buffer[num] = '\0';
This will ensure that there is a terminating null at the end of the received message and the comparison should work.
A string is defined to be an array of chars upto and including the terminating \0 byte. Initially your buffer contains arbitrary bytes, and is not even guaranteed to contain a string. You have to set buffer[num] = '\0' to make it a string.
That of course means that recv should not read sizeof buffer bytes but one byte less.