I'm writing a console application using c++ which using sockets and send an HTTP GET request to a server but the response is an html file bigger than 1000000 infact my buffer: char buffer[1000000]; is too small.
I need to receive bigger data from the server than the size of buffer.
I use this code but what is the way to receive a bigger response? I'm a beginner in this programming area so please help me with code and explenations thanks:
char buffer[1000000];
int nDataLength;
while ((nDataLength = recv(Socket, buffer, 1000000, 0)) > 0) {
int i = 0;
while (buffer[i] >= 32 || buffer[i] == '\n' || buffer[i] == '\r') {
myString += buffer[i];
i += 1;
}
}
cout << myString << "\n";
You need to use a smaller fixed length buffer when reading from the socket, and then append the received data to a dynamically growing buffer (like a std::string, or a file) on each loop iteration. recv() tells you how many bytes were actually received, do not access more than that many bytes when accessing the buffer.
char buffer[1024];
std::string myString;
int nDataLength;
while ((nDataLength = recv(Socket, buffer, sizeof(buffer), 0)) > 0) {
myString.append(buffer, nDataLength);
}
std::cout << myString << "\n";
recv return value is total size of receved data.
so you can know total data size, if your buffer is smaller than total data size there is 2 solutions. I guess...
1. allocate buffer on the heap. using like new, allcoc etc.
2. store received data to data structure(like circular queue, queue) while tatal data size is zero(recv function return)
I prefer to use 2nd solution.
Googling about recv function , socket programming sample codes.
That'll helpfull.
Related
So i have created a java server and a c++ client.
The java server sends a message with a printwriter to the c++ client to execute a command (the data transfer is correct, no problems with that)
Im using a strcmp() to check if the string that client recieved with recv() is the string i want but when i try to check it, it doesn't work. I've tried to print out the line with the recieved buffer and i dont see any problems.
Here is the code that recieves and checks the buffer(c++, ignore some values becouse this is a small piece of the code)
char buffer[1024];
if (recv(s, buffer, sizeof(buffer), 0) == SOCKET_ERROR) {
cout << "Error CR#001" << endl;
return -1;
}
if (strcmp(buffer, "||clear") == 0) {
system("cls");
return 1;
}
In c++ you can use std::string for the buffer:
const ssize_t MAX_BYTES = 1024;
ssize_t noreceivedbytes;
std::string buffer;
buffer.reserve(MAX_BYTES + 1);
noreceivedbytes = recv(s, buffer.data(), MAX_BYTES, 0)
if (noreceivedbytes <= 0) {
cout << "Error CR#001" << endl;
return -1;
}
buffer.data()[noreceivedbytes] = '\0';
if (buffer == "||clear") {
system("cls");
return 1;
}
Safer c solution for completeness:
#define MAX_BYTES = 1024;
ssize_t noreceivedbytes;
chat buffer[MAX_BYTES];
noreceivedbytes = recv(s, buffer, MAX_BYTES - 1, 0)
if (noreceivedbytes <= 0) {
cout << "Error CR#001" << endl;
return -1;
}
buffer[noreceivedbytes] = '\0';
if (strcmp(buffer, "||clear") == 0) {
system("cls");
return 1;
}
Please note:
This answer brings you only over the top of the iceberg. There are many more things that could go wrong when dealing with sockets (as mentioned by others in comments).
recv() doesn't guarantee that the whole chunk of data sent from the server will be read completely. You could easily end up with partial strings like "||cle" or "||c".
The least thing you'll need to do is to receive the bytes from the socket in a loop, until you have something at hand you can reasonably parse and match.
The simplest way to do so is to define a very primitive protocol, which preceeds payload data sent with it's size (take care of endianess problems when converting the size sent as integer value from the received data).
Having that at hand, you'll know exactly how many bytes you have to read until you have the payload chunk completed, such it can be parsed and compared reasonably.
How to do all that in detail exactly would lead to far to be answered here. There are whole books written about the topic (I'd recommend Stevens, "Unix network programming").
I am reading an Image URL sent from a Java client to a C++ server from Sockets. The server stops reading through recv() when it detects there is a null character in the char buffer[] as I do below in the following code:
void * SocketServer::clientController(void *obj)
{
// Retrieve client connection information
dataSocket *data = (dataSocket*) obj;
// Receive data from a client step by step and append data in String message
string message;
int bytes = 0;
do
{
char buffer[12] = {0};
bytes = recv(data->descriptor, buffer, 12, 0);
if (bytes > 0) // Build message
{
message.append(buffer, bytes);
cout << "Message: " << message << endl;
}
else // Error when receiving it
cout << "Error receiving image URL" << endl;
// Check if we are finished reading the image link
unsigned int i = 0;
bool finished = false;
while (i < sizeof(buffer) / sizeof(buffer[0]) && !finished)
{
finished = buffer[i] == '\0';
i++;
}
if (finished)
break;
}
while (bytes > 0);
cout << message << endl;
close(data->descriptor);
pthread_exit(NULL);
}
Is there a better and more elegant way to make this?
I read about sending first the size of the URL, but I do not know exactly how to stop recv() with it. I guess it is done by counting the bytes received until the size of the URL is reached. At that moment, we should be finished reading.
Another approach could be closing the Java socket so that recv() will return -1 and the loop will be finished. However, considering my Java client waits for a response from C++ server, closing the socket and then reopen it does not seem a suitable option.
Thank you,
Héctor
Apart from that your buffer has an unusual size (one typically chooses a power of 2, so 8, 16, 32, ...) and it looks a little small for your intent, your approach seems fine to me:
I assume that your java client will send a null terminated string and then wait anyway, i. e. especially it does not send any further data. So after you received the 0 character, there won't be any data to receive any more anyway, so there is not need to bother for something explicitly that recv does implicitly (recv normally returns only the data available, even if less than the buffer could consume).
Be aware that you initialized buffer with 0, so if you check the entire buffer (instead of the range [buffer, buffer + bytes), you might detect a false positive (if you receive less than 12 characters in the first iteration)! Detection of the 0 character can be done more elegantly, though, anyway:
if(std::find(buffer, buffer + bytes, 0) < buffer + bytes)
{
// found the 0 character!
break;
}
I am trying to implement a simple file transfer. Below here is two methods that i have been testing:
Method one: sending and receiving without splitting the file.
I hard coded the file size for easier testing.
sender:
send(sock,buffer,107,NULL); //sends a file with 107 size
receiver:
char * buffer = new char[107];
recv(sock_CONNECTION,buffer,107,0);
std::ofstream outfile (collector,std::ofstream::binary);
outfile.write (buffer,107);
The output is as expected, the file isn't corrupted because the .txt file that i sent contains the same content as the original.
Method two: sending and receiving by splitting the contents on receiver's side. 5 bytes each loop.
sender:
send(sock,buffer,107,NULL);
Receiver:
char * buffer = new char[107]; //total file buffer
char * ptr = new char[5]; //buffer
int var = 5;
int sizecpy = size; //orig size
while(size > var ){ //collect bytes
recv(sock_CONNECTION,ptr,5,0);
strcat(buffer,ptr); //concatenate
size= size-var; //decrease
std::cout<<"Transferring.."<<std::endl;
}
std::cout<<"did it reach here?"<<std::endl;
char*last = new char[size];
recv(sock_CONNECTION,last,2,0); //last two bytes
strcat(buffer,last);
std::ofstream outfile (collector,std::ofstream::binary);
outfile.write (buffer,107);
Output: The text file contains invalid characters especially at the beginning and the end.
Questions: How can i make method 2 work? The sizes are the same but they yield different results. the similarity of the original file and the new file on method 2 is about 98~99% while it's 100% on method one. What's the best method for transferring files?
What's the best method for transferring files?
Usually I'm not answering questions like What's the best method. But in this case it's obvious:
You sent the file size and a checksum in network byte order, when starting a transfer
Sent more header data (e.g filename) optionally
The client reads the file size and the checksum, and decodes it to host byte order
You sent the file's data in reasonably sized chunks (5 bytes isn't a reasonable size), chunks should match tcp/ip frames maximum available payload size
You receive chunk by chunk at the client side until the previously sent file size is matched
You calculate the checksum for the received data at the client side, and check if it matches the one that was received beforhand
Note: You don't need to combine all chunks in memory at the client side, but just append them to a file at a storage medium. Also the checksum (CRC) usually can be calculated from running through data chunks.
Disagree with Galik. Better not to use strcat, strncat, or anything but the intended output buffer.
TCP is knda fun. You never really know how much data you are going to get, but you will get it or an error.
This will read up to MAX bytes at a time. #define MAX to whatever you want.
std::unique_ptr<char[]> buffer (new char[size]);
int loc = 0; // where in buffer to write the next batch of data
int bytesread; //how much data was read? recv will return -1 on error
while(size > MAX)
{ //collect bytes
bytesread = recv(sock_CONNECTION,&buffer[loc],MAX,0);
if (bytesread < 0)
{
//handle error.
}
loc += bytesread;
size= size-bytesread; //decrease
std::cout<<"Transferring.."<<std::endl;
}
bytesread = recv(sock_CONNECTION,&buffer[loc],size,0);
if (bytesread < 0)
{
//handle error
}
std::ofstream outfile (collector,std::ofstream::binary);
outfile.write (buffer.get(),size);
Even more fun, write into the output buffer so you don't have to store the whole file. In this case MAX should be a bigger number.
std::ofstream outfile (collector,std::ofstream::binary);
char buffer[MAX];
int bytesread; //how much data was read? recv will return -1 on error
while(size)
{ //collect bytes
bytesread = recv(sock_CONNECTION,buffer,MAX>size?size:MAX,0);
// MAX>size?size:MAX is like a compact if-else: if (MAX>size){size}else{MAX}
if (bytesread < 0)
{
//handle error.
}
outfile.write (buffer,bytesread);
size -= bytesread; //decrease
std::cout<<"Transferring.."<<std::endl;
}
The initial problems I see are with std::strcat. You can't use it on an uninitialized buffer. Also you are not copying a null terminated c-string. You are copying a sized buffer. Better to use std::strncat for that:
char * buffer = new char[107]; //total file buffer
char * ptr = new char[5]; //buffer
int var = 5;
int sizecpy = size; //orig size
// initialize buffer
*buffer = '\0'; // add null terminator
while(size > var ){ //collect bytes
recv(sock_CONNECTION,ptr,5,0);
strncat(buffer, ptr, 5); // strncat only 5 chars
size= size-var; //decrease
std::cout<<"Transferring.."<<std::endl;
}
beyond that you should really as error checking so the sockets library can tell you if anything went wrong with the communication.
I am trying to send and receive large amounts of data at once in udp c++, with the following code. I can send at once just 16000 bits, char. How can one send/receive millions of bytes of data without closing the socket?
//sends the data contained in aliceBuf, which is, char of size 16000.
if (sendto(aliceSocket, aliceBuf, strlen(aliceBuf), 0, (struct sockaddr *)&bobAddr, sizeof (bobAddr)) == -1) {
perror("sendto");
exit(1);
}
// receiver code: it is receiving just 16000 char.
recvlen = recvfrom(aliceSocket, aliceBuf1, receiveBuffer, 0, (struct sockaddr*)&bobAddr, &bobAddrSize);
if (recvlen >= 0) {
aliceBuf1[recvlen] = 0; /* expect a printable string - terminate it */
}
You can send a large amount of data in one go, but the question you have to ask yourself is: How will the receiver know how much data to expect?
I normally handle these cases by either encoding the length explicitly by prefixing the data with the length and then the receiver loops until that amount of data has arrived, or by having some sort of end of data marker like 'C' strings or more implicitly like json data and the receiver loops looking for something in the data itself.
You wil have to add a protocol on top of UDP, just as if you were using TCP. I'm sorry that you have to do some work, but that's just how things are. Some of the datagrams may get lost, so you may have to add a layer for that too. 1M bits is ~twice as large as the largest possible UDP datagram anyway, so even if you reconfigure your network stack to allow larger datagrams, you will still hit the 64k limit, so requiring a protocol.
I did with loopiing like this:
int totalGoLength= no of blocks you want to send
int dataLengthOneGo = length of data in one block you want to send
//start loop
int iii=1 ;
while (iii <= totalGoLength){ //send by dividing into packets
////--SEND/by converting to char * for less memory occupation----
// theString has the string data to send
std::string part(theString.substr(0, dataLengthOneGo));
char * s4;
s4 = new char[part.size() + 1];
memcpy(s4, part.c_str(), part.size() + 1);
if (sendto(aliceSocket, s4, strlen(s4), 0, (struct sockaddr *)&bobAddr, sizeof (bobAddr)) == -1) {
perror("sendto");
exit(1);
}
delete [] s4;
////----------------------Receiving------------
// receive buffer should have sufficient memory allocation
char *aliceBuf1;
aliceBuf1 = new char[receiveBuffer];
recvlen = recvfrom(aliceSocket, aliceBuf1, receiveBuffer, 0, (struct sockaddr *)&bobAddr, &bobAddrSize);
if (recvlen >= 0) {
aliceBuf1[recvlen] = 0; /* expect a printable string - terminate it */
//convert char to string
string s1(aliceBuf1);
//erase the white space
s1.erase(remove_if(s1.begin(), s1.end(), isspace), s1.end());
//convert string into integer vector
std::vector<int> ints;
ints.reserve(s1.size());
std::transform(std::begin(s1), std::end(s1), std::back_inserter(ints), [](char c) {
return c - '0'; });
}
delete[] aliceBuf1;
justCopy=ints;
KeepData.insert(KeepData.end(),justCopy .begin(), justCopy.end());
justCopy.erase(justCopy.begin(), justCopy.end()); //erase for next time
ints.erase(ints.begin(), ints.end()); //erase for next time
theString.erase(theString.begin(), theString.begin() + dataLengthOneGo);//keep the remaining
iii=iii+1;
}//end of the while
i developed client server program using c++,so i want to receive more than 500kb , my client message is terminated with "!" ,so i want to receive until my last byte(!) receive ,
this is my code it doesn't work.what is wrong with it.
do
{
int num = recv(*csock, buf, bytesLeft,0);
if (num == 0)
{
break;
}
else if (num < 0 && errno != EINTR)
{
fprintf(stderr, "Exit %d\n", __LINE__);
exit(1);
}
else if (num > 0)
{
numRd += num;
buf += num;
bytesLeft -= num;
fprintf(stderr, "read %d bytes - remaining = %d\n", num, bytesLeft);
}
}
while (bytesLeft != 0);
fprintf(stderr, "read total of %d bytes\n", numRd);
While I'm not sure exactly what your problem is because of the wording of your question, you generally can't use strcat to append raw buffers received over the network unless you know specifically they will be NULL-terminated, and even then, that's not really "safe" in the event you get an unexpected data transmission. The assumption with c-strings is that they are NULL-terminated, but a raw network buffer may not be, and using strcat will cause you to over-run the input buffer should it not be NULL-terminated. Instead of strcat, use a known fixed-size buffer of size N bytes for receiving the data into, and increment a temporary pointer through the buffer until you reach the end of the buffer or the end of the packet transmission. That way you will always read from the network up to N bytes and no more, and prevent buffer over-run situations from occuring.
For instance, you can do the following (this is not the fastest or more efficient solution because of all the copying, but it works):
unsigned char buf[10000]; //10Kb fixed-size buffer
unsigned char buffer[MAXRECV]; //temporary buffer
unsigned char* temp_buf = buf;
unsigned char* end_buf = buf + sizeof(buf);
do
{
iByteCount = recv(GetSocketId(), buffer,MAXRECV,0);
if ( iByteCount > 0 )
{
//make sure we're not about to go over the end of the buffer
if (!((temp_buf + iByteCount) <= end_buf))
break;
fprintf(stderr, "Bytes received: %d\n",iByteCount);
memcpy(temp_buf, buffer, iByteCount);
temp_buf += iByteCount;
}
else if ( iByteCount == 0 )
{
if(temp_buf != buf)
{
//do process with received data
}
else
{
fprintf(stderr, "receive failed");
break;
}
}
else
{
fprintf(stderr, "recv failed: ");
break;
}
} while(iByteCount > 0 && temp_ptr < end_buf); //check for end of buffer
Do you need all 1MB+ of data in one contiguous byte buffer? If so, and you stick with that protocol that has a terminating '!' and does not have a header that includes the length, then you ar stuck with memcpy() and realloc() a lot or some other buffer type like std::vector which, really just does the same thing.
If you don't need all those bytes in one string, you can store them in some other way, eg. a vector of *buffer, and so avoid copying.
Assuming you are using a blocking socket (which is the default mode for sockets), then recv() will block waiting for the full MAXRECV number of bytes to arrive. If the client sends less than that number of bytes, recv() will block waiting for data that does not arrive.
To work around that, you need to either:
1) call recv() with a 1-byte buffer, calling recv() until you encounter your ! byte.
2) call select() before calling recv() to detect when the socket actually has data to read, then call ioctlsocket(FIONREAD) to determine how many bytes can actually be read with recv() without blocking, then have recv() read that number of bytes.