I'm having a problem with unix local sockets. While reading a message that's longer than my temp buffer size, the request takes too long (maybe indefinitely).
Added after some tests:
there is still problem with freeze at ::recv. when I send (1023*8) bytes or less to the UNIX socket - all ok, but when sended more than (1023*9) - i get freeze on recv command.
maybe its FreeBSD default UNIX socket limit or C++ default socket settings? Who know?
i made some additational tests and I am 100% sure that its "freeze" on the last 9th itteration when executing ::recv command, when trying to read message >= (1023*9) bytes long. (first 8th itterationg going well.)
What I'm doing:
The idea is to read in a do/while loop from a socket with
::recv (current_socket, buf, 1024, 0);
and check buf for a SPECIAL SYMBOL. If not found:
merge content of buffer to stringxxx += buf;
bzero temp buf
continue the ::recv loop
How do I fix the issue with the request taking too long in the while loop?
Is there a better way to clear the buffer? Currently, it's:
char buf [1025];
bzero(buf, 1025);
But I know bzero is deprecated in the new c++ standard.
EDIT:
*"Why need to clean the buffer*
I see questions at comments with this question. Without buffer cleanup on the next(last) itteration of reading to the buffer, it will contain the "tail" of first part of the message.
Example:
// message at the socket is "AAAAAACDE"
char buf [6];
::recv (current_socket, buf, 6, 0); // read 6 symbols, buf = "AAAAAA"
// no cleanup, read the last part of the message with recv
::recv (current_socket, buf, 6, 0);
// read 6 symbols, but buffer contain only 3 not readed before symbols, therefore
// buf now contain "CDEAAA" (not correct, we waiting for CDE only)
When your recv() enters an infinite loop, this probably means that it's not making any progress whatsoever on the iterations (i.e., you're always getting a short read of zero size immediately, so your loop never exits, because you're not getting any data). For stream sockets, a recv() of zero size means that the remote end has disconnected (it's something like read()ing from a file when the input is positioned at EOF also gets you zero bytes), or at least that it has shut down the sending channel (that's for TCP specifically).
Check whether your PHP script is actually sending the amount of data you claim it sends.
To add a small (non-sensical) example for properly using recv() in a loop:
char buf[1024];
std::string data;
while( data.size() < 10000 ) { // what you wish to receive
::ssize_t rcvd = ::recv(fd, buf, sizeof(buf), 0);
if( rcvd < 0 ) {
std::cout << "Failed to receive\n"; // Receive failed - something broke, see errno.
std::abort();
} else if( !rcvd ) {
break; // No data to receive, remote end closed connection, so quit.
} else {
data.append(buf, rcvd); // Received into buffer, attach to data buffer.
}
}
if( data.size() < 10000 ) {
std::cout << "Short receive, sender broken\n";
std::abort();
}
// Do something with the buffer data.
Instead of bzero, you can just use
memset(buf, 0, 1025);
These are 2 separate issues. The long time is probably some infinite loop due to a bug in your code and has nothing to do with the way you clear your buffer. As a matter of fact you shouldn't need to clear the buffer; receive returns the number of bytes read, so you can scan the buffer for your SPECIAL_SYMBOL up to that point.
If you paste the code maybe I can help. more.
Just to clarify: bzero is not deprecated in C++ 11. Rather, it's never been part of any C or C++ standard. C started out with memset 20+ years ago. For C++, you might consider using std::fill_n instead (or just using std::vector, which can zero-fill automatically). Then again, I'm not sure there's a good reason to zero-fill the buffer in this case at all.
Related
I know C++ for quite long, but started using it for my purposes some what a year and a half ago.
I started learning network programming on C++ and the first networking project is "File Transfering between hosts over TCP/IP" which sounds kind easy but I am stuck with sending data.
I am trying to send small buffer less than 4KB, so buffer[4096] works fine for me, but I am planning to expand this. WSAStartup(), socket(), bind(), listen(), accept() functions work fine and values for them are initialised for both Server and Client, but I am dealing with other problems, maybe recv(), send() etc.
I still couldn't find the source of the problem.
Also it would be a ton helpful if somebody give me an example of transfering files over TCP/IP, but not in one packet, I want the file to be chunked and sent in parts or as it's called "ring model", but I couldn't find a working model;
P.S. This is first time I am asking here, pls give feedback about how well all of this is written, so that I could write more informative for community help, thanks)
Server
char* buffer = new char[4096];
ZeroMemory(buffer, sizeof(buffer));
ofstream file("a.txt", ios::binary);
int err = recv(conn, buffer, sizeof(buffer), 0);
file << buffer;
file.close();
if (err == 0)
{
printf("Client diconnected...\n");
}
printf("Quitting...\n");
delete[] buffer;
Client
ifstream file("a.txt", ios::binary);
file.seekg(0, ios::end);
int size = file.tellg();
file.seekg(0, ios::beg);
char* buffer = new char[size];
file.read(buffer, size);
file.close();
int err = send(client, buffer, size, 0);
if (err == 0)
{
printf("Disconnecting...\n");
}
printf("Quitting...\n");
delete[] buffer;
"a.txt" file on Client side is 45 bytes in here are 45 * 'a'
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
And this is what I get on Server side, file size is 14 bytes
aaaaaaaa ’pÈ/
In C++, sizeof(buffer) is the size of the pointer type.
You may want to read up on more modern (as in after 1998) C++. We have std::vector nowadays, and that has a convenient size method. It would return 4096 for your buffer. Also, vector handles new[] and delete[] for you.
The fact that you get 8 "a"'s suggests that you built for x64. The remaining bytes are garbage; you should check how many bytes recv actually wrote to buffer. You cannot assume that you got all the bytes you asked for (whether that's 8 or 4096).
I believe that sizeof(buffer) in this line -->
int err = recv(conn, buffer, sizeof(buffer), 0);
will return sizeof(char*) which is 4 bytes in a 32 bit program or 8 bytes in a 64 bits program instead of 4096 because it is not a static array as in you did not declare it as char buffer[4096]. So, either declare it as char buffer[4096] or convert the above code to
int err = recv(conn, buffer, 4096, 0);
Two additional points:
TCP is a streaming protocol (not "message based"), so there's no guarantee that a single recv() will get everything sent in a single send().
The server line file << buffer; assumes buffer is zero terminated.
MSDN state that :
If no error occurs, recv returns the number of bytes received and the
buffer pointed to by the buf parameter will contain this data
received. If the connection has been gracefully closed, the return
value is zero.
Otherwise, a value of SOCKET_ERROR is returned, and a specific error
code can be retrieved by calling WSAGetLastError.
https://learn.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-recv
test if you actually read 45 bytes and check if there's an error (WSAGetLastError function)
I'm writing a C++ program. I need to receive a file and I'm using recv() function over a TCP socket to do that.
download_file() {
while (left_bytes != 0 && !connection_closed) {
if (left_bytes >= buffer_max_size)
bytes_to_download = buffer_max_size;
else
bytes_to_download = left_bytes;
if (request.conn->read_data(buffer, bytes_to_download))
{
left_bytes -= buffer->get_size();
temporary_file.write_data(buffer);
} else connection_closed = true;
}
}
read_data() {
while (bytes_received < size && alive_) {
bytes_read = recv(sock_, read_buffer, size, 0);
if (bytes_read == SOCKET_ERROR) {
delete[] local_buffer;
throw SocketException(WSAGetLastError());
}
// the connection is closed
if (bytes_read == 0) alive_ = false;
else {
bytes_received += bytes_read;
buffer->add(local_buffer, bytes_read);
}
}
}
The problem is that the recv never returns. It receives the whole file except for few KB and it freeze on the recv(). The buffer size is 1460.
I receive the file only if I print something to the console with cout every time the recv is called. Only in this case I receive the whole file.
Otherwise if I set as socket option the WAITALL and the client closes the connection after the file is sent, I receive the whole file.
Here's the code for the Client side that sends the file:
TransmitFile(file_request->connection_->get_handle_socket(), file_handler.get_file_handle(), file_request->file_size_, 65535, nullptr, nullptr, TF_USE_SYSTEM_THREAD)
EDIT
Here's how I send and read the file size between the Client and Server.
std::stringstream stream_;
stream_.str(std::string());
// append the file size
const __int64 file_size = htonll(GetFileSize(file_handle_, nullptr););
stream_ << ' ' << file_size << ' ';
Then I use the send to send this string
Here's how I read the file size
// Within stream_ there is all the content of the received packet
std::string message;
std::getline(stream_, message, ' ');
this->request_body_.file_size_ = ntohll(strtoll(message.c_str(), nullptr, 0));
EDIT
I cleaned up the code and I found out that read_data() is obviously called once and I was updating the buffer variable wrongly. Hence I was tracking the size of the content within the buffer in a wrong way which make me call the recv() once more.
First thing: recv() will block if there are no bytes left to read but the connection is still open. So whatever you might say about what your code is doing, that must be what is happening here.
That could be for any of the following reasons:
the sender lied about the size of the file, or did not send the promised number of bytes
the file size was not interpreted correctly at the receiving end for whatever reason
the logic that 'counts down' the number of bytes left in the receiver is somehow flawed
Trouble is, looking at the code samples you have posted, it's hard to say which because the code is a bit muddled and, in my eyes, more complicated than it needs to be. I'm going to recommend you sort that out.
Sending the size of the file.
Don't mess about sending this as a string. Send it instead in binary, using (say) htonll() at the sending end and ntohll() at the receiving end. Then, the receiver knows to read exactly 8 bytes to figure out what's coming next. It's hard to get that wrong.
Sending the file itself.
TransmitFile() looks to be a good choice here. Stick with it.
Receiving the file and counting down how many bytes are left.
Take a closer look at that code and consider rewriting it. It's a bit of a mess.
What to do if it still doesn't work.
Check with WireShark that the expected data is being sent and then walk through the code in the receiver in the debugger. There is absolutely no excuse for not doing this unless you don't have a debugger for some reason, in which case please say so and somebody will try to help you. The fact that logging to cout fixes your problems is a red-herring. That just changes the timing and then it just happens to work right.
That's all. Best of luck.
This is more of a request for confirmation than a question, so I'll keep it brief. (I am away from my PC and so can't simply implement this solution to test).
I'm writing a program to send an image file taken via webcam (along with meta data) from a raspberryPi to my PC.
I've worked out that the image is roughly around 130kb, the packet header is 12b and the associated meta data another 24b. Though I may increase the image size in future, once I have a working prototype.
At the moment I am not able to retrieve this whole packet successfully as, after sending it to the PC I only ever get approx 64kb recv'd in the buffer.
I have assumed that this is because for whatever reason the default buffer size for a socket declared like:
SOCKET sock = socket(PF_INET, SOCK_STREAM, 0);
is 64kb (please could someone clarify this if you're 'in the know')
So - to fix this problem I intend to increase the socket size to 1024kb via the setsockopt(x..) command.
Please could someone confirm that my diagnosis of the problem, and proposed solution are correct?
I ask this question as I am away form my PC right now and am unable to try it until I get back home.
This most likely has nothing to do with the socket buffers, but with the fact that recv() and send() do not have to receive and send all the data you want. Check the return value of those function calls, it indicates how many bytes have actually been sent and received.
The best way to deal with "short" reads/writes is to put them in a loop, like so:
char *buf; // pointer to your data
size_t len; // length of your data
int fd; // the socket filedescriptor
size_t offset = 0;
ssize_t result;
while (offset < len) {
result = send(fd, buf + offset, len - offset, 0);
if (result < 0) {
// Deal with errors here
}
offset += result;
}
Use a similar construction for receiving data. Note that one possible error condition is that the function call was interrupted (errno = EAGAIN or EWOULDBLOCK), in that case you should retry the send command, in all other cases you should exit the loop.
First question: I am confused between Buffers in TCP. I am trying to explain my proble, i read this documentation TCP Buffer, author said a lot about TCP Buffer, thats fine and a really good explanation for a beginner. What i need to know is this TCP Buffer is same buffer with the one we use in our basic client server program (Char *buffer[Some_Size]) or its some different buffer hold by TCP internally ?
My second question is that i am sending a string data with prefix length (This is data From me) from client over socket to server, when i print my data at console along with my string it prints some garbage value also like this "This is data From me zzzzzz 1/2 1/2....." ?. However i fixed it by right shifting char *recvbuf = new char[nlength>>3]; nlength to 3 bits but why i need to do it in this way ?
My third question is in relevance with first question if there is nothing like TCP Buffer and its only about the Char *buffer[some_size] then whats the difference my program will notice using such static memory allocation buffer and by using dynamic memory allocation buffer using char *recvbuf = new char[nlength];. In short which is best and why ?
Client Code
int bytesSent;
int bytesRecv = SOCKET_ERROR;
char sendbuf[200] = "This is data From me";
int nBytes = 200, nLeft, idx;
nLeft = nBytes;
idx = 0;
uint32_t varSize = strlen (sendbuf);
bytesSent = send(ConnectSocket,(char*)&varSize, 4, 0);
assert (bytesSent == sizeof (uint32_t));
std::cout<<"length information is in:"<<bytesSent<<"bytes"<<std::endl;
// code to make sure all data has been sent
while (nLeft > 0)
{
bytesSent = send(ConnectSocket, &sendbuf[idx], nLeft, 0);
if (bytesSent == SOCKET_ERROR)
{
std::cerr<<"send() error: " << WSAGetLastError() <<std::endl;
break;
}
nLeft -= bytesSent;
idx += bytesSent;
}
std::cout<<"Client: Bytes sent:"<< bytesSent;
Server code:
int bytesSent;
char sendbuf[200] = "This string is a test data from server";
int bytesRecv;
int idx = 0;
uint32_t nlength;
int length_received = recv(m_socket,(char*)&nlength, 4, 0);//Data length info
char *recvbuf = new char[nlength];//dynamic memory allocation based on data length info
//code to make sure all data has been received
while (nlength > 0)
{
bytesRecv = recv(m_socket, &recvbuf[idx], nlength, 0);
if (bytesRecv == SOCKET_ERROR)
{
std::cerr<<"recv() error: " << WSAGetLastError() <<std::endl;
break;
}
idx += bytesRecv;
nlength -= bytesRecv;
}
cout<<"Server: Received complete data is:"<< recvbuf<<std::endl;
cout<<"Server: Received bytes are"<<bytesRecv<<std::endl;
WSACleanup();
system("pause");
delete[] recvbuf;
return 0;
}
You send 200 bytes from the client, unconditionally, but in the server you only receive the actual length of the string, and that length does not include the string terminator.
So first of all you don't receive all data that was sent (which means you will fill up the system buffers), and then you don't terminate the string properly (which leads to "garbage" output when trying to print the string).
To fix this, in the client only send the actual length of the string (the value of varSize), and in the receiving server allocate one more character for the terminator, which you of course needs to add.
First question: I am confused between Buffers in TCP. I am trying to
explain my proble, i read this documentation TCP Buffer, author said a
lot about TCP Buffer, thats fine and a really good explanation for a
beginner. What i need to know is this TCP Buffer is same buffer with
the one we use in our basic client server program (Char
*buffer[Some_Size]) or its some different buffer hold by TCP internally ?
When you call send(), the TCP stack will copy some of the bytes out of your char array into an in-kernel buffer, and send() will return the number of bytes that it copied. The TCP stack will then handle the transmission of those in-kernel bytes to its destination across the network as quickly as it can. It's important to note that send()'s return value is not guaranteed to be the same as the number of bytes you specified in the length argument you passed to it; it could be less. It's also important to note that sends()'s return value does not imply that that many bytes have arrived at the receiving program; rather it only indicates the number of bytes that the kernel has accepted from you and will try to deliver.
Likewise, recv() merely copies some bytes from an in-kernel buffer to the array you specify, and then drops them from the in-kernel buffer. Again, the number of bytes copied may be less than the number you asked for, and generally will be different from the number of bytes passed by the sender on any particular call of send(). (E.g if the sender called send() and his send() returned 1000, that might result in you calling recv() twice and having recv() return 500 each time, or recv() might return 250 four times, or (1, 990, 9), or any other combination you can think of that eventually adds up to 1000)
My second question is that i am sending a string data with prefix
length (This is data From me) from client over socket to server, when
i print my data at console along with my string it prints some garbage
value also like this "This is data From me zzzzzz 1/2 1/2....." ?.
However i fixed it by right shifting char *recvbuf = new
char[nlength>>3]; nlength to 3 bits but why i need to it in this way ?
Like Joachim said, this happens because C strings depend on the presence of a NUL-terminator byte (i.e. a zero byte) to indicate their end. You are receiving strlen(sendbuf) bytes, and the value returned by strlen() does not include the NUL byte. When the receiver's string-printing routine tries to print the string, it keeps printing until if finds a NUL byte (by chance) somewhere later on in memory; in the meantime, you get to see all the random bytes that are in memory before that point. To fix the problem, either increase your sent-bytes counter to (strlen(sendbuf)+1), so that the NUL terminator byte gets received as well, or alternatively have your receiver manually place the NUL byte at the end of the string after it has received all of the bytes of the string. Either way is acceptable (the latter way might be slightly preferable as that way the receiver isn't depending on the sender to do the right thing).
Note that if your sender is going to always send 200 bytes rather than just the number of bytes in the string, then your receiver will need to always receive 200 bytes if it wants to receive more than one block; otherwise when it tries to receive the next block it will first get all the extra bytes (after the string) before it gets the next block's send-length field.
My third question is in relevance with first question if there is
nothing like TCP Buffer and its only about the Char *buffer[some_size]
then whats the difference my program will notice using such static
memory allocation buffer and by using dynamic memory allocation buffer
using char *recvbuf = new char[nlength];. In short which is best and
why ?
In terms of performance, it makes no difference at all. send() and receive() don't care a bit whether the pointers you pass to them point at the heap or the stack.
In terms of design, there are some tradeoffs: if you use new, there is a chance that you can leak memory if you don't always call delete[] when you're done with the buffer. (This can particularly happen when exceptions are thrown, or when error paths are taken). Placing the buffer on the stack, on the other hand, is guaranteed not to leak memory, but the amount of space available on the stack is finite so a really huge array could cause your program to run out of stack space and crash. In this case, a single 200-byte array on the stack is no problem, so that's what I would use.
I'm writing a distributed system in c++ using TCP/IP and sockets.
For each of my messages, I need to receive the first 5 bytes to know the full length of the incoming message.
What's the best way to do this?
recv() only 5 bytes, then recv() again. if I choose this, would it be safe to assume I'll get 0 or 5 bytes in the recv (aka not write a loop to keep trying)?
use MSG_PEEK
recv() some larger buffer size, then read the first 5 bytes and allocate the final buffer then.
You don't need to know anything. TCP is a stream protocol, and at any given moment you can get as little as one byte, or as much as multiple megabytes of data. The correct and only way to use a TCP socket is to read in a loop.
char buf[4096]; // or whatever
std::deque<char> data;
for (int res ; ; )
{
res = recv(fd, buf, sizeof buf, MSG_DONTWAIT);
if (res == -1)
{
if (errno == EAGAIN || errno == EWOULDBLOCK)
{
break; // done reading
}
else
{
// error, break, die
}
}
if (res == 0)
{
// socket closed, finalise, break
}
else
{
data.insert(data.end(), buf, buf + res);
}
}
The only purpose of the loop is to transfer data from the socket buffer to your application. Your application must then decide independently if there's enough data in the queue to attempt extraction of some sort of higher-level application message.
For example, in your case you would check if the queue's size is at least 5, then inspect the first five bytes, and then check if the queue holds a complete application message. If no, you abort, and if yes, you extract the entire message and pop if off from the front of the queue.
Use a state machine with two states:
First State.
Receive bytes as they arrive into a buffer. When there are 5 or more bytes perform your check on those first 5 and possibly process the rest of the buffer. Switch to the second state.
Second State.
Receive and process bytes as they arrive to the end of the message.
to answer your question specifically:
it's not safe to assume you'll get 0 or 5. it is possible to get 1-4 as well. loop until you get 5 or an error as others have suggested.
i wouldn't bother with PEEK, most of the time you'll block (assuming blocking calls) or get 5 so skip the extra call into the stack.
this is fine too but adds complexity for little gain.