This is more of a request for confirmation than a question, so I'll keep it brief. (I am away from my PC and so can't simply implement this solution to test).
I'm writing a program to send an image file taken via webcam (along with meta data) from a raspberryPi to my PC.
I've worked out that the image is roughly around 130kb, the packet header is 12b and the associated meta data another 24b. Though I may increase the image size in future, once I have a working prototype.
At the moment I am not able to retrieve this whole packet successfully as, after sending it to the PC I only ever get approx 64kb recv'd in the buffer.
I have assumed that this is because for whatever reason the default buffer size for a socket declared like:
SOCKET sock = socket(PF_INET, SOCK_STREAM, 0);
is 64kb (please could someone clarify this if you're 'in the know')
So - to fix this problem I intend to increase the socket size to 1024kb via the setsockopt(x..) command.
Please could someone confirm that my diagnosis of the problem, and proposed solution are correct?
I ask this question as I am away form my PC right now and am unable to try it until I get back home.
This most likely has nothing to do with the socket buffers, but with the fact that recv() and send() do not have to receive and send all the data you want. Check the return value of those function calls, it indicates how many bytes have actually been sent and received.
The best way to deal with "short" reads/writes is to put them in a loop, like so:
char *buf; // pointer to your data
size_t len; // length of your data
int fd; // the socket filedescriptor
size_t offset = 0;
ssize_t result;
while (offset < len) {
result = send(fd, buf + offset, len - offset, 0);
if (result < 0) {
// Deal with errors here
}
offset += result;
}
Use a similar construction for receiving data. Note that one possible error condition is that the function call was interrupted (errno = EAGAIN or EWOULDBLOCK), in that case you should retry the send command, in all other cases you should exit the loop.
Related
I'm writing a C++ program. I need to receive a file and I'm using recv() function over a TCP socket to do that.
download_file() {
while (left_bytes != 0 && !connection_closed) {
if (left_bytes >= buffer_max_size)
bytes_to_download = buffer_max_size;
else
bytes_to_download = left_bytes;
if (request.conn->read_data(buffer, bytes_to_download))
{
left_bytes -= buffer->get_size();
temporary_file.write_data(buffer);
} else connection_closed = true;
}
}
read_data() {
while (bytes_received < size && alive_) {
bytes_read = recv(sock_, read_buffer, size, 0);
if (bytes_read == SOCKET_ERROR) {
delete[] local_buffer;
throw SocketException(WSAGetLastError());
}
// the connection is closed
if (bytes_read == 0) alive_ = false;
else {
bytes_received += bytes_read;
buffer->add(local_buffer, bytes_read);
}
}
}
The problem is that the recv never returns. It receives the whole file except for few KB and it freeze on the recv(). The buffer size is 1460.
I receive the file only if I print something to the console with cout every time the recv is called. Only in this case I receive the whole file.
Otherwise if I set as socket option the WAITALL and the client closes the connection after the file is sent, I receive the whole file.
Here's the code for the Client side that sends the file:
TransmitFile(file_request->connection_->get_handle_socket(), file_handler.get_file_handle(), file_request->file_size_, 65535, nullptr, nullptr, TF_USE_SYSTEM_THREAD)
EDIT
Here's how I send and read the file size between the Client and Server.
std::stringstream stream_;
stream_.str(std::string());
// append the file size
const __int64 file_size = htonll(GetFileSize(file_handle_, nullptr););
stream_ << ' ' << file_size << ' ';
Then I use the send to send this string
Here's how I read the file size
// Within stream_ there is all the content of the received packet
std::string message;
std::getline(stream_, message, ' ');
this->request_body_.file_size_ = ntohll(strtoll(message.c_str(), nullptr, 0));
EDIT
I cleaned up the code and I found out that read_data() is obviously called once and I was updating the buffer variable wrongly. Hence I was tracking the size of the content within the buffer in a wrong way which make me call the recv() once more.
First thing: recv() will block if there are no bytes left to read but the connection is still open. So whatever you might say about what your code is doing, that must be what is happening here.
That could be for any of the following reasons:
the sender lied about the size of the file, or did not send the promised number of bytes
the file size was not interpreted correctly at the receiving end for whatever reason
the logic that 'counts down' the number of bytes left in the receiver is somehow flawed
Trouble is, looking at the code samples you have posted, it's hard to say which because the code is a bit muddled and, in my eyes, more complicated than it needs to be. I'm going to recommend you sort that out.
Sending the size of the file.
Don't mess about sending this as a string. Send it instead in binary, using (say) htonll() at the sending end and ntohll() at the receiving end. Then, the receiver knows to read exactly 8 bytes to figure out what's coming next. It's hard to get that wrong.
Sending the file itself.
TransmitFile() looks to be a good choice here. Stick with it.
Receiving the file and counting down how many bytes are left.
Take a closer look at that code and consider rewriting it. It's a bit of a mess.
What to do if it still doesn't work.
Check with WireShark that the expected data is being sent and then walk through the code in the receiver in the debugger. There is absolutely no excuse for not doing this unless you don't have a debugger for some reason, in which case please say so and somebody will try to help you. The fact that logging to cout fixes your problems is a red-herring. That just changes the timing and then it just happens to work right.
That's all. Best of luck.
Currently, I'm learning how to build a transparent HTTP proxy in C++. There had two issues on the proxy client side I couldn't resolve for long time. Hope someone can point out the root causes based on following scenarios. Thanks a lot. :D
The HTTP proxy I built right now is somehow work partially only. For example, I could access google's main page through proxy while I couldn't get any search result after I typed keyword(the google instant is also not working at all). On the other hand, youtube is working perfectly includes searching, loading video and commenting. What's more, there also got some websites like yahoo even couldn't display main page after I keyed in its URL.
The reason why I said the issues are on the proxy client side at the begining is because I traced the data flow of my program. I found out the written size returned by socket programming function write() was smaller than the data size I passed to my write back function. The most weird observation for me was the data losing issue is independent from the size of data. The socket write() function could work properly for youtube video data which is nearly 2MB while it would loss data for google search request which is just 20KB.
Furthermore, there also had another situation that browser displayed blank when the data size I passed to my write back function and the written size returned by socket write function() are the same. I used wireshark to trace the flow of communication and compared mine with pure IP communication without proxy involved. I found out that browser didn't continuously send out HTTP requests after it received certain HTTP responses comparing with pure IP communication flow. I couldn't find out why the browser didn't send out rest of HTTP requests.
Following is my code for write back function:
void Proxy::get_data(char* buffer, size_t length)
{
cout<<"Length:"<<length<<endl;
int connfd;
size_t ret;
// get connfd from buffer
memset(&connfd, 0, sizeof(int));
memcpy(&connfd, buffer, sizeof(int));
cout<<"Get Connection FD:"<<connfd<<endl;
// get receive data size
size_t rData_length = length-sizeof(int);
cout<<"Data Size:"<<rData_length<<endl;
// create receive buffer
char* rBuf = new char[rData_length];
// allocate memory to receive buffer
memset(rBuf, 0, rData_length);
// copy data to buffer
memcpy(rBuf, buffer+sizeof(int), rData_length);
ret = write(connfd, rBuf, rData_length);
if(ret < 0)
{
cout<< "received data failed"<< endl;
close(connfd);
delete[] rBuf;
exit(1);
}
else
{
printf("Write Data[%d] to Socket\n", ret);
}
close(connfd);
delete[] rBuf;
}
May be you could try this
int curr = 0;
while( curr < rData_length ) {
ret = write( connfd, rBuf + curr, rData_length - curr );
if( ret == -1 ) { /* ERROR */ }
else
curr += ret;
}
instead of
ret = write(connfd, rBuf, rData_length);
In general, the number of bytes written by write() could differ from what you ask to write. You should better read some manual. Say, http://linux.die.net/man/2/write
Copying bytes between an input socket and an output socket is much simpler than this. You don't need to dynamically allocate buffers according to how much data was read by the last read. You just need to read into a char[] array and write from that array to the target, taking due account of the length value returned by the read.
I'm having a problem with unix local sockets. While reading a message that's longer than my temp buffer size, the request takes too long (maybe indefinitely).
Added after some tests:
there is still problem with freeze at ::recv. when I send (1023*8) bytes or less to the UNIX socket - all ok, but when sended more than (1023*9) - i get freeze on recv command.
maybe its FreeBSD default UNIX socket limit or C++ default socket settings? Who know?
i made some additational tests and I am 100% sure that its "freeze" on the last 9th itteration when executing ::recv command, when trying to read message >= (1023*9) bytes long. (first 8th itterationg going well.)
What I'm doing:
The idea is to read in a do/while loop from a socket with
::recv (current_socket, buf, 1024, 0);
and check buf for a SPECIAL SYMBOL. If not found:
merge content of buffer to stringxxx += buf;
bzero temp buf
continue the ::recv loop
How do I fix the issue with the request taking too long in the while loop?
Is there a better way to clear the buffer? Currently, it's:
char buf [1025];
bzero(buf, 1025);
But I know bzero is deprecated in the new c++ standard.
EDIT:
*"Why need to clean the buffer*
I see questions at comments with this question. Without buffer cleanup on the next(last) itteration of reading to the buffer, it will contain the "tail" of first part of the message.
Example:
// message at the socket is "AAAAAACDE"
char buf [6];
::recv (current_socket, buf, 6, 0); // read 6 symbols, buf = "AAAAAA"
// no cleanup, read the last part of the message with recv
::recv (current_socket, buf, 6, 0);
// read 6 symbols, but buffer contain only 3 not readed before symbols, therefore
// buf now contain "CDEAAA" (not correct, we waiting for CDE only)
When your recv() enters an infinite loop, this probably means that it's not making any progress whatsoever on the iterations (i.e., you're always getting a short read of zero size immediately, so your loop never exits, because you're not getting any data). For stream sockets, a recv() of zero size means that the remote end has disconnected (it's something like read()ing from a file when the input is positioned at EOF also gets you zero bytes), or at least that it has shut down the sending channel (that's for TCP specifically).
Check whether your PHP script is actually sending the amount of data you claim it sends.
To add a small (non-sensical) example for properly using recv() in a loop:
char buf[1024];
std::string data;
while( data.size() < 10000 ) { // what you wish to receive
::ssize_t rcvd = ::recv(fd, buf, sizeof(buf), 0);
if( rcvd < 0 ) {
std::cout << "Failed to receive\n"; // Receive failed - something broke, see errno.
std::abort();
} else if( !rcvd ) {
break; // No data to receive, remote end closed connection, so quit.
} else {
data.append(buf, rcvd); // Received into buffer, attach to data buffer.
}
}
if( data.size() < 10000 ) {
std::cout << "Short receive, sender broken\n";
std::abort();
}
// Do something with the buffer data.
Instead of bzero, you can just use
memset(buf, 0, 1025);
These are 2 separate issues. The long time is probably some infinite loop due to a bug in your code and has nothing to do with the way you clear your buffer. As a matter of fact you shouldn't need to clear the buffer; receive returns the number of bytes read, so you can scan the buffer for your SPECIAL_SYMBOL up to that point.
If you paste the code maybe I can help. more.
Just to clarify: bzero is not deprecated in C++ 11. Rather, it's never been part of any C or C++ standard. C started out with memset 20+ years ago. For C++, you might consider using std::fill_n instead (or just using std::vector, which can zero-fill automatically). Then again, I'm not sure there's a good reason to zero-fill the buffer in this case at all.
I'm writing a UDP client that sends a string to a server, when the server sends back several packets, the behavior of the program is not as my expectation. I want to process any incoming packet by process() one by one until the entry buffer gets empty, but I think there is a problem related to blocking behavior of recv.
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <winsock.h>
using namespace std;
void process(const char *in, int size)
{
fprintf(stdout, "%s\n", in);
}
int main()
{
char quack_addr[] = "127.0.0.1";
unsigned short quack_port = 9091;
WSAData data;
WSAStartup(MAKEWORD(2, 2), &data);
sockaddr_in qserver;
qserver.sin_family = AF_INET;
qserver.sin_addr.s_addr = inet_addr(quack_addr);
qserver.sin_port = htons(quack_port);
SOCKET client = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
if (client <= 0)
{
fprintf(stderr, "Error - Can not create socket.\n");
exit(1);
}
while (true)
{
const int MAX = 1024;
char sbuf[MAX];
char rbuf[MAX];
fprintf(stdout, ": ");
fgets(sbuf, MAX, stdin);
int slen = strlen(sbuf);
int r = sendto(client,sbuf,slen,0,(sockaddr*)&qserver,sizeof(qserver));
// Current code:
// int rlen = recv(client, rbuf, MAX, 0);
// if (rlen > 0)
// {
// rbuf[rlen] = 0;
// process(rbuf, rlen);
// }
// Question starts here:
//
// While there is data in queue do:
// {
// (break if there is no data)
// int rlen = recv(client, rbuf, MAX, 0);
// rbuf[rlen] = 0;
// process(rbuf, rlen);
// }
}
return 0;
}
How can I check if the buffer is empty or not, before calling recv(...) ?
The problem occurs in this scenario:
User is typing a command in the client program (cmd1).
Simultaneously, the server sends 3 packets to client (pkt1, pkt2, pkt3).
After pressing Enter in the client side, I expect to receive those 3 packets and probably the result corresponding to cmd1, and process() all of them one by one.
But after pressing Enter in stage 3, I receive pkt1! and after sending another command to the server I will receive pkt2 and so on ...!
I know my code is not enough to handle this issue, so, my question is how to handle it?
Note: I'm using netcat -L -p 9091 -u as UDP server
I think the problems (unsatisfying behavior you do not describe) come from a different source. Let me just list some ideas and comments c./ what was said before:
(1) recvfrom() blocks too. However, you want to use it. Your communication currently sends and receives from loopback, which is fine for your toy program (but: see below). When receiving UDP data, with recv() you don't know who sent it, as the socket was never connect()ed. Use recvfrom() to prepare yourself for some minimal error checking in a more serious program
(2) as select() suspends the program to i/o availibity, it would only put any issue with your socket blocking to a different level. But this is not the problem
(3) to check is the receive buffer is empty, use flag MSG_PEEK in recvfrom() in an appropriate position. It's usually only used to deal with scarce memory, but it should do the job.
(4) reason 1 why I believe you see the issues you don't describe in more detail:
UDP datagrams preserve message boundaries. This means that recvfrom() will read in an entire chunk of data making up any message sent. However, if the buffer you read this into is smaller than the data read, any surplus will be silently discarded. So make sure you have a big buffer (65k something ideally).
(5) reason 2:
You receive any data sent to the loopback. If you're currently also connected to some net (sat, the Internet), what you catch might actually be from a different source than you expect. So at least in a resting phase, disconnect.
Blocking shouldn't be an issue. Your basic logic, when coded cleanly, is:
Recvfrom() (block/wait until ready)
Process
Peek if buffer empty
Exit if yes
Loop back to receive more if not,
and you seem to want to do this currently. As you don't multi-thread, optimize fie perfiormance, or similar, you shouldn't care about blocking. If you find your receive buffer too small, increase its size using
Setsockopt() for optName SO_RCVBUF
Use select() (with a suitable timeout) to check for incoming data prior to calling recv().
Something like the following (nonportable code)
#include <winsock2.h>
...
/* somewhere after your sendto, or your first recv */
fd_set recv_set;
timeval tv = {1, 0}; /* one second */
FD_ZERO(&recv_set);
FD_SET(client, &recv_set);
while (select(0, &recv_set, NULL, NULL, &tv) > 0)
{
/* recv... */
FD_SET(client, &recv_set); /* actually redundant, since it is already set */
}
iPhone sometimes bugs out and doesn't let me post comments. Thanks, Steve. This is just continuing the conversation.
I assume this means 'uncomment to 'question starts here'. Partial answer, as this still depends on my 2nd comment; this is more or less than what to expect. Assuming the three messages to be sent by the server are already queued up, after you hit enter for the first time, your packet is sent (never blocked as sendto() doesn't block for UDP), received by the server and (I assume, see above, echoed back and added to the FIFO receive buffer in which you alread have three messages queued up. You then have a recv() in your program which receives the first queued up message, printing it out. Your current logic goes back to top of loop, expect another input and waits for it (so this isn't blocked on a socket level, but as your program requests input, eg simply 'enter'), then comes to the second originally sent message (by the server) and processes that one. One more loop, and all three are done. Hitting enter again, and assuming the server echoes what you sent, you should start receiving your typed in messages (which might be empty if you only hit enter). The loop will currently not exit except you kill it.
I am using read function to read data from a socket, but when the data is more than 4k, read function just read part of the data, for example, less than 4k. Here is the key code:
mSockFD = socket(AF_INET, SOCK_STREAM, 0);
if (connect(mSockFD, (const sockaddr*)(&mSockAdd), sizeof(mSockAdd)) < 0)
{
cerr << "Error connecting in Crawl" << endl;
perror("");
return false;
}
n = write(mSockFD, httpReq.c_str(), httpReq.length());
bzero(mBuffer, BUFSIZE);
n = read(mSockFD, mBuffer, BUFSIZE);
Note than BUFSIZE is much larger than 4k.
When data is just a few hundred bytes, read function works as expected.
This is by design and to be expected.
The short answer to your question is you should continue calling "read" until you get all the data you expect. That is:
int total_bytes = 0;
int expected = BUFSIZE;
int bytes_read;
char *buffer = malloc(BUFSIZE+1); // +1 for null at the end
while (total_bytes < expected)
{
int bytes_read = read(mSockFD, buffer+total_bytes, BUFSIZE-total_bytes);
if (bytes_read <= 0)
break;
total_bytes += bytes_read;
}
buffer[total_bytes] = 0; // null terminate - good for debugging as a string
From my experience, one of the biggest misconceptions (resulting in bugs) that you'll receive as much data as you ask for. I've seen shipping code in real products written with the expectation that sockets work this way (and no one certain as to why it doesn't work reliably).
When the other side sends N bytes, you might get lucky and receive it all at once. But you should plan for receiving N bytes spread out across multiple recv calls. With the exception of a real network error, you'll eventually get all N bytes. Segmentation, fragmentation, TCP window size, MTU, and the socket layer's data chunking scheme are the reasons for all of this. When partial data is received, the TCP layer doesn't know about how much more is yet to come. It just passes what it has up to the app. It's up to the app to decide if it got enough.
Likewise, "send" calls can get conglomerated into the same packet together.
There may be ioctls and such that will make a socket block until all the expected data is received. But I don't know of any off hand.
Also, don't use read and write for sockets. Use recv and send.
Read this book. It will change your life with regards to sockets and TCP: