I'm writing a UDP client that sends a string to a server, when the server sends back several packets, the behavior of the program is not as my expectation. I want to process any incoming packet by process() one by one until the entry buffer gets empty, but I think there is a problem related to blocking behavior of recv.
#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <winsock.h>
using namespace std;
void process(const char *in, int size)
{
fprintf(stdout, "%s\n", in);
}
int main()
{
char quack_addr[] = "127.0.0.1";
unsigned short quack_port = 9091;
WSAData data;
WSAStartup(MAKEWORD(2, 2), &data);
sockaddr_in qserver;
qserver.sin_family = AF_INET;
qserver.sin_addr.s_addr = inet_addr(quack_addr);
qserver.sin_port = htons(quack_port);
SOCKET client = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
if (client <= 0)
{
fprintf(stderr, "Error - Can not create socket.\n");
exit(1);
}
while (true)
{
const int MAX = 1024;
char sbuf[MAX];
char rbuf[MAX];
fprintf(stdout, ": ");
fgets(sbuf, MAX, stdin);
int slen = strlen(sbuf);
int r = sendto(client,sbuf,slen,0,(sockaddr*)&qserver,sizeof(qserver));
// Current code:
// int rlen = recv(client, rbuf, MAX, 0);
// if (rlen > 0)
// {
// rbuf[rlen] = 0;
// process(rbuf, rlen);
// }
// Question starts here:
//
// While there is data in queue do:
// {
// (break if there is no data)
// int rlen = recv(client, rbuf, MAX, 0);
// rbuf[rlen] = 0;
// process(rbuf, rlen);
// }
}
return 0;
}
How can I check if the buffer is empty or not, before calling recv(...) ?
The problem occurs in this scenario:
User is typing a command in the client program (cmd1).
Simultaneously, the server sends 3 packets to client (pkt1, pkt2, pkt3).
After pressing Enter in the client side, I expect to receive those 3 packets and probably the result corresponding to cmd1, and process() all of them one by one.
But after pressing Enter in stage 3, I receive pkt1! and after sending another command to the server I will receive pkt2 and so on ...!
I know my code is not enough to handle this issue, so, my question is how to handle it?
Note: I'm using netcat -L -p 9091 -u as UDP server
I think the problems (unsatisfying behavior you do not describe) come from a different source. Let me just list some ideas and comments c./ what was said before:
(1) recvfrom() blocks too. However, you want to use it. Your communication currently sends and receives from loopback, which is fine for your toy program (but: see below). When receiving UDP data, with recv() you don't know who sent it, as the socket was never connect()ed. Use recvfrom() to prepare yourself for some minimal error checking in a more serious program
(2) as select() suspends the program to i/o availibity, it would only put any issue with your socket blocking to a different level. But this is not the problem
(3) to check is the receive buffer is empty, use flag MSG_PEEK in recvfrom() in an appropriate position. It's usually only used to deal with scarce memory, but it should do the job.
(4) reason 1 why I believe you see the issues you don't describe in more detail:
UDP datagrams preserve message boundaries. This means that recvfrom() will read in an entire chunk of data making up any message sent. However, if the buffer you read this into is smaller than the data read, any surplus will be silently discarded. So make sure you have a big buffer (65k something ideally).
(5) reason 2:
You receive any data sent to the loopback. If you're currently also connected to some net (sat, the Internet), what you catch might actually be from a different source than you expect. So at least in a resting phase, disconnect.
Blocking shouldn't be an issue. Your basic logic, when coded cleanly, is:
Recvfrom() (block/wait until ready)
Process
Peek if buffer empty
Exit if yes
Loop back to receive more if not,
and you seem to want to do this currently. As you don't multi-thread, optimize fie perfiormance, or similar, you shouldn't care about blocking. If you find your receive buffer too small, increase its size using
Setsockopt() for optName SO_RCVBUF
Use select() (with a suitable timeout) to check for incoming data prior to calling recv().
Something like the following (nonportable code)
#include <winsock2.h>
...
/* somewhere after your sendto, or your first recv */
fd_set recv_set;
timeval tv = {1, 0}; /* one second */
FD_ZERO(&recv_set);
FD_SET(client, &recv_set);
while (select(0, &recv_set, NULL, NULL, &tv) > 0)
{
/* recv... */
FD_SET(client, &recv_set); /* actually redundant, since it is already set */
}
iPhone sometimes bugs out and doesn't let me post comments. Thanks, Steve. This is just continuing the conversation.
I assume this means 'uncomment to 'question starts here'. Partial answer, as this still depends on my 2nd comment; this is more or less than what to expect. Assuming the three messages to be sent by the server are already queued up, after you hit enter for the first time, your packet is sent (never blocked as sendto() doesn't block for UDP), received by the server and (I assume, see above, echoed back and added to the FIFO receive buffer in which you alread have three messages queued up. You then have a recv() in your program which receives the first queued up message, printing it out. Your current logic goes back to top of loop, expect another input and waits for it (so this isn't blocked on a socket level, but as your program requests input, eg simply 'enter'), then comes to the second originally sent message (by the server) and processes that one. One more loop, and all three are done. Hitting enter again, and assuming the server echoes what you sent, you should start receiving your typed in messages (which might be empty if you only hit enter). The loop will currently not exit except you kill it.
Related
This is more of a request for confirmation than a question, so I'll keep it brief. (I am away from my PC and so can't simply implement this solution to test).
I'm writing a program to send an image file taken via webcam (along with meta data) from a raspberryPi to my PC.
I've worked out that the image is roughly around 130kb, the packet header is 12b and the associated meta data another 24b. Though I may increase the image size in future, once I have a working prototype.
At the moment I am not able to retrieve this whole packet successfully as, after sending it to the PC I only ever get approx 64kb recv'd in the buffer.
I have assumed that this is because for whatever reason the default buffer size for a socket declared like:
SOCKET sock = socket(PF_INET, SOCK_STREAM, 0);
is 64kb (please could someone clarify this if you're 'in the know')
So - to fix this problem I intend to increase the socket size to 1024kb via the setsockopt(x..) command.
Please could someone confirm that my diagnosis of the problem, and proposed solution are correct?
I ask this question as I am away form my PC right now and am unable to try it until I get back home.
This most likely has nothing to do with the socket buffers, but with the fact that recv() and send() do not have to receive and send all the data you want. Check the return value of those function calls, it indicates how many bytes have actually been sent and received.
The best way to deal with "short" reads/writes is to put them in a loop, like so:
char *buf; // pointer to your data
size_t len; // length of your data
int fd; // the socket filedescriptor
size_t offset = 0;
ssize_t result;
while (offset < len) {
result = send(fd, buf + offset, len - offset, 0);
if (result < 0) {
// Deal with errors here
}
offset += result;
}
Use a similar construction for receiving data. Note that one possible error condition is that the function call was interrupted (errno = EAGAIN or EWOULDBLOCK), in that case you should retry the send command, in all other cases you should exit the loop.
//server side
void* s2(void *arg){
info *s = (info*)arg;
char buffer[MAXS];
int k;
sockaddr_in addr;
socklen_t aSize = sizeof(sockaddr_in);
int sfd = accept(s->fd,(sockaddr*)&addr,(socklen_t*)&aSize);
if(sfd<0){
s->current--;
pthread_exit(0);
}
while(1){
k = recv(sfd,buffer,MAXS,0);
cout<<buffer<<"\n";
//1. k ==0 socket has been closed by client
//2. k==-1 error in recv
//3. recv quit
if((k==-1)||(!strncmp(buffer,"quit",4))||(k==0))break;
sprintf(buffer,"%d\n",(int)strlen(buffer)); //Convert length to string using sprintf()
send(sfd,buffer,strlen(buffer),0); //send buffer to client
}
close(sfd);
if(s->limit==s->current)
FD_SET(s->fd,&sfds);
s->current--; //decreament the client number
pthread_exit(0);
}
//client side
1. send(sockfd,"sadhdag",8,0);
2. send(sockfd,"ss",3,0);
Server recv sadhdag in first call to recv .
In second call to recv server recv
ss
dag
server side:
function s2 is run by a thread and argument is passed with socket information connection is accepted there and send and recv is called onn newly accepted client.
Why this is happening ? or how to avoid this ?
You're ignoring the count returned by recv(). Contrary to suggestions in comments, it isn't necessary to memset() the buffer to zero before recv(), but it is necessary to only use that buffer afterwards up to that count. For example:
printf("%s", buffer);
is wrong, and
printf("%.*s", count, buffer);
is right.
NB
if((k==-1)||(!strncmp(buffer,"quit",4))||(k==0))break;
isn't correct either. It should be
if((k==-1)||(k==0)||(k >= 4 && !strncmp(buffer,"quit",4))) break;
It isn't valid to look into the buffer at all unless k is positive, and it isn't valid to compare four characters in it unless there are four characters in it.
You forgot to actually implement any kind of protocol or application-level logic of any kind. This sums it up:
k = recv(sfd,buffer,MAXS,0);
cout<<buffer<<"\n";
This code suggests you're terminating your messages with a newline:
sprintf(buffer,"%d\n",(int)strlen(buffer)); //Convert length to string using sprintf()
send(sfd,buffer,strlen(buffer),0); //send buffer to client
If so, where's the code to parse that on the other end? You throw the returned length away, so you wouldn't even know which characters to check for a newline.
TCP is a byte-stream protocol that does not preserve application-level message boundaries. If you want to use it to send strings or other application-level messages, you have to specify and implement an application-level protocol to do it.
You call recv to get the raw TCP data, then you output it as if it was a string. If you're going to define and implement a protocol to send and receive strings over TCP, you have to actually do it. You can't just imagine it happens by magic.
If you want to receive newline-terminated messages into C-style strings, you can certainly do it. But you have to actually write the code to do it.
I'm writing a distributed system in c++ using TCP/IP and sockets.
For each of my messages, I need to receive the first 5 bytes to know the full length of the incoming message.
What's the best way to do this?
recv() only 5 bytes, then recv() again. if I choose this, would it be safe to assume I'll get 0 or 5 bytes in the recv (aka not write a loop to keep trying)?
use MSG_PEEK
recv() some larger buffer size, then read the first 5 bytes and allocate the final buffer then.
You don't need to know anything. TCP is a stream protocol, and at any given moment you can get as little as one byte, or as much as multiple megabytes of data. The correct and only way to use a TCP socket is to read in a loop.
char buf[4096]; // or whatever
std::deque<char> data;
for (int res ; ; )
{
res = recv(fd, buf, sizeof buf, MSG_DONTWAIT);
if (res == -1)
{
if (errno == EAGAIN || errno == EWOULDBLOCK)
{
break; // done reading
}
else
{
// error, break, die
}
}
if (res == 0)
{
// socket closed, finalise, break
}
else
{
data.insert(data.end(), buf, buf + res);
}
}
The only purpose of the loop is to transfer data from the socket buffer to your application. Your application must then decide independently if there's enough data in the queue to attempt extraction of some sort of higher-level application message.
For example, in your case you would check if the queue's size is at least 5, then inspect the first five bytes, and then check if the queue holds a complete application message. If no, you abort, and if yes, you extract the entire message and pop if off from the front of the queue.
Use a state machine with two states:
First State.
Receive bytes as they arrive into a buffer. When there are 5 or more bytes perform your check on those first 5 and possibly process the rest of the buffer. Switch to the second state.
Second State.
Receive and process bytes as they arrive to the end of the message.
to answer your question specifically:
it's not safe to assume you'll get 0 or 5. it is possible to get 1-4 as well. loop until you get 5 or an error as others have suggested.
i wouldn't bother with PEEK, most of the time you'll block (assuming blocking calls) or get 5 so skip the extra call into the stack.
this is fine too but adds complexity for little gain.
I'm having a problem with unix local sockets. While reading a message that's longer than my temp buffer size, the request takes too long (maybe indefinitely).
Added after some tests:
there is still problem with freeze at ::recv. when I send (1023*8) bytes or less to the UNIX socket - all ok, but when sended more than (1023*9) - i get freeze on recv command.
maybe its FreeBSD default UNIX socket limit or C++ default socket settings? Who know?
i made some additational tests and I am 100% sure that its "freeze" on the last 9th itteration when executing ::recv command, when trying to read message >= (1023*9) bytes long. (first 8th itterationg going well.)
What I'm doing:
The idea is to read in a do/while loop from a socket with
::recv (current_socket, buf, 1024, 0);
and check buf for a SPECIAL SYMBOL. If not found:
merge content of buffer to stringxxx += buf;
bzero temp buf
continue the ::recv loop
How do I fix the issue with the request taking too long in the while loop?
Is there a better way to clear the buffer? Currently, it's:
char buf [1025];
bzero(buf, 1025);
But I know bzero is deprecated in the new c++ standard.
EDIT:
*"Why need to clean the buffer*
I see questions at comments with this question. Without buffer cleanup on the next(last) itteration of reading to the buffer, it will contain the "tail" of first part of the message.
Example:
// message at the socket is "AAAAAACDE"
char buf [6];
::recv (current_socket, buf, 6, 0); // read 6 symbols, buf = "AAAAAA"
// no cleanup, read the last part of the message with recv
::recv (current_socket, buf, 6, 0);
// read 6 symbols, but buffer contain only 3 not readed before symbols, therefore
// buf now contain "CDEAAA" (not correct, we waiting for CDE only)
When your recv() enters an infinite loop, this probably means that it's not making any progress whatsoever on the iterations (i.e., you're always getting a short read of zero size immediately, so your loop never exits, because you're not getting any data). For stream sockets, a recv() of zero size means that the remote end has disconnected (it's something like read()ing from a file when the input is positioned at EOF also gets you zero bytes), or at least that it has shut down the sending channel (that's for TCP specifically).
Check whether your PHP script is actually sending the amount of data you claim it sends.
To add a small (non-sensical) example for properly using recv() in a loop:
char buf[1024];
std::string data;
while( data.size() < 10000 ) { // what you wish to receive
::ssize_t rcvd = ::recv(fd, buf, sizeof(buf), 0);
if( rcvd < 0 ) {
std::cout << "Failed to receive\n"; // Receive failed - something broke, see errno.
std::abort();
} else if( !rcvd ) {
break; // No data to receive, remote end closed connection, so quit.
} else {
data.append(buf, rcvd); // Received into buffer, attach to data buffer.
}
}
if( data.size() < 10000 ) {
std::cout << "Short receive, sender broken\n";
std::abort();
}
// Do something with the buffer data.
Instead of bzero, you can just use
memset(buf, 0, 1025);
These are 2 separate issues. The long time is probably some infinite loop due to a bug in your code and has nothing to do with the way you clear your buffer. As a matter of fact you shouldn't need to clear the buffer; receive returns the number of bytes read, so you can scan the buffer for your SPECIAL_SYMBOL up to that point.
If you paste the code maybe I can help. more.
Just to clarify: bzero is not deprecated in C++ 11. Rather, it's never been part of any C or C++ standard. C started out with memset 20+ years ago. For C++, you might consider using std::fill_n instead (or just using std::vector, which can zero-fill automatically). Then again, I'm not sure there's a good reason to zero-fill the buffer in this case at all.
I am using read function to read data from a socket, but when the data is more than 4k, read function just read part of the data, for example, less than 4k. Here is the key code:
mSockFD = socket(AF_INET, SOCK_STREAM, 0);
if (connect(mSockFD, (const sockaddr*)(&mSockAdd), sizeof(mSockAdd)) < 0)
{
cerr << "Error connecting in Crawl" << endl;
perror("");
return false;
}
n = write(mSockFD, httpReq.c_str(), httpReq.length());
bzero(mBuffer, BUFSIZE);
n = read(mSockFD, mBuffer, BUFSIZE);
Note than BUFSIZE is much larger than 4k.
When data is just a few hundred bytes, read function works as expected.
This is by design and to be expected.
The short answer to your question is you should continue calling "read" until you get all the data you expect. That is:
int total_bytes = 0;
int expected = BUFSIZE;
int bytes_read;
char *buffer = malloc(BUFSIZE+1); // +1 for null at the end
while (total_bytes < expected)
{
int bytes_read = read(mSockFD, buffer+total_bytes, BUFSIZE-total_bytes);
if (bytes_read <= 0)
break;
total_bytes += bytes_read;
}
buffer[total_bytes] = 0; // null terminate - good for debugging as a string
From my experience, one of the biggest misconceptions (resulting in bugs) that you'll receive as much data as you ask for. I've seen shipping code in real products written with the expectation that sockets work this way (and no one certain as to why it doesn't work reliably).
When the other side sends N bytes, you might get lucky and receive it all at once. But you should plan for receiving N bytes spread out across multiple recv calls. With the exception of a real network error, you'll eventually get all N bytes. Segmentation, fragmentation, TCP window size, MTU, and the socket layer's data chunking scheme are the reasons for all of this. When partial data is received, the TCP layer doesn't know about how much more is yet to come. It just passes what it has up to the app. It's up to the app to decide if it got enough.
Likewise, "send" calls can get conglomerated into the same packet together.
There may be ioctls and such that will make a socket block until all the expected data is received. But I don't know of any off hand.
Also, don't use read and write for sockets. Use recv and send.
Read this book. It will change your life with regards to sockets and TCP: