I'm doing something similar to Stack Overflow question Handling partial return from recv() TCP in C.
The data receive is bigger than the buffer initialised (for example, 1000 bytes). Therefore a temporary buffer of a bigger size (for example, 10000 bytes) is used. The problem is that the multiple data received is rubbish. I've already checked the offset to memcpy to the temporary buffer, but I keep receiving rubbish data.
This sample shows what I do:
First message received:
memcpy(tmpBuff, dataRecv, 1000);
offSet = offSet + 1000;
Second msg onwards:
memcpy(tmpBuffer + offSet, dataRecv, 1000);
Is there something I should check?
I've checked the TCP hex that was sent out. Apparently, the sender is sending an incomplete message. How my program works is that when the sender sends the message, it will pack (message header + actual message). the message header has some meta data, and one of it is the message length.
When the receiver receives the packet, it will get the message header using the message header offset and message header length. It will extract the message length, check if the current packet size is more than or equal to the message length and return the correct message size to the users. If there's a remaining amount of message left in the packet, it will store it into a temporary buffer and wait to receive the next packet. When it receives the next packet, it will check the message header for the message length and do the same thing.
If the sender pack three messages in a packet, each message have its own message header indicating the message length. Assume all three messages are 300 bytes each in length. Also assume that the second message sent is incomplete and turns out to be only 100 bytes.
When the receiver receives the three messages in a packet, it will return the first message correctly. Since the second message is incomplete, my program wouldn't know, and so it will return 100 bytes from the second message and 200 bytes from the third message since the message header indicates the total size is 300 bytes. Thus the second message returned will have some rubbish data.
As for the third message, my program will try to get the message length from the message header. Since the first 200 bytes are already returned, the message header is invalid. Thus, the message length returned to my program will be rubbish as well. Is there a way to check for a complete message?
Suppose you are expecting 7000 bytes over the tcp connection. In this case it is very likely that your messages will be split into tcp packets with an actual payload size of let's say 1400 bytes (so 5 messages).
In this case it is perfectly possible consecutive recv calls with a target buffer of 1000 bytes will behave as follows:
recv -> reads 1000 bytes (packet 1)
recv -> reads 400 bytes (packet 1)
recv -> reads 1000 bytes (packet 2)
recv -> reads 400 bytes (packet 2)
...
Now, in this case, when reading the 400 bytes packet you still copy the full 1000 bytes to your larger buffer, actually pasting 600 bytes of rubbish in between. You should actually only memcpy the number of bytes received, which is the return value of recv itself. Of course you should also check if this value is 0 (socket closed) or less than zero (socket error).
Related
I have been reading some socket guides such as Beej's guide to network programming. It is quite clear now that there is no guarantee on how many bytes are received in a single recv() call. Therefore a mechanism of e.g. first two bytes stating the message length should be sent and then the message. So the receiver receives the first two bytes and then receives in a loop until the whole message has been received. All good and dandy!?
I was asked by a colleague about messages going out of sync. E.g. what if, somehow, I receive two bytes in once recv() call that are actually in the middle of the message itself and it would appear as a integer of some value? Does that mean that the rest of the data sent will be out of sync? And what about receiving the header partially, i.e. one byte at a time?
Maybe this is overthinking, but I can't find this mentioned anywhere and I just want to be sure that I would handle this if it could be a possible threat to the integrity of the communication.
Thanks.
It is not overthinking. TCP presents a stream so you should treat it this way. A lot of problems concerning TCP are due to network issues and will probably not happen during development.
Start a message with a (4 byte) magic that you can look for followed by a (4 byte) length in an expected order (normally big endian). When receiving, read each byte of the header at the time, so you can handle it anyway the bytes were received. Based on that you can accept messages in a lasting TCP connection.
Mind you that when starting a new connection per message, you know the starting point. However, it doesn't hurt sending a magic either, if only to filter out some invalid messages.
A checksum is not necessary because TCP shows a reliable stream of bytes which was already checked by the receiving part of TCP, and syncing will only be needed if there was a coding issue with sending/receiving.
On the other hand, UDP sends packets, so you know what to expect, but then the delivery and order is not guaranteed.
Your colleague is mistaken. TCP data cannot arrive out of order. However you should investigate the MSG_WAITALL flag to recv() to overcome the possibility of the two length bytes arriving separately, and to eliminate the need for a loop when receiving the message body.
Its your responsibility to make you client and server syncing together, how ever in TCP there is no out of order delivery, if you got something by calling recv() you can think there isn't anything behind that that you doesn't received.
So the question is how to synchronize sender and receiver ? its easy, as stefaanv said, sender and receiver are knowing their starting point. so you can define a protocol for your network communication. for example a protocol could be defined this way :
4 bytes of header including message type and payload length
Rest of message is payload length
By this, you have to send 4 byte header before sending actual payload, then sending actual payload followed.
Because TCP has garauntied Inorder reliable delivery, you can make two recv() call for each pack. one recv() call with length of 4 bytes for getting next payload size, and another call to recv() with size specified in header. Its necessary to make both recv() blocking to getting synchronized all the time.
An example would be like this:
#define MAX_BUF_SIZE 1024 // something you know
char buf[MAX_BUF_SIZE];
int recvLen = recv(fd, buff, 4, MSG_PEEK);
if(recvLen==4){
recvLen = recv(fd, buff, 4);
if(recvLen != 4){
// fatal error
}
int payloadLen = extractPayloadLenFromHeader(buf);
recvLen = recv(fd, buff, payloadLen, MSG_PEEK);
if(recvLen == payloadLen){
recvLen = recv(fd, buff, payloadLen); // actual recv
if(recvLen != payloadLen){
// fatal error
}
// do something with received payload
}
}
As you can see, i have first called recv with MSG_PEEK flag to ensure is there really 4 bytes available or not, then received actual header. same for payload
I've been reading through Beej's Guide to Network Programming to get a handle on TCP connections. In one of the samples the client code for a simple TCP stream client looks like:
if ((numbytes = recv(sockfd, buf, MAXDATASIZE-1, 0)) == -1) {
perror("recv");
exit(1);
}
buf[numbytes] = '\0';
printf("Client: received '%s'\n", buf);
close(sockfd);
I've set the buffer to be smaller than the total number of bytes that I'm sending. I'm not quite sure how I can get the other bytes. Do I have to loop over recv() until I receive '\0'?
*Note on the server side I'm also implementing his sendall() function, so it should actually be sending everything to the client.
See also 6.1. A Simple Stream Server in the guide.
Yes, you will need multiple recv() calls, until you have all data.
To know when that is, using the return status from recv() is no good - it only tells you how many bytes you have received, not how many bytes are available, as some may still be in transit.
It is better if the data you receive somehow encodes the length of the total data. Read as many data until you know what the length is, then read until you have received length data. To do that, various approaches are possible; the common one is to make a buffer large enough to hold all data once you know what the length is.
Another approach is to use fixed-size buffers, and always try to receive min(missing, bufsize), decreasing missing after each recv().
The first thing you need to learn when doing TCP/IP programming: 1 write/send call might take
several recv calls to receive, and several write/send calls might need just 1 recv call to receive. And anything in-between.
You'll need to loop until you have all data. The return value of recv() tells you how much data you received. If you simply want to receive all data on the TCP connection, you can loop until recv() returns 0 - provided that the other end closes the TCP connection when it is done sending.
If you're sending records/lines/packets/commands or something similar, you need to make your own protocol over TCP, which might be as simple as "commands are delimited with \n".
The simple way to read/parse such a command would be to read 1 byte at a time, building up a buffer with the received bytes and check for a \n byte every time. Reading 1 byte is extremely inefficient, so you should read larger chunks at a time.
Since TCP is stream oriented and does not provide record/message boundaries it becomes a bit more tricky - you'd
have to recv a piece of bytes, check in the received buffer for a \n byte, if it's there - append the bytes to previously received bytes and output that message. Then check the remainder of the buffer after the \n - which might contain another whole message or just the start of another message.
Yes, you have to loop over recv() until you receive '\0' or an
error happen (negative value from recv) or 0 from recv().
For the first option: only if this zero is part of your
protocol (the server sends it). However from your code it seems that
the zero is just to be able to use the buffer content as a
C-string (on the client side).
The check for a return value of 0 from recv:
this means that the connection was closed (it could be part
of your protocol that this happens.)
In order to determine the type of message received in a UDP packet, there is a need to look at specific buffer element [i] received from "recvfrom" in order to discern the type of message intended. first, i use a buffer in the stack to populate the buffer (of recvfrom), i know the maximum size of the message i should receive.
So say my array buffer is of 300 bytes, and i receive a packets of different sizes (e.g. 30, 80, 210 byes etc)....how can i know the size received (this is because there are few other criteria i test for to determine the nature of the message )
Knowing the size will enable me to use memcpy to an object.
i'm thinking of strlen(udp packet) because it is determined at runtime as opposed to compile time.
the problem is what if the rest of packet was filled with junk....
I appreciate it
recv(2), which is used to receive a UDP packet, returns the number of bytes received.
I've been reading through Beej's Guide to Network Programming to get a handle on TCP connections. In one of the samples the client code for a simple TCP stream client looks like:
if ((numbytes = recv(sockfd, buf, MAXDATASIZE-1, 0)) == -1) {
perror("recv");
exit(1);
}
buf[numbytes] = '\0';
printf("Client: received '%s'\n", buf);
close(sockfd);
I've set the buffer to be smaller than the total number of bytes that I'm sending. I'm not quite sure how I can get the other bytes. Do I have to loop over recv() until I receive '\0'?
*Note on the server side I'm also implementing his sendall() function, so it should actually be sending everything to the client.
See also 6.1. A Simple Stream Server in the guide.
Yes, you will need multiple recv() calls, until you have all data.
To know when that is, using the return status from recv() is no good - it only tells you how many bytes you have received, not how many bytes are available, as some may still be in transit.
It is better if the data you receive somehow encodes the length of the total data. Read as many data until you know what the length is, then read until you have received length data. To do that, various approaches are possible; the common one is to make a buffer large enough to hold all data once you know what the length is.
Another approach is to use fixed-size buffers, and always try to receive min(missing, bufsize), decreasing missing after each recv().
The first thing you need to learn when doing TCP/IP programming: 1 write/send call might take
several recv calls to receive, and several write/send calls might need just 1 recv call to receive. And anything in-between.
You'll need to loop until you have all data. The return value of recv() tells you how much data you received. If you simply want to receive all data on the TCP connection, you can loop until recv() returns 0 - provided that the other end closes the TCP connection when it is done sending.
If you're sending records/lines/packets/commands or something similar, you need to make your own protocol over TCP, which might be as simple as "commands are delimited with \n".
The simple way to read/parse such a command would be to read 1 byte at a time, building up a buffer with the received bytes and check for a \n byte every time. Reading 1 byte is extremely inefficient, so you should read larger chunks at a time.
Since TCP is stream oriented and does not provide record/message boundaries it becomes a bit more tricky - you'd
have to recv a piece of bytes, check in the received buffer for a \n byte, if it's there - append the bytes to previously received bytes and output that message. Then check the remainder of the buffer after the \n - which might contain another whole message or just the start of another message.
Yes, you have to loop over recv() until you receive '\0' or an
error happen (negative value from recv) or 0 from recv().
For the first option: only if this zero is part of your
protocol (the server sends it). However from your code it seems that
the zero is just to be able to use the buffer content as a
C-string (on the client side).
The check for a return value of 0 from recv:
this means that the connection was closed (it could be part
of your protocol that this happens.)
I want to hold onto packets that fail the crc check. To do this I have commented out a section of the CC2420RecieveP readDone function that checks the msb bit of the LQI byte in the received buffer.
I think this is working,
However, once I receive the packet in my own receive function I send it through the serial component (not just the payload, I copy the whole received radio packet into the payload area of the serial packet).
When I use the program Listen, it seems that the crc bool value is not there (only the LQI and RSSI) even though the crc is clearly copied into the bufPTR in the function receiveDone_task.
:(
Help!
Mike.
i was only copying the first 28 bytes (not the header plus a 28 byte payload plus the metadata)
:P