when using recvmsg I use MSG_TRUNC and MSG_PEEK like so:
msgLen = recvmsg(fd, &hdr, MSG_PEEK | MSG_TRUNC)
this gives me the size of the buffer to allocate for the next message
my question is how do I get the size of the buffer I should allocate for the msg_control field inside the header
Based on the doc, you need to allocate the buffer for msg_control of the size msg_controllen. To know the size beforehand, you could call like you did recvmsg(fd, &hdr, MSG_PEEK | MSG_TRUNC). MSG_PEEK won't remove the message and MSG_TRUNC will allow to return the size of the message, even if the buffer is too small.
a few solutions:
call recvmsg(fd, &hdr, MSG_PEEK | MSG_TRUNC) and init the buffer in hdr based on the size returned, and call it again without the flags.
allocate a buffer big enough, if you know the size of your messages beforehand, and call recvmsg. If an error occurs (returned -1), check the error code if the message was truncated (MSG_TRUNC or MSG_CTRUNC)
I cannot speak for other platforms than macOS (whose core is based upon a FreeBSD core, so maybe it's no different in BSD-systems, too) and the POSIX standard is not helpful either as it leaves pretty much all details to be defined by the protocol, but by default behavior of recvmsg on macOS for a UDP socket is to not deliver any control data at all. No matter what size you set msg_control on input, it will always be 0 on output. If you wish to receive any control data, you first have to explicitly enable that for the socket.
E.g. if you want to know both addresses, source and destination address of a packet (msg_name only gives you the source address of a received packet), then you have to do this:
int yes = 1;
setsockopt(soc, IPPROTO_IP, IP_RECVDSTADDR, &yes, sizeof(yes));
And now you'll get the destination address for IPv4 sockets documented as
The msg_control field in the msghdr structure points to a buffer that
contains a cmsghdr structure followed by the IP address. The cmsghdr
fields have the following values:
cmsg_len = sizeof(struct in_addr)
cmsg_level = IPPROTO_IP
cmsg_type = IP_RECVDSTADDR
This means you need to provide at least 16 bytes storage on my system, as struct cmsghdr alone is always 12 bytes on that system (four times 32 bit) and an IPv4 address is another 4 bytes, that's 16 bytes together. This value needs to be correctly rounded using CMSG_SPACE macro, but on my system the macro only makes sure it's a multiple of 32 bit and 16 byte already is such a multiple, so CMSG_SPACE(16) returns 16 for me.
As I know in advance which options I have enabled and which control data I will receive, I can exactly calculate the required space in advance.
For raw and other more obscure sockets, certain control data may always be included in the output by default, even if not explicitly enabled, but this control data will then always be equal in size and won't fluctuate from packet to packet as the packet payload size does. Thus once you know the correct size, you can rely upon the fact that it won't change, at least not without you enabling/disabling any options.
If your control data buffer was too small, the MSG_CTRUNC flag is set in the output, always (even if you don't set any flags on input), then you need to increase the control data buffer size and try again (with the next packet or with the same packet if you used MSG_PEEK as input flag), until you've once been able to make that call without getting the MSG_CTRUNC flag on output. Finally look at what the msg_control field says. On input it's the amount of buffer space available but on output it contains the exact amount of buffer space that was actually used. This is the exact buffer size you need to receive the control data of all future packets of that socket, unless you change options that will cause more/less control data to be sent and then you just have to detect that size again the same way as before.
For a more complete example, you may also have a look at:
https://stackoverflow.com/a/49308499/15809
I am afraid you can't get that value from the Posix.1g sockets API. Not sure about all implementations, but not possible in Linux. As you may notice, no control flow is provided in ancillary data buffers, so you will need to implement it yourself in case you are sending a lot of info between processes. On the other hand, for common case uses, you already know what you are going to receive at compile time (but you probably already know this). If you need to implement you own control flow, take into account that, in Linux, ancillary data seems to behave like a stream socket.
However, you can get/set the buffer length of the worst case scenario in /proc/sys/net/core/optmem_max, see cmsg(3). So, I guess you could set it to a reasonable value and declare a buffer that big.
Related
I'm realizing UART-DMA with STM_HAL library and I want to know if message size is counted by hardware (counting clock ticks till line is idle for example) or by some program method(something like strlen). So if Size in
HAL_UARTEx_RxEventCallback(UART_HandleTypeDef *huart, uint16_t Size)
is counted by hardware, I can send data in pure HEX format, but if it is calculated by something like strline, I may recieve problems if data is 0x00 and have to send data in ASCII.
I've tried to make some research in generated code in Keil but failed (maybe I didn't try hard enough) so maybe somebody can help me.
If you are using UART DMA, it is calculated by hardware.
If you check the call hierarchy of HAL_UARTEx_RxEventCallback using your ide, you can see how the Size variable is calculated.
The function is executed in the following flow.(Depending on the version of HAL Driver, it may be slightly different)
UART Idle Interrupt occur
Call HAL_UART_IRQHandler()
If DMA mod is enabled, Call HAL_UARTEx_RxEventCallback(huart, (huart->RxXferSize - huart->RxXferCount))
Therefore, Size variable is calculated as (huart->RxXferSize - huart->RxXferCount)
huart->RxXferSize is a set value when initializing RX DMA.
huart->RxXferCount is (huart->hdmarx)->Instance->NDTR
NDTR is a value calculated by hardware as the size of the buffer remaining after DMA transfer data to memory!!
I need to send data to another process every 0.02s.
The Server code:
//set socket, bind, listen
while(1){
sleep(0.02);
echo(newsockfd);
}
void echo (int sock)
{
int n;
char buffer[256]="abc";
n=send(sock,buffer,strlen(buffer),0);
if (n < 0) error("ERROR Sending");
}
The Client code:
//connect
while(1)
{
bzero(buffer,256);
n = read(sock,buffer,255);
printf("Recieved data:%s\n",buffer);
if (n < 0)
error("ERROR reading from socket");
}
The problem is that:
The client shows something like this:
Recieved data:abc
Recieved data:abcabcabc
Recieved data:abcabc
....
How does it happen? When I set sleep time:
...
sleep(2)
...
It would be ok:
Recieved data:abc
Recieved data:abc
Recieved data:abc
...
TCP sockets do not guarantee framing. When you send bytes over a TCP socket, those bytes will be received on the other end in the same order, but they will not necessarily be grouped the same way — they may be split up, or grouped together, or regrouped, in any way the operating system sees fit.
If you need framing, you will need to send some sort of packet header to indicate where each chunk of data starts and ends. This may take the form of either a delimiter (e.g, a \n or \0 to indicate where each chunk ends), or a length value (e.g, a number at the head of each chunk to denote how long it is).
Also, as other respondents have noted, sleep() takes an integer, so you're effectively not sleeping at all here.
sleep takes unsigned int as argument, so sleep(0.02) is actually sleep(0).
unsigned int sleep(unsigned int seconds);
Use usleep(20) instead. It will sleep in microseconds:
int usleep(useconds_t usec);
The OS is at liberty to buffer data (i.e. why not just send a full packet instead of multiple packets)
Besides sleep takes a unsigned integer.
The reason is that the OS is buffering data to be sent. It will buffer based on either size or time. In this case, you're not sending enough data, but you're sending it fast enough the OS is choosing to bulk it up before putting it on the wire.
When you add the sleep(2), that is long enough that the OS chooses to send a single "abc" before the next one comes in.
You need to understand that TCP is simply a byte stream. It has no concept of messages or sizes. You simply put bytes on the wire on one end and take them off on the other. If you want to do specific things, then you need to interpret the data special ways when you read it. Because of this, the correct solution is to create an actual protocol for this. That protocol could be as simple as "each 3 bytes is one message", or more complicated where you send a size prefix.
UDP may also be a good solution for you, depending on your other requirements.
sleep(0.02)
is effectively
sleep(0)
because argument is unsigned int, so implicit conversion does it for you. So you have no sleep at all here. You can use sleep(2) to sleep for 2 microseconds.Next, even if you had, there is no guarantee that your messages will be sent in a different frames. If you need this, you should apply some sort of delimiter, I have seen
'\0'
character in some implementation.
TCPIP stacks buffer up data until there's a decent amount of data, or until they decide that there's no more coming from the application and send what they've got anyway.
There are two things you will need to do. First, turn off Nagle's algorithm. Second, sort out some sort of framing mechanism.
Turning off Nagle's algorithm will cause the stack to "send data immediately", rather than waiting on the off chance that you'll be wanting to send more. It actually leads to less network efficiency because you're not filling up Ethernet frames, something to bare in mind on Gigabit where jumbo frames are required to get best throughput. But in your case timeliness is more important than throughput.
You can do your own framing by very simple means, eg by send an integer first that says how long the rest if the message will be. At the reader end you would read the integer, and then read that number of bytes. For the next message you'd send another integer saying how long that message is, etc.
That sort of thing is ok but not hugely robust. You could look at something like ASN.1 or Google Protocol buffers.
I've used Objective System's ASN.1 libraries and tools (they're not free) and they do a good job of looking after message integrity, framing, etc. They're good because they don't read data from a network connection one byte at a time so the efficiency and speed isn't too bad. Any extra data read is retained and included in the next message decode.
I've not used Google Protocol Buffers myself but it's possible that they have similar characteristics, and there maybe other similar serialisation mechanisms out there. I'd recommend avoiding XML serialisation for speed/efficiency reasons.
struct ss
{ int data ;string name;}*o;
this is my structure in the client application
i want to send that to the server (in TCP Server under windows);
How to do that..
i Know the serilization is the solution.
but i don't know how to do that.. please help me if you can.
http://www.parashift.com/c++-faq-lite/serialization.html
If you use Boost for serialization then read this
Serialize and send objects by TCP using boost
I normally write my own data format for transferring this data.
I will create a character buffer.
Put the size of the packet.
Convert integer to bytes and copy it.
Append the string length
Copy the name
send it across.
First one must know whether client and server are known to be always the same architecture or not. This decides whether you can just send data as it is or whether you must care about endianness and the size of an integer.
In any case, ntohl and htonl will take care of byte ordering and allow you to transfer the int in a simple, defined way (no-op on machines that are already network-byte-order).
About the string, you can send both the size and the contents over TCP just fine (converting the size with htonl), assuming that the string data is either in the same encoding on both sides, or a "general, agnostic" encoding is always used, such as UTF-8.
If you don't know what encodings the machines on both ends are using, you are in trouble. In that case, you must include a message that defines this and convert accordingly (similar to as for example webservers do).
Having TCP operate in "normal mode" means that Nagle's algorithm will be enabled, so you can just use 3 calls to send and the network layer will coerce that into as few packets as it believes is reasonable (instead of sending an individual packet just for an integer).
That all for the simple case in your example, or you can do some proper serialization, which is much more work, of course.
This is a very good guide: http://beej.us/guide/bgnet/output/html/multipage/index.html. it is not strictly for windows but the changes are very slight.
You basically need to serialize your data to a buffer and then use send function. pass your socket identifier and buffer etc...
int send(
__in SOCKET s,
__in const char *buf,
__in int len,
__in int flags
);
from http://msdn.microsoft.com/en-us/library/ms740149(v=vs.85).aspx
// send some data to a socket
send(
socket, // the open socket
o, // pointer to the data
sizeof( ss ), // number of bytes
0 ); // no special flags
The difficulty is at the other end! The recipient needs to know how many bytes to read, and what structure to store them in.
You either need to write your own code to deal with these problems, or use a protocol that works on top of sockets. The code isn't hard, but some experience with some of the many protocols available would help with the gotchas!
Our server is seemingly packet based. It is an adaptation from an old serial based system. It has been added, modified, re-built, etc over the years. Since TCP is a stream protocol and not a packet protocol, sometimes the packets get broken up. The ServerSocket is designed in such a way that when the Client sends data, part of the data contains the size of our message such as 55. Sometimes these packets are split into multiple pieces. They arrive in order but since we do not know how the messages will be split, our server sometimes does not know how to identify the split message.
So, having given you the background information. What is the best method to rebuild the packets as they come in if they are split? We are using C++ Builder 5 (yes I know, old IDE but this is all we can work with at the moment. ALOT of work to re-design in .NET or newer technology).
TCP guarantees that the data will arrive in the same order it was sent.
That beeing said, you can just append all the incoming data to a buffer. Then check if your buffer contains one or more packets, and remove them from the buffer, keeping all the remaining data into the buffer for future check.
This, of course, suppose that your packets have some header that indicates the size of the following data.
Lets consider packets have the following structure:
[LEN] X X X...
Where LEN is the size of the data and each X is an byte.
If you receive:
4 X X X
[--1--]
The packet is not complete, you can leave it in the buffer. Then, other data arrives, you just append it to the buffer:
4 X X X X 3 X X X
[---2---]
You then have 2 complete messages that you can easily parse.
If you do it, don't forget to send any length in a host-independant form (ntohs and ntohl can help).
This is often accomplished by prefixing messages with a one or two-byte length value which, like you said, gives the length of the remaining data. If I've understood you correctly, you're sending this as plain text (i.e., '5', '5') and this might get split up. Since you don't know the length of a decimal number, it's somewhat ambiguous. If you absolutely need to go with plain text, perhaps you could encode the length as a 16-bit hex value, i.e.:
00ff <255 bytes data>
000a <10 bytes data>
This way, the length of the size header is fixed to 4 bytes and can be used as a minimum read length when receiving on the socket.
Edit: Perhaps I misunderstood -- if reading the length value isn't a problem, deal with splits by concatenating incoming data to a string, byte buffer, or whatever until its length is equal to the value you read in the beginning. TCP will take care of the rest.
Take extra precautions to make sure that you can't get stuck in a blocking read state should the client not send a complete message. For example, say you receive the length header, and start a loop that keeps reading through blocking recv() calls until the buffer is filled. If a malicious client intentionally stops sending data, your server might be locked until the client either disconnects, or starts sending.
I would have a function called readBytes or something that takes a buffer and a length parameter and reads until that many bytes have been read. You'll need to capture the number of bytes actually read and if it's less than the number you're expecting, advance your buffer pointer and read the rest. Keep looping until you've read them all.
Then call this function once for the header (containing the length), assuming that the header is a fixed length. Once you have the length of the actual data, call this function again.
c++
#define BUF_LEN 1024
the below code only receives one byte when its called then immediately moves on.
output = new char[BUF_LEN];
bytes_recv = recv(cli, output, BUF_LEN, 0);
output[bytes_recv] = '\0';
Any idea how to make it receive more bytes?
EDIT: the client connecting is Telnet.
The thing to remember about networking is that you will be able to read as much data as has been received. Since your code is asking for 1024 bytes and you only read 1, then only 1 byte has been received.
Since you are using a telnet client, it sounds like you have it configured in character mode. In this mode, as soon as you type a character, it will be sent.
Try to reconfigure your telnet client in line mode. In line mode, the telnet client will wait until you hit return before it sends the entire line.
On my telnet client. In order to do that, first I type ctrl-] to get to the telnet prompt and then type "mode line" to configure telnet in line mode.
Update
On further thought, this is actually a very good problem to have.
In the real world, your data can get fragmented in unexpected ways. The client may make a single send() call of N bytes but the data may not arrive in a single packet. If your code can handle byte arriving 1 by 1, then you know it will work know matter how the data arrives.
What you need to do is make sure that you accumulate your data across multiple receives. After your recv call returns, you should then append the data a buffer. Something like:
char *accumulate_buffer = new char[BUF_LEN];
size_t accumulate_buffer_len = 0;
...
bytes_recv = recv(fd,
accumulate_buffer + accumulate_buffer_len,
BUF_LEN - accumulate_buffer_len,
0);
if (bytes_recv > 0)
accumulate_buffer_len += bytes_recv;
if (can_handle_data(accumulate_buffer, accumulate_buffer_len))
{
handle_data(accumulate_buffer, accumulate_buffer_len);
accumulate_buffer_len = 0;
}
This code keeps accumulating the recv into a buffer until there is enough data to handle. Once you handle the data, you reset the length to 0 and you start accumulating afresh.
First, in this line:
output[bytes_recv] = '\0';
you need to check if bytes_recv < 0 first before you do that because you might have an error. And the way your code currently works, you'll just randomly stomp on some random piece of memory (likely the byte just before the buffer).
Secondly, the fact you are null terminating your buffer indicates that you're expecting to receive ASCII text with no embedded null characters. Never assume that, you will be wrong at the worst possible time.
Lastly stream sockets have a model that's basically a very long piece of tape with lots of letters stamped on it. There is no promise that the tape is going to be moving at any particular speed. When you do a recv call you're saying "Please give me as many letters from the tape as you have so far, up to this many.". You may get as many as you ask for, you may get only 1. No promises. It doesn't matter how the other side spit bits of the tape out, the tape is going through an extremely complex bunch of gears and you just have no idea how many letters are going to be coming by at any given time.
If you care about certain groupings of characters, you have to put things in the stream (ont the tape) saying where those units start and/or end. There are many ways of doing this. Telnet itself uses several different ones in different circumstances.
And on the receiving side, you have to look for those markers and put the sequences of characters you want to treat as a unit together yourself.
So, if you want to read a line, you have to read until you get a '\n'. If you try to read 1024 bytes at a time, you have to take into account that the '\n' might end up in the middle of your buffer and so your buffer may contain the line you want and part of the next line. It might even contain several lines. The only promise is that you won't get more characters than you asked for.
Force the sending side to send more bytes using Nagle's algorithm, then you will receive them in packages.