Calls such as send() and sendto() in the Winsock API take a primitive int to dictate the size of their buffer parameters. This obviously places a 32-bit limit on the maximum size buffer that may be sent.
Why is this? Is there a 64-bit Winsock2 API available that might use a more appropriate size type (e.g. size_t)?
On Linux similar calls use the size_t type for defining sizes.
There is no need for a function with larger size, actually the type could be short and still it won’t be a problem.
Sockets don’t send messages, they just transfer bytes. When you call send() the data may not get received in one chunk at the recv() call. You have to implement logic when receiving bytes to know if you got it all or not and call recv() again if not. So if you wanna send something larger than can be placed in an int? Just make multiple calls to send(). If your recv() code can’t handle that, it’s a bug, because it should.
Related
I'm trying to make a simple tcp server according to this lesson. Everything works fine, and now I'm trying to encapsulate functions in the Socket class. I try to make a method that checks the amount of available bytes to read, and I can't find necessary the function. It could be some kind of ftell() or another method.
You should be respecting the communication protocol you are implementing on top of TCP. It dictates how you should be reading data and how you should be determining the sizes of the data to be read. If the protocol expects an integer, read an integer. If the protocol expects a length-preceded string, read the length value and then read how many bytes it specifies. If the protocol expects a delimited string, read until you encounter the delimiter. And so on.
That being said, to answer the actual question you asked - "[how to] check the amount of available bytes to read":
on Windows, you can use the ioctlsocket() function, where the cmd parameter is set to FIONREAD and the argp parameter is set to a pointer to a u_long variable.
u_long bytesAvailable;
ioctlsocket(socket, FIONREAD, &bytesAvailable);
on *Nix, you can use the ioctl() function, where the request parameter is set to FIONREAD and the third parameter is set to a pointer to a int variable.
int bytesAvailable;
ioctl(socket, FIONREAD, &bytesAvailable);
In either case, the output variable receives the number of unread bytes currently waiting on the socket. It is guaranteed that you can read at most this many bytes from the socket using recv() without blocking the calling thread (if the socket is running in a blocking mode). The socket may receive more bytes between the time you query FIONREAD and the time you call recv(), so if you try to read more than FIONREAD indicates, recv() may or may not have to block the calling thread waiting for more bytes.
Can you elaborate on why you would need to know that?
The preferred way to read data from a socket is to first make sure it has data you can read, via a call to select and then read into a predefined array, whose size is set to that of maximum size you and the client respect. Read will then report how many bytes it has read.
I might also add you should also select a socket before writing data to it.
Finally, you might want to look up circular buffers as they offer a way to append data to a buffer without risking to fill up your RAM. You would then read the data from the circular buffer once you have enough data.
As far as I know, it is not recommended to check how many bytes are available to be read on a socket.
I am using the boost asio library to read some data using tcp. After using a.accept(*sock);, how to get the size of the 1st packet the client will send?
I use (sock->remote_endpoint().address()).to_string() to get the IP address of the user, so I guess there must be a similar simple way to get the size of the packet, right?
At the application level, it is often far more useful to know the number of bytes currently available for reading, rather than the packet size. The amount of data available for reading may be constructed from one or more TCP segments. In the OSI model, a TCP segment (Layer 4: Transport) may be constructed from one or more IP Layer packets (Layer 3: Network), and each packet may be constructed from one or more Ethernet frames (Layer 2: Data Link).
Therefore, I am going to assume the application is interested in knowing how many bytes to read, rather than knowing lower level details, such as the size of a packet. There are a few solutions to this problem:
Query the socket for how much data is available via socket::available(), then allocate the buffer accordingly.
std::vector<char> data(socket_.available());
boost::asio::read(socket_, boost::asio::buffer(data));
Use a class that Boost.Asio can grow in memory, such as boost::asio::streambuf. Some operations, such as boost::asio::read() accept streambuf objects as their buffer and will allocate memory as is required for the operation. However, a completion condition should be provided; otherwise, the operation will continue until the buffer is full.
boost::asio::streambuf data;
boost::asio::read(socket_, data,
boost::asio::transfer_at_least(socket_.available()));
As Igor R. suggests in the comments, incorporate length as part of the communication protocol. Check the Boost.Asio examples for examples of communication protocols. Focus on the protocol, not necessarily on the Boost.Asio API.
In a fixed length protocol, a constant byte size is used to indicate message boundaries, such as in the Boost.Asio Porthopper example. As the reader knows the size of the message, the reader can allocate a buffer in advance.
In a variable length protocol, such as the one used in the Boost.Asio Chat example, a message is often divided into two parts: a header and a body. One approach is to have a a fixed size header that contains various meta-information, such as the length of the body. This allows an application to read a header into a fixed size buffer, extract the body length, allocate a buffer for the body, then read the body.
// Read fixed header.
std::vector<char> data(fixed_header_size);
boost::asio::read(socket_, boost::asio::buffer(data));
protocol::header header(data);
network_to_local(header); // Handle endianess.
// Read body.
data.resize(header.body_length());
boost::asio::read(socket_, boost::asio::buffer(data));
protocol::body body(data);
network_to_local(body); // Handle endianess.
On the other hand, if I am mistaken, and you do need the total length of a packet, then one can use the basic_raw_socket. Boost.Asio's ICMP example demonstrates reading IPv4 packets from a socket, and extracting the header's field values.
I am using berkeley sockets and TCP (SOCK_STREAM sockets).
The process is:
I connect to a remote address.
I send a message to it.
I receive a message from it.
Imagine I am using the following buffer:
char recv_buffer[3000];
recv(socket, recv_buffer, 3000, 0);
Questions are:
How can I know if after calling recv first time the read buffer is empty or not? If it's not empty I would have to call recv again, but if I do that when it's empty I would have it blocking for much time.
How can I know how many bytes I have readed into recv_buffer? I can't use strlen because the message I receive can contain null bytes.
Thanks.
How can I know if after calling recv
first time the read buffer is empty or
not? If it's not empty I would have to
call recv again, but if I do that when
it's empty I would have it blocking
for much time.
You can use the select or poll system calls along with your socket descriptor to tell if there is data waiting to be read from the socket.
However, usually there should be an agreed-upon protocol that both sender and receiver follow, so that both parties know how much data is to be transferred. For example, perhaps the sender first sends a 2-byte integer indicating the number of bytes it will send. The receiver then first reads this 2-byte integer, so that it knows how many more bytes to read from the socket.
Regardless, as Tony pointed out below, a robust application should use a combination of length-information in the header, combined with polling the socket for additional data before each call to recv, (or using a non-blocking socket). This will prevent your application from blocking in the event that, for example, you know (from the header) that there should still be 100 bytes remaining to read, but the peer fails to send the data for whatever reason (perhaps the peer computer was unexpectedly shut off), thus causing your recv call to block.
How can I know how many bytes I have
readed into recv_buffer? I can't use
strlen because the message I receive
can contain null bytes.
The recv system call will return the number of bytes read, or -1 if an error occurred.
From the man page for recv(2):
[recv] returns the number of bytes
received, or -1 if an error occurred.
The return value will be 0 when the
peer has performed an orderly
shutdown.
How can I know if after calling recv first time the read buffer is empty or not?
Even the first time (after accepting a client), the recv can block and fail if the client connection has been lost. You must either:
use select or poll (BSD sockets) or some OS-specific equivalent, which can tell you whether there is data available on specific socket descriptors (as well as exception conditions, and buffer space you can write more output to)
you can set the socket to be nonblocking, such that recv will only return whatever is immediately available (possibly nothing)
you can create a thread that you can afford to have block recv-ing data, knowing other threads will be doing the other work you're concerned to continue with
How can I know how many bytes I have readed into recv_buffer? I can't use strlen because the message I receive can contain null bytes.
recv() returns the number of bytes read, or -1 on error.
Note that TCP is a byte stream protocol, which means that you're only guaranteed to be able to read and write bytes from it in the correct order, but the message boundaries are not guaranteed to be preserved. So, even if the sender has made a large single write to their socket, it can be fragmented en route and arrive in several smaller blocks, or several smaller send()/write()s can be consolidated and retrieved by one recv()/read().
For that reason, make sure you loop calling recv until you either get all the data you need (i.e. a complete logical message you can process) or an error. You should be prepared/able to handle getting part/all of subsequent sends from your client (if you don't have a protocol where each side only sends after getting a complete message from the other, and are not using headers with message lengths). Note that doing recvs for the message header (with length) then the body can result in a lot more calls to recv(), with a potential adverse affect on performance.
These reliability issues are often ignored. They manifest less often when on a single host, a reliable and fast LAN, with less routers and switches involved, and fewer or non-concurrent messages. Then they may break under load and over more complex networks.
If the recv() returns fewer than 3000 bytes, then you can assume that the read buffer was empty. If it returns 3000 bytes in your 3000 byte buffer, then you'd better know whether to continue. Most protocols include some variation on TLV - type, length, value. Each message contains an indicator of the type of message, some length (possibly implied by the type if the length is fixed), and the value. If, on reading through the data you did receive, you find that the last unit is incomplete, you can assume there is more to be read. You can also make the socket into a non-blocking socket; then the recv() will fail with EAGAIN or EWOULDBLOCK if there is no data read for reading.
The recv() function returns the number of bytes read.
ioctl() with the FIONREAD option tells you how much data can currently be read without blocking.
I'm trying to receive many udp messages from one async_receive call. My messages are approx. 60 bytes long.
I'm giving an async_receive call a buffer array very similar to the boost docs but can't seem to get all the buffers filled.
char d1[128];
char d2[128];
char d3[128];
boost::array<boost::asio::mutable_buffer, 3> bufs =
{
boost::asio::buffer(d1),
boost::asio::buffer(d2),
boost::asio::buffer(d3)
};
_socket.async_receive(bufs, handler);
When my handler gets called the bytes_transferred is equal to one message length (i.e. 60).
Any thoughts on how I can get the second and third buffer populated? Also, how do I now if the second and third mutable buffer were populated?
If you want to receive multiple datagrams in a single call, you generally (regardless of Boost) need to use recvmmsg. From what I can tell, Boost does not use recvmmsg, so you would need to use it yourself with the native socket held by Boost ASIO. The advantage of doing this is that you can reduce system calls when multiple datagrams are available.
What's the difference between async_read and async_receive?
async_receive is a function that just receives into a buffer, but may not receive the amount you asked for. (It'll be equal or less, never more.)
async_read, however, will always receive the amount you asked for, as it states:
This function is used to asynchronously read a certain number of bytes of data from a stream. The function call always returns immediately. The asynchronous operation will continue until one of the following conditions is true:
The supplied buffers are full. That is, the bytes transferred is equal to the sum of the buffer sizes.
An error occurred.
The only thing the page is a bit vague on is what async_read does if it doesn't get that many bytes, and the connection closes gracefully. (Does that count as "error"?) This can probably be determined with a quick test. (async_receive, however, would just give you what it got.)
The first is a free function, the second is a member function.
Another difference is socket_base::message_flags flags parameter. See possible values, for example, in the recv(2) manual page.
Edit:
With async_receive you need to check how many bytes you got. Use it if you want to read at max N bytes, vs. exactly N bytes with async_read. Sorry, thought that was sort of obvious from boost docs.