I have the following code to measure the total time taken for a socket write from client to server. (assuming that the call back method invocation is done on successful write of the data to the destination socket (TCP-ACK received)) Does this ensure that - this time is the actual "network time" for the data transfer ?
void on_successful_read_from_client(const boost::system::error_code& error,
const size_t& bytes_transferred)
{
if (!error)
{
m_telnet_server_write_time = posix_time::microsec_clock::universal_time();
async_write(telnet_server,
boost::asio::buffer(data_from_device_,bytes_transferred),
boost::bind(&bridge::on_successful_send_to_server,
shared_from_this(),
boost::asio::placeholders::error));
}
else
close();
}
void on_successful_send_to_server(const boost::system::error_code& error)
{
if (!error)
{
posix_time::ptime now = posix_time::microsec_clock::universal_time();
if ((now - m_telnet_server_write_time).total_milliseconds() > 0)
{
std::ostringstream log;
log << "Time Taken for server write: " << (now - m_telnet_server_write_time).total_milliseconds() << " ms";
write_log(log.str());
}
No I don't think it does. I think it measures the time needed to get the data into the buffers in the network stack of your OS, but not the time taken for the data to be transferred to and read by the program at the other end of the connection. The only way you can do that is to have accurate clocks at both ends and have the send time sent to the recipient as part of the message so that it can do the elapsed time calculation.
it measures the time to put the data into send buffers in your network stack.
if you want to calculate the "network time", use icmp::socket, just like "ping" command does.
here is the example in boost documents
boost examples
search "ICMP" in this page
Does this ensure that - this time is the actual "network time" for the data transfer?
No. It measures the time to transfer the data into the socket send buffer. If the socket send buffer was full enough such that not all the data fitted into it immediately, it also measures the time to drain the socket send buffer sufficiently to accomodate all the new data. It does not measure time to send all this data to the server in any way.
assuming that the call back method invocation is done on successful write of the data to the destination socket (TCP-ACK received)
It isn't. It is called when all the data has been transferred into the socket send buffer.
Related
I am trying to receive a large amount of data using a boost::beast::websocket, fed by another boost::beast::websocket. Normally, this data is sent to a connected browser but I'd like to set up a purely C++ unit test validating certain components of the traffic. I set the auto fragmentation to true from the sender with a max size of 1MB but after a few messages, the receiver spits out:
Read 258028 bytes of binary
Read 1547176 bytes of binary
Read 168188 bytes of binary
"Failed read: The WebSocket message exceeded the locally configured limit"
Now, I should have no expectation that a fully developed and well supported browser should exhibit the same characteristics as my possibly poorly architected unit test, which it does not. The browser has no issue reading 25MB messages over the websocket. My boost::beast::websocket on the other hand hits a limit.
So before I go down a rabbit hole, I'd like to see if anyone has any thoughts on this. My read sections looks like this:
void on_read(boost::system::error_code ec, std::size_t bytes_transferred)
{
boost::ignore_unused(bytes_transferred);
if (ec)
{
m_log.error("Failed read: " + ec.message());
// Stop the websocket
stop();
return;
}
std::string data(boost::beast::buffers_to_string(m_buffer.data()));
// Yes I know this looks dangerous. The sender always sends as binary but occasionally sends JSON
if (data.at(0) == '{')
m_log.debug("Got message: " + data);
else
m_log.debug("Read " + utility::to_string(m_buffer.data().buffer_bytes()) + " of binary data");
// Do the things with the incoming doata
for (auto&& callback : m_read_callbacks)
callback(data);
// Toss the data
m_buffer.consume(bytes_transferred);
// Wait for some more data
m_websocket.async_read(
m_buffer,
std::bind(
&WebsocketClient::on_read,
shared_from_this(),
std::placeholders::_1,
std::placeholders::_2));
}
I saw in a separate example that instead of doing an async read, you can do a for/while loop reading some data until the message is done (https://www.boost.org/doc/libs/1_67_0/libs/beast/doc/html/beast/using_websocket/send_and_receive_messages.html). Would this be the right approach for an always open websocket that could send some pretty massive messages? Would I have to send some indicator to the client that the message is indeed done? And would I run into the exceeded buffer limit issue using this approach?
If your use pattern is fixed:
std::string data(boost::beast::buffers_to_string(m_buffer.data()));
And then, in particular
callback(data);
Then there will be no use at all reading block-wise, since you will be allocating the same memory anyways. Instead, you can raise the "locally configured limit":
ws.read_message_max(20ull << 20); // sets the limit to 20 miB
The default value is 16 miB (as of boost 1.75).
Side Note
You can probably also use ws.got_binary() to detect whether the last message received was binary or not.
Summary: Some of the memory within the TCP socket to be overwritten by other incoming data.
Application:
A client/server system that utilizes TCP within Qt (QTcpSocket and QTcpServer). The client request a frame from the server(just a simple string message), and the response (Server -> Client) which consists of that frame (614400 bytes for testing purposes). Frame sizes are established in advance and are fixed.
Implementation Details:
From the guarantees of the TCP protocol (Server -> Client), I know that I should be able to read the 614400 bytes from the socket and that they are in order. If any either of these two things fails, the connection must have failed.
Important Code:
Assuming the socket is connected.
This code requests a frame from the server. Known as the GetFrame() function.
// Prompt the server to send a frame over
if(socket->isWritable() && !is_receiving) { // Validate that socket is ready
is_receiving = true; // Forces only one request to go out at a time
qDebug() << "Getting frame from socket..." << image_no;
int written = SafeWrite((char*)"ReadyFrame"); // Writes then flushes the write buffer
if (written == -1) {
qDebug() << "Failed to write...";
return temp_frame.data();
}
this->SocketRead();
is_receiving = false;
}
qDebug() << image_no << "- Image Received";
image_no ++;
return temp_frame.data();
This code waits for the frame just requested to be read. This is the SocketRead() function
size_t byte_pos = 0;
qint64 bytes_read = 0;
do {
if (!socket->waitForReadyRead(500)) { // If it timed out return existing frame
if (!(socket->bytesAvailable() > 0)) {
qDebug() << "Timed Out" << byte_pos;
break;
}
}
bytes_read = socket->read((char*)temp_frame.data() + byte_pos, frame_byte_size - byte_pos);
if (bytes_read < 0) {
qDebug() << "Reading Failed" << bytes_read << errno;
break;
}
byte_pos += bytes_read;
} while (byte_pos < frame_byte_size && is_connected); // While we still have more pixels
qDebug() << "Finished Receiving Frame: " << byte_pos;
As shown in the code above, I read until the frame is fully received (where the number of bytes read is equal to the number of bytes in the frame).
The issue that I'm having is that the QTcpSocket read operation is skipping bytes in ways that are not in line with the guarantees of the TCP protocol. Since I skip bytes I end up not reaching the end of the while loop and just "Time Out". Why is this happening?
What I have done so far:
The data that the server sends is directly converted into uint16_t (short) integers which are used in other parts of the client. I have changed the server to simply output data that just counts up adding one for each number sent. Since the data type is uint16_t and the number of bytes exceeds that maximum number for that integer type, the int-16's will loop every 65535.
This is a data visualization software so this debugging configuration (on the client side) leads to something like this:
I have determined (and as you can see a little at the bottom of the graphic) that some bytes are being skipped. In the memory of temp_frame it is possible to see the exact point at which the memory skipped:
Under correct circumstances, this should count up sequentially.
From Wireshark and following this specific TCP connection I have determined that all of the bytes are in fact arriving (all 6114400), and that all the numbers are in order (I used a python script to ensure counting was sequential).
This is work on an open source project so this is the whole code base for the client.
Overall, I don't see how I could be doing something wrong in this solution, all I am doing is reading from the socket in the standard way.
Caveat: This isn't a definitive answer to your problem, but some things to try (it's too large for a comment).
With (e.g.) GigE, your data rate is ~100MB/s. With a [total] amount of kernel buffer space of 614400, this will be refilled ~175 times per second. IMO, this is still too small. When I've used SO_RCVBUF [for a commercial product], I've used a minimum of 8MB. This allows a wide(er) margin for task switch delays.
Try setting something huge like 100MB to eliminate this as a factor [during testing/bringup].
First, it's important to verify that the kernel and NIC driver can handle the throughput/latency.
You may be getting too many interrupts/second and the ISR prolog/epilog overhead may be too high. The NIC card driver can implement polled vs interrupt driver with NAPI for ethernet cards.
See: https://serverfault.com/questions/241421/napi-vs-adaptive-interrupts
See: https://01.org/linux-interrupt-moderation
You process/thread may not have high enough priority to be scheduled quickly.
You can use the R/T scheduler with sched_setscheduler, SCHED_RR, and a priority of (e.g.) 8. Note: going higher than 11 kills the system because at 12 and above you're at a higher priority than most internal kernel threads--not a good thing.
You may need to disable IRQ balancing and set the IRQ affinity to a single CPU core.
You can then set your input process/thread locked to that core [with sched_setaffinity and/or pthread_setaffinity].
You might need some sort of "zero copy" to bypass the kernel copying from its buffers into your userspace buffers.
You can mmap the kernel socket buffers with PACKET_MMAP. See: https://sites.google.com/site/packetmmap/
I'd be careful about the overhead of your qDebug output. It looks like an iostream type implementation. The overhead may be significant. It could be slowing things down significantly.
That is, you're not measuring the performance of your system. You're measuring the performance of your system plus the debugging code.
When I've had to debug/trace such things, I've used a [custom] "event" log implemented with an in-memory ring queue with a fixed number of elements.
Debug calls such as:
eventadd(EVENT_TYPE_RECEIVE_START,some_event_specific_data);
Here eventadd populates a fixed size "event" struct with the event type, event data, and a hires timestamp (e.g. struct timespec from clock_gettime(CLOCK_MONOTONIC,...).
The overhead of each such call is quite low. The events are just stored in the event ring. Only the last N are remembered.
At some point, your program triggers a dump of this queue to a file and terminates.
This mechanism is similar to [and modeled on] a H/W logic analyzer. It is also similar to dtrace
Here's a sample event element:
struct event {
long long evt_tstamp; // timestamp
int evt_type; // event type
int evt_data; // type specific data
};
I'm using the boost::asio::async_read_until function to read from a serial port in Windows 10. The delimiter is a Regex pattern. It works as expected as long as the data recieved is not larger than 512 bytes.
If the data received is larger than 512 bytes, it is simply truncated and the "readComplete" function will not be called again. However if I send more data, 1 byte is enough, the missing data is received together with the new data.
I have used the same implementation on a tcp/socket and that works flawlessly. Is there any limit in the native serial interface in Windows causing this behaviour?
EDIT 1: I have noted that if the baud rate is lowered from 115200 to 28800 no data is missing.
// from .h-file: boost::asio::streambuf streamBuf_;
void RS232Instrument::readAsyncChars()
{
boost::asio::async_read_until(
serial_,
streamBuf_,
boost::regex(regexStr_.substr(6, regexStr_.length() - 7)),
boost::bind(
&RS232Instrument::readComplete,
this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
void RS232Instrument::readComplete(const boost::system::error_code& error, size_t bytes_transferred)
{
if(error)
{
// Error handling
}
else
{
std::string rawStr(
boost::asio::buffers_begin(streamBuf_.data()),
boost::asio::buffers_begin(streamBuf_.data()) + bytes_transferred);
// Log the data in rawStr....
// Remove data from beginning until all data sent to log
streamBuf_.consume(bytes_transferred);
if(abort_ == false)
{
readAsyncChars();
}
}
}
Since I have found out was caused this problem I'll answer the question myself.
I had left out some code above for the sake of clarity, code which I did not realise was actually the problem.
Example of code left out:
LOG_DEBUG("Rs232Data received");
I use the boost:log functionality and I have added more "sinks" to the the log framework. The sink used in this case logs to a vector in ram and prints to console when triggered from user input.
It turns out that the log framework consumes about 1 ms before the "consume" function in the sink is called. That is enough to cause loss of data from the serial port when using async_read_until.
Lessons learned: Do not call any time consuming tasks in the handler function in async_read_until
I'm using boost::asio::write() to write data from a buffer to a com-Port. It's a serial port with a baud rate 115200 which means (as far as my understanding goes) that I can write effectively 11520 byte/s or 11,52KB/s data to the socket.
Now I'm having a quite big chunk of data (10015 bytes) which i want to write. I think that this should take little less than a second to really write on the port. But boost::asio::write() returns already 300 microseconds after the call with the transferred bytes 10015. I think this is impossible with that baud rate?
So my question is what is it actually doing? Really writing it to the port, or just some other kind of buffer maybe, which later writes it to the port.
I'd like the write() to only return after all the bytes have really been written to the port.
EDIT with code example:
The problem is that i always run into the timeout for the future/promise because it takes alone more than 100ms to send the message, but I think the timer should only start after the last byte is sent. Because write() is supposed to block?
void serial::write(std::vector<uint8_t> message) {
//create new promise for the request
promise = new boost::promise<deque<uint8_t>>;
boost::unique_future<deque<uint8_t>> future = promise->get_future();
// --- Write message to serial port --- //
boost::asio::write(serial_,boost::asio::buffer(message));
//wait for data or timeout
if (future.wait_for(boost::chrono::milliseconds(100))==boost::future_status::timeout) {
cout << "ACK timeout!" << endl;
//delete pointer and set it to 0
delete promise;
promise=nullptr;
}
//delete pointer and set it to 0 after getting a message
delete promise;
promise=nullptr;
}
How can I achieve this?
Thanks!
In short, boost::asio::write() blocks until all data has been written to the stream; it does not block until all data has been transmitted. To wait until data has been transmitted, consider using tcdrain().
Each serial port has both a receive and transmit buffer within kernel space. This allows the kernel to buffer received data if a process cannot immediately read it from the serial port, and allows data written to a serial port to be buffered if the device cannot immediately transmit it. To block until the data has been transmitted, one could use tcdrain(serial_.native_handle()).
These kernel buffers allow for the write and read rates to exceed that of the transmit and receive rates. However, while the application may write data at a faster rate than the serial port can transmit, the kernel will transmit at the appropriate rates.
I am having difficulties in implementing a simple TCP server. The following code is taken from boost::asio examples, "Http Server 1" to be precise.
void connection::start() {
socket_.async_read_some(
boost::asio::buffer(buffer_),
boost::bind(
&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred
)
);
}
void connection::handle_read(const boost::system::error_code& e, std::size_t bytes_transferred) {
if (!e && bytes_transferred) {
std::cout << " " << bytes_transferred <<"b" << std::endl;
data_.append(buffer_.data(), buffer_.data()+bytes_transferred);
//(1) what here?
socket_.async_read_some(
boost::asio::buffer(buffer_),
boost::bind(
&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred
)
);
}
else// if (e != boost::asio::error::operation_aborted)
{
std::cout << data_ << std::endl;
connection_manager_.stop(shared_from_this());
}
}
In the original code the buffer_ is big enough to keep the entire request. It's not what I need. I've changed the size to 32bytes.
The server compiles and listens at port 80 of localhost, so I try to connect to it via my web browser.
Now if the statement (1) is commented-out, then only the first 32bytes of the request are read and the connection hangs. Web browser keeps waiting for the response, the server does.. I dont know what.
If (1) is uncommented, then the entire request is read (and appeded to data_), but it never stops - I have to cancel the request in my browser and only then does the else { } part run - I see my request on stdout.
Question 1: How should I handle a large request?
Question 2: How should I cache the request (currently I append the buffer to a string)?
Question 3: How can I tell that the request is over? In HTTP there always is a response, so my web-browser keeps waiting for it and doesnt close the connection, but how can my server know that the request is over (and perhaps close it or reply some "200 OK")?
Suppose browser send you 1360 bytes of data, you say asio to read some data into your buffer that you say it only have 32 bytes.
then first time that you call it your handler will be called with 32 bytes start of data. here if you comment (1) then browser try to send rest of its data(actually browser already sent it and it is in the OS buffer that wait for you to peek it from there) and you are possibly blocked behind io_service::run for some miracle!!
if you uncomment (1) as you say your loop started, you read first block, then next and another and ... until the data that the browser sent finished, but after that when you say asio to read some more data it will wait for some more data that never come from the browser( since browser already sent its information and is waiting for your answer ) and when you cancel the request from the browser, it will close its socket and then your handler will be called whith an error that say I can't read more data, since the connection is closed.!!
but what you should do here to make it work is: you should learn HTTP format and thus know what is the data that your browser sent to you and provide a good answer for it and then your communication with the client will be proceeded. in this case end of buffer is \r\n\r\n and when you see it you shouldn't read any more data, you should process what you read till now and then send a response to the browser.