I know that TCP guarantees that all packets will arrive. But can a packet be separated into 2 or more? I am using Qt with class QTcpSocket, and I want to know is ReadyRead() signal is emitted only when full packet arrives. Or in other words, is there any sense to send packet size in first bytes and then wait in loop until all bytes had arrived? Or I can just call socket->readAll() and I must get one full packet?
If a large amount of data is sent, the packet can arrive in separate parts. Alternatively, multiple messages can be received in one readyRead slot.
It's good practice to control this by setting the first byte(s) to the number of bytes that will be sent. Then, in readyRead, you read the first bytes and append the data to a buffer until the expected amount of data has been received.
In receiving data, this also means that if multiple messages are received in one call to readyRead(), you can know where the first message ends and the next one begins.
Here's an example of a client that receives data in a readyRead function()
void MyClass::readyRead()
{
// m_pConnection is a QTcpSocket
while(m_pConnection->bytesAvailable())
{
QByteArray buffer;
int dataSize;
m_pConnection->read((char*)&dataSize, sizeof(int));
buffer = m_pConnection->read(dataSize);
while(buffer.size() < dataSize) // only part of the message has been received
{
m_pConnection->waitForReadyRead(); // alternatively, store the buffer and wait for the next readyRead()
buffer.append(m_pConnection->read(dataSize - buffer.size())); // append the remaining bytes of the message
}
QString msg(buffer); // data in this case is JSON, so we can use a QString
emit Log(QString("\tMessage Received: %1").arg(msg));
// Do something with the message
ProcessMessage(msg);
}
}
Related
I have server-client Qt application, where client sends data packets to server and server reads them at a set time intervals. It happens that client sends data faster than server can read thus filling all the memory on the server side. I am using QAbstractSocket::setReadBufferSize(size) to set max read buffer size on the server side and when it fills up, socket data transferring stops, and data is buffered on client side, which is what i want, but the problem is when server's QTcpSocket's internal read buffer frees up (is not full anymore), data transfer between client and server does not resume.
I've tried to use QAbstractSocket::resume() which seems to work, but Qt5.10 documentation says:
Continues data transfer on the socket. This method should only be used
after the socket has been set to pause upon notifications and a
notification has been received. The only notification currently
supported is QSslSocket::sslErrors(). Calling this method if the
socket is not paused results in undefined behavior.
I feel like I should not use that function in this situation, but is there any other solution? How do i know if socket is paused? Why data transfer does not continue automaticaly when QTcpSocket's internal read buffer is not full anymore?
EDIT 1 :
I have downloaded Qt(5.10.0) sources and pdb's to debug this situation and I can see that QAbstractSocket::readData() internal function have line "d->socketEngine->setReadNotificationEnabled(true)" which re-enables data transfering, but QAbstractSocket::readData() gets called only when QTcpSocket internal read buffer is empty (qiodevice.cpp; QIODevicePrivate::read(); line 1176) and in My situation it is never empty, because I read it only when it has enough data for complete packet.
Shouldn't QAbstractSocket::readData() be called when read buffer is not full anymore and not when it's completely empty? Or maybe i do something wrong?
Found a Workaround!
In Qt5.10 sources i can clearly see that QTcpSpcket internal read notifications is disabled (qabstractsocket.cpp; bool QAbstractSocketPrivate::canReadNotification(); line 697) when read buffer is full and to enable read notifications you need to read all buffer to make it empty OR use QAbstractSocket::setReadBufferSize(newSize) which internally enables read notifications WHEN newSize is not 0 (unlimited) and not equal to oldSize (qabstractsocket.cpp; void QAbstractSocket::setReadBufferSize(qint64 size); line 2824).
Here's a short function for that:
QTcpSocket socket;
qint64 readBufferSize; // Current max read buffer size.
bool flag = false; // flag for changing max read buffer size.
bool isReadBufferLimitReached = false;
void App::CheckReadBufferLimitReached()
{
if (readBufferSize <= socket.bytesAvailable())
isReadBufferLimitReached = true;
else if (isReadBufferLimitReached)
{
if (flag)
{
readBufferSize++;
flag = !flag;
}
else
{
readBufferSize--;
flag = !flag;
}
socket.setReadBufferSize(readBufferSize);
isReadBufferLimitReached = false;
}
}
In the function which reads data from QTcpSocket at the set intervals, BEFORE reading data, I call this function, which checks if read buffer is full and sets isReadBufferLimitReached if true. Then I read needed amount of data from QTcpSocket and AT THE END I call that function again, which, if buffer were full before, calls QAbstractSocket::setReadBufferSize(size) to set new buffer size and enable internal read notifications. Changing read buffer size by +/-1 should be safe, because you read at least 1 byte from socket.
I know that TCP guarantees that all packets will arrive. But can a packet be separated into 2 or more? I am using Qt with class QTcpSocket, and I want to know is ReadyRead() signal is emitted only when full packet arrives. Or in other words, is there any sense to send packet size in first bytes and then wait in loop until all bytes had arrived? Or I can just call socket->readAll() and I must get one full packet?
If a large amount of data is sent, the packet can arrive in separate parts. Alternatively, multiple messages can be received in one readyRead slot.
It's good practice to control this by setting the first byte(s) to the number of bytes that will be sent. Then, in readyRead, you read the first bytes and append the data to a buffer until the expected amount of data has been received.
In receiving data, this also means that if multiple messages are received in one call to readyRead(), you can know where the first message ends and the next one begins.
Here's an example of a client that receives data in a readyRead function()
void MyClass::readyRead()
{
// m_pConnection is a QTcpSocket
while(m_pConnection->bytesAvailable())
{
QByteArray buffer;
int dataSize;
m_pConnection->read((char*)&dataSize, sizeof(int));
buffer = m_pConnection->read(dataSize);
while(buffer.size() < dataSize) // only part of the message has been received
{
m_pConnection->waitForReadyRead(); // alternatively, store the buffer and wait for the next readyRead()
buffer.append(m_pConnection->read(dataSize - buffer.size())); // append the remaining bytes of the message
}
QString msg(buffer); // data in this case is JSON, so we can use a QString
emit Log(QString("\tMessage Received: %1").arg(msg));
// Do something with the message
ProcessMessage(msg);
}
}
I'm using boost::asio::write() to write data from a buffer to a com-Port. It's a serial port with a baud rate 115200 which means (as far as my understanding goes) that I can write effectively 11520 byte/s or 11,52KB/s data to the socket.
Now I'm having a quite big chunk of data (10015 bytes) which i want to write. I think that this should take little less than a second to really write on the port. But boost::asio::write() returns already 300 microseconds after the call with the transferred bytes 10015. I think this is impossible with that baud rate?
So my question is what is it actually doing? Really writing it to the port, or just some other kind of buffer maybe, which later writes it to the port.
I'd like the write() to only return after all the bytes have really been written to the port.
EDIT with code example:
The problem is that i always run into the timeout for the future/promise because it takes alone more than 100ms to send the message, but I think the timer should only start after the last byte is sent. Because write() is supposed to block?
void serial::write(std::vector<uint8_t> message) {
//create new promise for the request
promise = new boost::promise<deque<uint8_t>>;
boost::unique_future<deque<uint8_t>> future = promise->get_future();
// --- Write message to serial port --- //
boost::asio::write(serial_,boost::asio::buffer(message));
//wait for data or timeout
if (future.wait_for(boost::chrono::milliseconds(100))==boost::future_status::timeout) {
cout << "ACK timeout!" << endl;
//delete pointer and set it to 0
delete promise;
promise=nullptr;
}
//delete pointer and set it to 0 after getting a message
delete promise;
promise=nullptr;
}
How can I achieve this?
Thanks!
In short, boost::asio::write() blocks until all data has been written to the stream; it does not block until all data has been transmitted. To wait until data has been transmitted, consider using tcdrain().
Each serial port has both a receive and transmit buffer within kernel space. This allows the kernel to buffer received data if a process cannot immediately read it from the serial port, and allows data written to a serial port to be buffered if the device cannot immediately transmit it. To block until the data has been transmitted, one could use tcdrain(serial_.native_handle()).
These kernel buffers allow for the write and read rates to exceed that of the transmit and receive rates. However, while the application may write data at a faster rate than the serial port can transmit, the kernel will transmit at the appropriate rates.
I'm using QT. I need to broadcast data, so I try to use QUdpSocket. But data can be too big(after writeDatagram QUdpSocket::error returns DatagramTooLargeError). So I split data and call writeDatagram several times to the parts. But Received socket receive data only once, only first packet. Receive code is
connect(&m_socketReceiver, &QUdpSocket::readyRead, this, &LocalNetSharing::onDataRead);
void LocalNetSharing::onDataRead()
{
while (m_socketReceiver.hasPendingDatagrams())
{
QByteArray datagram;
datagram.resize(m_socketReceiver.pendingDatagramSize());
m_socketReceiver.readDatagram(datagram.data(), datagram.size());
//process data
}
}
From the Qt documentation about QUdpSocket Class :
Note: An incoming datagram should be read when you receive the
readyRead() signal, otherwise this signal will not be emitted for the
next datagram.
So it seems that you are not reading the entire datagram in each call of onDataRead.
You don't specify host and port in readDatagram. I am not sure if it is the reason but the correct form is :
while (m_socketReceiver.hasPendingDatagrams())
{
QByteArray datagram;
datagram.resize(m_socketReceiver.pendingDatagramSize());
m_socketReceiver.readDatagram(datagram.data(), datagram.size(), host, &port);
//process data
}
I have a server that receives a compressed string (compressed with zlib) from a client, and I was using async_receive from the boost::asio library to receive this string, it turns out however that there is no guarantee that all bytes will be received, so I now have to change it to async_read. The problem I face is that the size of the bytes received is variable, so I am not sure how to use async_read without knowing the number of bytes to be received. With the async_receive I just have a boost::array<char, 1024>, however this is a buffer that is not necessarily filled completely.
I wondered if anyone can suggest a solution where I can use async_read even though I do not know the number of bytes to be received in advance?
void tcp_connection::start(boost::shared_ptr<ResolverQueueHandler> queue_handler)
{
if (!_queue_handler.get())
_queue_handler = queue_handler;
std::fill(buff.begin(), buff.end(), 0);
//socket_.async_receive(boost::asio::buffer(buff), boost::bind(&tcp_connection::handle_read, shared_from_this(), boost::asio::placeholders::error));
boost::asio::async_read(socket_, boost::asio::buffer(buff), boost::bind(&tcp_connection::handle_read, shared_from_this(), boost::asio::placeholders::error));
}
buff is a boost::array<char, 1024>
How were you expecting to do this using any other method?
There are a few general methods to sending data of variable sizes in an async manor:
By message - meaning that you have a header that defines the length of the expected message followed by a body which contains data of the specified length.
By stream - meaning that you have some marker (and this is very broad) method of knowing when you've gotten a complete packet.
By connection - each complete packet of data is sent in a single connection which is closed once the data is complete.
So can your data be parsed, or a length sent etc...
Use async_read_until and create your own match condition, or change your protocol to send a header including the number of bytes to expect in the compressed string.
A single IP packet is limited to an MTU size of ~1500 bytes, and yet still you can download gigabyte-large files from your favourite website, and watch megabyte-sized videos on YouTube.
You need to send a header indicating the actual size of the raw data, and then receive the data by pieces on smaller chunks until you finish receiving all the bytes.
For example, when you download a large file over HTTP, there is a field on the header indicating the size of the file: Content-Length:.