How should QLocalSocket/QDataStream be read?
I have a program that communicates with another via named pipes using QLocalSocket and QDataStream. The recieveMessage() slot below is connected to the QLocalSocket's readyRead() signal.
void MySceneClient::receiveMessage()
{
qint32 msglength;
(*m_stream) >> msglength;
char* msgdata = new char[msglength];
int read = 0;
while (read < msglength) {
read += m_stream->readRawData(&msgdata[read], msglength - read);
}
...
}
I find that the application sometimes hangs on readRawData(). That is, it succesfully reads the 4 byte header, but then never returns from readRawData().
If I add...
if (m_socket->bytesAvailable() < 5)
return;
...to the start of this function, the application works fine (with the short test message).
I am guessing then (the documentation is very sparse) that there is some sort of deadlock occurring, and that I must use the bytesAvailable() signal to gradually build up the buffer rather than blocking.
Why is this? And what is the correct approach to reading from QLocalSocket?
Your loop blocks the event loop, so you will never get data if all did not arrive pn first read, is what causes your problem I think.
Correct approach is to use signals and slots, readyRead-signal here, and just read the available data in your slot, and if there's not enough, buffer it and return, and read more when you get the next signal.
Be careful with this alternative approach: If you are absolutely sure all the data you expect is going to arrive promptly (perhaps not unreasonable with a local socket where you control both client and server), or if the whole thing is in a thread which doesn nothing else, then it may be ok to use waitForReadyRead method. But the event loop will remain blocked until data arrives, freezing GUI for example (if in GUI thread), and generally troublesome.
Related
I use Qt creator 5.5.1 in windows 7.
The compiler is VC 2010 32 bits.
I have written a socket client. It could connect well but sometimes its readyRead() signal could not be triggered after receiving message from server. So the readMessageFromTCPServer() slot could not be triggered and the thread could not run.
void MainWindow::on_pushBtn_LoadCfg_clicked()
{
if (tcpClient == NULL)
{
tcpClient = new QTcpSocket;
tcpClient->connectToHost(ui->txtIPServer->text(),ui->txtPortServer-
>text().toInt());
QObject::connect(tcpClient,SIGNAL(readyRead()),this,
SLOT(readMessageFromTCPServer()));
}
}
void MainWindow::readMessageFromTCPServer()
{
std::string r="start";
QByteArray qba;
qba= tcpClient->readAll();
if (qba.contains(r.c_str()))
{
cout<<"thread run";
}
}
When I tried to debug this program. I put a break point at this line: Sleep(800), but sometimes this slot could not be triggered after receiving message from socket server. And I have checked that the socket is still connected, why the slot could not be triggered?
There are some errors in your code. You will only have the slot triggered once.
Get rid of those Sleep commands. There is no good reason to use it if you are doing it in the main thread.
QByteArray qba= NULL; makes no sense. QByteArray qba; is ok.
while(1) means forever. You should break the loop at some point. Actually, you do not need this loop at all. Put the code inside this loop out of it and remove the loop. When the readyRead() signal is emitted, it means that there is some data to be read. You can readAll() that data chunk, do your stuff and return.
There is no guarantee that you will get your entire message in one round. So, in some circumstances, you may get "St" in one signal and "art" in another. So, you should implement your own buffering mechanism for such a situation. It may happen on big chunks of data. If you are sure that you are getting very short packages, then it's ok to rely on the internal buffer of QTCPSocket.
I am using linux epoll in edge trigger mode.
Each time a new connection is incoming, I add the file descriptor to epoll with EPOLLIN|EPOLLOUT|EPOLLET flag. My first question is: What's the right way to check which kind of event(s) occur for each ready file descriptor after the epoll_wait returns? I mean, I see some example code e.g from https://github.com/yedf/handy/blob/master/raw-examples/epoll-et.cc line 124 do it like this:
for (int i = 0; i < n; i++) {
//...
if (events & (EPOLLIN | EPOLLERR)) {
if (fd == lfd) {
handleAccept(efd, fd);
} else {
handleRead(efd, fd);
}
} else if (events & EPOLLOUT) {
if (output_log)
printf("handling epollout\n");
handleWrite(efd, fd);
} else {
exit_if(1, "unknown event");
}
}
What caught my attention is: it uses "if and else if and else" to check which event occurs, which means if it handleRead, then it can't handleWrite at the same time. And I think this may cause loss of event in the following condition: Both socket read and write operation have meet EAGAIN and then the remote end both read and send some data, thus the epoll wait may set both EPOLLIN and EPOLLOUT, but it can only handleRead, and the data remaining in output buffer can't be sent since handleWrite is not being called.
So is the above usage wrong?
According man 7 epoll QA:
If more than one event occurs between epoll_wait(2) calls, are
they combined or reported separately?
They will be combined.
If i got it right, several events can occur on a single file descriptor between epoll_wait calls. So I think I should use multiple "if if and if" to check on by one whether readable/writable/error events occur instead of using "if and else if". I went to see how nginx epoll module do, from https://github.com/nginx/nginx/blob/953f53921505a884f3912f2d8db5217a71c0479a/src/event/modules/ngx_epoll_module.c#L867 I see the following code:
if (revents & (EPOLLERR|EPOLLHUP)) {
//...
}
if ((revents & EPOLLIN) && rev->active) {
//....
rev->handler(rev);
}
if ((revents & EPOLLOUT) && wev->active) {
//....
wev->handler(wev);
}
It seems to adhere to my thoughts of checking all EPOLLERR..,EPOLLIN,EPOLLOUT events one after another.
Then I do the same kind of thing as nginx do in my application. But What I realized after experiment is: if I add the file descriptor to epoll with EPOLLIN|EPOLLOUT|EPOLLET flag, and I didn't fill up the output buffer, I will always get EPOLLOUT flag set after epoll_wait returns due to some data arrives and this fd becomes readable, therefore redundant write_handler would be called, which is not what I expect.
I did some search and found that this situation indeed exists and not caused by any bug in my application. According to the top voted answer at epoll with edge triggered event says:
On a somewhat related note: if you register for EPOLLIN and EPOLLOUT events and assuming you never fill up the send buffer, you still get the EPOLLOUT flag set in the event returned by epoll_wait each time EPOLLIN is triggered - see https://lkml.org/lkml/2011/11/17/234 for a more detailed explanation.
And the link in this answer says:
It's doesn't mean there's an EPOLLOUT "event", it just means a message
is triggered (by the socket becoming readable) so you get a status
update. In theory the program doesn't need to be told about EPOLLOUT
here (it should be assuming the socket is writable already), but it
doesn't do any harm.
So far What I understand about epoll edge trigger mode is:
the epoll_wait return when the state of any fd being monitored has changed, e.g from nothing to read -> readable or buffer is full-> buffer can write
the epoll_wait may return one or several event(flags) for each fd in the ready list.
the flags in sturct epoll_event.events field indicate the current state of this fd. Even if we don't fill out the output buffer, the EPOLLOUT flag would be set when epoll_wait return due to readable, because the current state of the fd is just writable.
Please correct me if I am wrong.
Then my question would be: Should I maintain a flag in each connection to indicate whether EAGAIN occurs when write to output buffer, if it is not set, don't call write_handler/handleWrite in "if (events & EPOLLOUT)" branch, so that my upper layer program would not be told about EPOLLOUT here?
What a great question (since I had pretty much the same question)! I'll just summarize what I think I know now wrt to your informative question/description and your helpful links and hopefully smarter folk will correct any mistakes.
Yes, the if/else handling of event flags is definitely bogus. For sure at least two can events can arrive at effectively the same time. E.g., both the read and write sides might have become unblocked since last you called epoll_wait(). And, of course, as soon as you accept() the connection, both reading and writing suddenly become possible, so you get an "event" of EPOLLIN|EPOLLOUT.
I really didn't grok that epoll_wait() is always delivering the entire current state, rather than only the parts of the state that changed -- thanks for clearing that up. To be perhaps clearer, epoll_wait() won't return an fd unless something changed on that socket, but if something did change, it returns all the flags representing the current state. So, I found myself staring at a stream of EPOLLIN|EPOLLOUT events wondering why it was claiming there was an "output" event, even though I hadn't written anything yet. Your answer being correct: it's just telling me the output side is still writeable.
"Should I maintain a flag..." Yes, but I would imagine that in all but the most trivial situations you were probably going to end up maintaining at least one bit of "am I currently blocked" state for your readers/writers anyway. For example, if you ever want to process data in an order different than how it arrives (e.g., prioritize responses over requests to make your server more resistant to overload) you instantly have to give up the simplicity of just having the arrival of I/O drive everything. In the particular case of writing, epoll simply doesn't have enough information to notify you at the "right" time. As soon as you accept a connection, there's an event that says "you can write now"--but you probably have nothing to write if you're a server who couldn't possibly have already gotten a request from the client. epoll just can't know whether you have something to write or not, so you were always going to have to either suffer essentially "extraneous" events, or maintain your own state.
In all but the simplest cases, the socket file descriptor ends up being insufficient information for handling I/O events, so you invariably have to associate some data structure with it, or object if you prefer. So, my C++ looks something like:
nAwake = epoll_wait(epollFd, events, 100, milliseconds);
if(nAwake < 0)
{
perror("epoll_wait failed");
assert(false);
}
for(int iSocket=0; iSocket < nAwake; ++iSocket)
{
auto This = static_cast<Eventable*>(events[iSocket].data.ptr);
auto eventFlags = events[iSocket].events;
fprintf(stderr, "%s event on socket [%d] -> %s\n",
This->ClassName(), This->fd, DumpEvent(eventFlags));
This->Event(eventFlags);
}
Where Eventable is a C++ class (or derivative thereof) that has all the state needed to decide how to handle the flags epoll delivers. (Of course, this is letting the kernel store a pointer to a C++ object, requiring a design that is very clear about pointer ownership/lifetimes.)
And since you're writing low-level code on Linux, you may also care about EPOLLRDHUP. This not-highly-portable flag lets you save one call to read(). If the client (curl seems pretty good at evoking this behavior) closes its write side of the connection (sends a FIN), you normally discover that when epoll tells you EPOLLIN, but read() returns zero bytes. However, Linux maintains an extra bit to indicate your client's write side (your read side) has been closed. So, if you tell epoll you want the EPOLLRDHUP event you can use it to avoid doing a read() whose sole purpose will turn out to be telling you the writer closed their side.
Note that EPOLLIN will still be turned on whenever EPOLLRDHUP is, AFAIK. Even after you do a shutdown(fd, SHUT_RD). Another example of how you will usually be driven to maintain your own idea of the state of the connection. You care more about clients who are kind enough to do half-shutdowns if you are implementing HTTP.
When used as an edge-triggered interface, for performance reasons,
it
is possible to add the file descriptor inside the epoll interface
(EPOLL_CTL_ADD) once by specifying (EPOLLIN|EPOLLOUT).
This allows you
to avoid continuously switching between EPOLLIN and EPOLLOUT calling
epoll_ctl(2) with EPOLL_CTL_MOD.
In my application, done with Qt5, I would like to set a security system when I download data from internet. First of all, I want to check how many bytes I am downloading and abort if it is the case. I want to set a maximum limit of downloaded bytes, and abort if it is exceeded.
I am using QNetworkReply class.
First of all I checked the size() function, but at that point, I already received all the data (the request is completed), so that is not enough.
Second check, I looked into the signal downloadProgress(qint64 bytesReceived, qint64 bytesTotal). I though, I could check that and abort if bytesReceived are too big. But I am not sure for two motivations: first, as written in the documentation
Note that the values of both bytesReceived and bytesTotal may be different from size(), the total number of bytes obtained through read() or readAll(), or the value of the header(ContentLengthHeader). The reason for that is that there may be protocol overhead or the data may be compressed during the download.
Second, I don't know when I will receive this signal....maybe it will be too late...
Finally, I checked setReadBufferSize(qint64 size). It seems a good solution, but I am not sure about the following two lines in the documentation:
first,
QNetworkReply will try to stop reading from the network once this buffer is full (i.e., bytesAvailable() returns size or more)
What does it mean try? should I rely on that?
second,
Unlike QAbstractSocket::setReadBufferSize(), QNetworkReply cannot guarantee precision in the read buffer size. That is, bytesAvailable() can return more than size.
so, it seems I can not rely on a precise limit to be set.
Someone can suggest me the best way to implement it?
The best solution I found is to use the readyRead signal (QNetworkReply derives from QIODevice), and then implement the security check by myself in a slot:
void onReadyRead()
{
for (;;)
{
QByteArray currentDataRead = httpReply->read(100); // Just read some bytes
if ( currentDataRead.isEmpty() )
return;
myResponseBody.append(currentDataRead);
if (myResponseBody.size() > myMaxResponseSize) // check my limit
{
qDebug() << "Error: response size bigger than expected - aborting";
httpReply->abort();
}
}
}
Note: the inifite for(;;) loop and the check if ( currentDataRead.isEmpty() ) return; are needed in order to not leave some bytes out at the end, because we are inside onReadyRead that is a slot connected to readyRead and from the documentation:
readyRead() is not emitted recursively; if you reenter the event loop or call waitForReadyRead() inside a slot connected to the readyRead() signal, the signal will not be reemitted (although waitForReadyRead() may still return true).
--------------------
PS: I am still open to better solutions, possibly native in Qt
I am streaming data as a string over UDP, into a Socket class inside Unreal engine. This is threaded, and runs in the background.
My read function is:
float translate;
void FdataThread::ReceiveUDP()
{
uint32 Size;
TArray<uint8> ReceivedData;
if (ReceiverSocket->HasPendingData(Size))
{
int32 Read = 0;
ReceivedData.SetNumUninitialized(FMath::Min(Size, 65507u));
ReceiverSocket->RecvFrom(ReceivedData.GetData(), ReceivedData.Num(), Read, *targetAddr);
}
FString str = FString(bytesRead, UTF8_TO_TCHAR((const UTF8CHAR *)ReceivedData));
translate = FCString::Atof(*str);
}
I then call the translate variable from another class, on a Tick, or timer.
My test case sends an incrementing number from another application.
If I print this number from inside the above Read function, it looks as expected, counting up incrementally.
When i print it from the other thread, it is missing some of the numbers.
I believe this is because I call it on the Tick, so it misses out some data due to processing time.
My question is:
Is there a way to queue the incoming data, so that when i pull the value, it is the next incremental value and not the current one? What is the best way to go about this?
Thank you, please let me know if I have not been clear.
Is this the complete code? ReceivedData isn't used after it's filled with data from the socket. Instead, an (in this code) undefined variable 'buffer' is being used.
Also, it seems that the while loop could run multiple times, overwriting old data in the ReceivedData buffer. Add some debugging messages to see whether RecvFrom actually reads all bytes from the socket. I believe it reads only one 'packet'.
Finally, especially when you're using UDP sockets over the network, note that the UDP protocol isn't guaranteed to actually deliver its packets. However, I doubt this is causing your problems if you're using it on a single computer or a local network.
Your read loop doesn't make sense. You are reading and throwing away all datagrams but the last in any given sequence that happen to be in the socket receive buffer at the same time. The translate call should be inside the loop, and the loop should be while(true), or while (running), or similar.
I am trying to connect a signal and a slot in C++ using the boost libraries. My code currently opens a file and reads data from it. However, I am trying to improve the code so that it can read and analyze data in real time using a serial port. What I would like to do is have the analyze functions called only once there is data available in the serial port.
How would I go about doing this? I have done it in Qt before, however I cannot use signals and slots in Qt because this code does not use their moc tool.
Your OS (Linux) provides you with the following mechanism when dealing with the serial port.
You can set your serial port to noncanonical mode (by unsetting ICANON flag in termios structure). Then, if MIN and TIME parameters in c_cc[] are zero, the read() function will return if and only if there is new data in the serial port input buffer (see termios man page for details). So, you may run a separate thread responsible for getting the incoming serial data:
ssize_t count, bytesReceived = 0;
char myBuffer[1024];
while(1)
{
if (count = read(portFD,
myBuffer + bytesReceived,
sizeof(myBuffer)-bytesReceived) > 0)
{
/*
Here we check the arrived bytes. If they can be processed as a complete message,
you can alert other thread in a way you choose, put them to some kind of
queue etc. The details depend greatly on communication protocol being used.
If there is not enough bytes to process, you just store them in buffer
*/
bytesReceived += count;
if (MyProtocolMessageComplete(myBuffer, bytesReceived))
{
ProcessMyData(myBuffer, bytesReceived);
AlertOtherThread(); //emit your 'signal' here
bytesReceived = 0; //going to wait for next message
}
}
else
{
//process read() error
}
}
The main idea here is that the thread calling read() is going to be active only when new data arrives. The rest of the time OS will keep this thread in wait state. Thus it will not consume CPU time. It is up to you how to implement the actual signal part.
The example above uses regular read system call to get data from port, but you can use the boost class in the same manner. Just use syncronous read function and the result will be the same.