In my application, done with Qt5, I would like to set a security system when I download data from internet. First of all, I want to check how many bytes I am downloading and abort if it is the case. I want to set a maximum limit of downloaded bytes, and abort if it is exceeded.
I am using QNetworkReply class.
First of all I checked the size() function, but at that point, I already received all the data (the request is completed), so that is not enough.
Second check, I looked into the signal downloadProgress(qint64 bytesReceived, qint64 bytesTotal). I though, I could check that and abort if bytesReceived are too big. But I am not sure for two motivations: first, as written in the documentation
Note that the values of both bytesReceived and bytesTotal may be different from size(), the total number of bytes obtained through read() or readAll(), or the value of the header(ContentLengthHeader). The reason for that is that there may be protocol overhead or the data may be compressed during the download.
Second, I don't know when I will receive this signal....maybe it will be too late...
Finally, I checked setReadBufferSize(qint64 size). It seems a good solution, but I am not sure about the following two lines in the documentation:
first,
QNetworkReply will try to stop reading from the network once this buffer is full (i.e., bytesAvailable() returns size or more)
What does it mean try? should I rely on that?
second,
Unlike QAbstractSocket::setReadBufferSize(), QNetworkReply cannot guarantee precision in the read buffer size. That is, bytesAvailable() can return more than size.
so, it seems I can not rely on a precise limit to be set.
Someone can suggest me the best way to implement it?
The best solution I found is to use the readyRead signal (QNetworkReply derives from QIODevice), and then implement the security check by myself in a slot:
void onReadyRead()
{
for (;;)
{
QByteArray currentDataRead = httpReply->read(100); // Just read some bytes
if ( currentDataRead.isEmpty() )
return;
myResponseBody.append(currentDataRead);
if (myResponseBody.size() > myMaxResponseSize) // check my limit
{
qDebug() << "Error: response size bigger than expected - aborting";
httpReply->abort();
}
}
}
Note: the inifite for(;;) loop and the check if ( currentDataRead.isEmpty() ) return; are needed in order to not leave some bytes out at the end, because we are inside onReadyRead that is a slot connected to readyRead and from the documentation:
readyRead() is not emitted recursively; if you reenter the event loop or call waitForReadyRead() inside a slot connected to the readyRead() signal, the signal will not be reemitted (although waitForReadyRead() may still return true).
--------------------
PS: I am still open to better solutions, possibly native in Qt
Related
I am using linux epoll in edge trigger mode.
Each time a new connection is incoming, I add the file descriptor to epoll with EPOLLIN|EPOLLOUT|EPOLLET flag. My first question is: What's the right way to check which kind of event(s) occur for each ready file descriptor after the epoll_wait returns? I mean, I see some example code e.g from https://github.com/yedf/handy/blob/master/raw-examples/epoll-et.cc line 124 do it like this:
for (int i = 0; i < n; i++) {
//...
if (events & (EPOLLIN | EPOLLERR)) {
if (fd == lfd) {
handleAccept(efd, fd);
} else {
handleRead(efd, fd);
}
} else if (events & EPOLLOUT) {
if (output_log)
printf("handling epollout\n");
handleWrite(efd, fd);
} else {
exit_if(1, "unknown event");
}
}
What caught my attention is: it uses "if and else if and else" to check which event occurs, which means if it handleRead, then it can't handleWrite at the same time. And I think this may cause loss of event in the following condition: Both socket read and write operation have meet EAGAIN and then the remote end both read and send some data, thus the epoll wait may set both EPOLLIN and EPOLLOUT, but it can only handleRead, and the data remaining in output buffer can't be sent since handleWrite is not being called.
So is the above usage wrong?
According man 7 epoll QA:
If more than one event occurs between epoll_wait(2) calls, are
they combined or reported separately?
They will be combined.
If i got it right, several events can occur on a single file descriptor between epoll_wait calls. So I think I should use multiple "if if and if" to check on by one whether readable/writable/error events occur instead of using "if and else if". I went to see how nginx epoll module do, from https://github.com/nginx/nginx/blob/953f53921505a884f3912f2d8db5217a71c0479a/src/event/modules/ngx_epoll_module.c#L867 I see the following code:
if (revents & (EPOLLERR|EPOLLHUP)) {
//...
}
if ((revents & EPOLLIN) && rev->active) {
//....
rev->handler(rev);
}
if ((revents & EPOLLOUT) && wev->active) {
//....
wev->handler(wev);
}
It seems to adhere to my thoughts of checking all EPOLLERR..,EPOLLIN,EPOLLOUT events one after another.
Then I do the same kind of thing as nginx do in my application. But What I realized after experiment is: if I add the file descriptor to epoll with EPOLLIN|EPOLLOUT|EPOLLET flag, and I didn't fill up the output buffer, I will always get EPOLLOUT flag set after epoll_wait returns due to some data arrives and this fd becomes readable, therefore redundant write_handler would be called, which is not what I expect.
I did some search and found that this situation indeed exists and not caused by any bug in my application. According to the top voted answer at epoll with edge triggered event says:
On a somewhat related note: if you register for EPOLLIN and EPOLLOUT events and assuming you never fill up the send buffer, you still get the EPOLLOUT flag set in the event returned by epoll_wait each time EPOLLIN is triggered - see https://lkml.org/lkml/2011/11/17/234 for a more detailed explanation.
And the link in this answer says:
It's doesn't mean there's an EPOLLOUT "event", it just means a message
is triggered (by the socket becoming readable) so you get a status
update. In theory the program doesn't need to be told about EPOLLOUT
here (it should be assuming the socket is writable already), but it
doesn't do any harm.
So far What I understand about epoll edge trigger mode is:
the epoll_wait return when the state of any fd being monitored has changed, e.g from nothing to read -> readable or buffer is full-> buffer can write
the epoll_wait may return one or several event(flags) for each fd in the ready list.
the flags in sturct epoll_event.events field indicate the current state of this fd. Even if we don't fill out the output buffer, the EPOLLOUT flag would be set when epoll_wait return due to readable, because the current state of the fd is just writable.
Please correct me if I am wrong.
Then my question would be: Should I maintain a flag in each connection to indicate whether EAGAIN occurs when write to output buffer, if it is not set, don't call write_handler/handleWrite in "if (events & EPOLLOUT)" branch, so that my upper layer program would not be told about EPOLLOUT here?
What a great question (since I had pretty much the same question)! I'll just summarize what I think I know now wrt to your informative question/description and your helpful links and hopefully smarter folk will correct any mistakes.
Yes, the if/else handling of event flags is definitely bogus. For sure at least two can events can arrive at effectively the same time. E.g., both the read and write sides might have become unblocked since last you called epoll_wait(). And, of course, as soon as you accept() the connection, both reading and writing suddenly become possible, so you get an "event" of EPOLLIN|EPOLLOUT.
I really didn't grok that epoll_wait() is always delivering the entire current state, rather than only the parts of the state that changed -- thanks for clearing that up. To be perhaps clearer, epoll_wait() won't return an fd unless something changed on that socket, but if something did change, it returns all the flags representing the current state. So, I found myself staring at a stream of EPOLLIN|EPOLLOUT events wondering why it was claiming there was an "output" event, even though I hadn't written anything yet. Your answer being correct: it's just telling me the output side is still writeable.
"Should I maintain a flag..." Yes, but I would imagine that in all but the most trivial situations you were probably going to end up maintaining at least one bit of "am I currently blocked" state for your readers/writers anyway. For example, if you ever want to process data in an order different than how it arrives (e.g., prioritize responses over requests to make your server more resistant to overload) you instantly have to give up the simplicity of just having the arrival of I/O drive everything. In the particular case of writing, epoll simply doesn't have enough information to notify you at the "right" time. As soon as you accept a connection, there's an event that says "you can write now"--but you probably have nothing to write if you're a server who couldn't possibly have already gotten a request from the client. epoll just can't know whether you have something to write or not, so you were always going to have to either suffer essentially "extraneous" events, or maintain your own state.
In all but the simplest cases, the socket file descriptor ends up being insufficient information for handling I/O events, so you invariably have to associate some data structure with it, or object if you prefer. So, my C++ looks something like:
nAwake = epoll_wait(epollFd, events, 100, milliseconds);
if(nAwake < 0)
{
perror("epoll_wait failed");
assert(false);
}
for(int iSocket=0; iSocket < nAwake; ++iSocket)
{
auto This = static_cast<Eventable*>(events[iSocket].data.ptr);
auto eventFlags = events[iSocket].events;
fprintf(stderr, "%s event on socket [%d] -> %s\n",
This->ClassName(), This->fd, DumpEvent(eventFlags));
This->Event(eventFlags);
}
Where Eventable is a C++ class (or derivative thereof) that has all the state needed to decide how to handle the flags epoll delivers. (Of course, this is letting the kernel store a pointer to a C++ object, requiring a design that is very clear about pointer ownership/lifetimes.)
And since you're writing low-level code on Linux, you may also care about EPOLLRDHUP. This not-highly-portable flag lets you save one call to read(). If the client (curl seems pretty good at evoking this behavior) closes its write side of the connection (sends a FIN), you normally discover that when epoll tells you EPOLLIN, but read() returns zero bytes. However, Linux maintains an extra bit to indicate your client's write side (your read side) has been closed. So, if you tell epoll you want the EPOLLRDHUP event you can use it to avoid doing a read() whose sole purpose will turn out to be telling you the writer closed their side.
Note that EPOLLIN will still be turned on whenever EPOLLRDHUP is, AFAIK. Even after you do a shutdown(fd, SHUT_RD). Another example of how you will usually be driven to maintain your own idea of the state of the connection. You care more about clients who are kind enough to do half-shutdowns if you are implementing HTTP.
When used as an edge-triggered interface, for performance reasons,
it
is possible to add the file descriptor inside the epoll interface
(EPOLL_CTL_ADD) once by specifying (EPOLLIN|EPOLLOUT).
This allows you
to avoid continuously switching between EPOLLIN and EPOLLOUT calling
epoll_ctl(2) with EPOLL_CTL_MOD.
I am trying to do communication from QT Application to Arduino. The flow is like this: QT Application sends a '1' and Arduino is expected to respond with some data(the data String length is huge, around 300). QT Application is sending '1' at the rate of around 5Hz(every 200ms).
The problem I am facing is, there is an accumulative delay between the Arduino to QT communication. That is, the data I receive from Arduino is not recent data but the frequency of data coming of Arduino is 5Hz only(which is as expected), just the data coming is not recent. This delay keeps on increasing with time. I believe there is some problem with buffer or something.
What I tried:
QSerialPort serialPort; is my device port
serialPort.clear()
serialPort.flush()
Increasing and decreasing Baud Rate from both ends.
Reduce character length from Arduino, here delay reduces significantly but the accumulated delay is observed after a long time.
to clear serial communication buffer, but the issue still persists.
Here is my code snippet:
connect(timer_getdat, SIGNAL(timeout()), this, SLOT(Rec()));
timer_getdat->start(200);
where Rec() is the function where I do communication part.
In Rec():
serialPort.write("1", 2);
// serialPort.waitForBytesWritten(100);
long long bytes_available = serialPort.bytesAvailable();
if (bytes_available >= 1)
{
serialPort.readLine(temp, 500);
serialPort.flush(); // no change
serialPort.clear(); // no change by .clear() also
}
I have been stuck on this issue for a quite long time. The above code snippet is what I think is necessary but if anyone needs more clarification, I may reveal more of the code.
I also encountered with the same issue, and yes QSerialPort.clear() and QSerialPort.flush() doesn't help. Try doing readAll()
So change the part in your Rec() function to something like this:
serialPort.write("1", 2);
long long bytes_available = serialPort.bytesAvailable();
if (bytes_available >= 1)
{
serialPort.readLine(temp, 500);
serialPort.readAll(); // This reads all the data in buffer at once and clears the queue.
}
Even on QT forums, I didn't find the answer to this, was playing with all functions available with QSerialPort class and readAll() seems to work.
About readAll(), Qt documentation says:
Reads all remaining data from the device, and returns it as a byte
array.
My explanation for the resolution is that readAll captures all of the data from the communication buffer and empties it.
This should be the job of clear() function but apparently readAll() seems to work.
I've written a tool which uses the QSerialPort to write to a serial device. It runs for a certain time until I get the following error message:
ASSERT: "bytesTransferred == writeChunkBuffer.size()" in file
qserialport_win.cpp, line 511
My sending function looks like this:
/**
* #brief Send text to device
* #param text
* #return Success/Fail
*/
bool Serial::send(QString text)
{
if (connectionStatus && qsp.isWritable()) {
QByteArray buffer = text.toLatin1();
if (buffer.size() != qsp.write(buffer))
qDebug() << "Send does not work";
qsp.flush();
msleep(15);
return true;
} else {
return false;
}
}
If I understand it correctly I write the text (which is around 20 chars) with flush to the device, wait 15ms and repeat it afterwards. I don't really understand why I get this message?
// EDIT:
After some time I figured out what the problem was. I forgot to mention, why I wait 15 ms. That was part of the documentation, to wait after I send the data. The biggest problem, related to sending data was, that I run QSerialPort in a separate thread. By using that I run in trouble. I moved it to the MainThread and use signal slot design by Qt.
Without knowing the API you're using or how it works, there is no inherent reason why a write() method should transfer all the data supplied, especially when it returns a write count: a clear signal that it may not transfer everything in one go.
The only problem is the assertion itself. You should loop until the data is written, not assume it is all written in a single write. Nor should you sleep between writes, in a vain attempt to outguess the device you're sending to. This is literally just a waste of time.
How should QLocalSocket/QDataStream be read?
I have a program that communicates with another via named pipes using QLocalSocket and QDataStream. The recieveMessage() slot below is connected to the QLocalSocket's readyRead() signal.
void MySceneClient::receiveMessage()
{
qint32 msglength;
(*m_stream) >> msglength;
char* msgdata = new char[msglength];
int read = 0;
while (read < msglength) {
read += m_stream->readRawData(&msgdata[read], msglength - read);
}
...
}
I find that the application sometimes hangs on readRawData(). That is, it succesfully reads the 4 byte header, but then never returns from readRawData().
If I add...
if (m_socket->bytesAvailable() < 5)
return;
...to the start of this function, the application works fine (with the short test message).
I am guessing then (the documentation is very sparse) that there is some sort of deadlock occurring, and that I must use the bytesAvailable() signal to gradually build up the buffer rather than blocking.
Why is this? And what is the correct approach to reading from QLocalSocket?
Your loop blocks the event loop, so you will never get data if all did not arrive pn first read, is what causes your problem I think.
Correct approach is to use signals and slots, readyRead-signal here, and just read the available data in your slot, and if there's not enough, buffer it and return, and read more when you get the next signal.
Be careful with this alternative approach: If you are absolutely sure all the data you expect is going to arrive promptly (perhaps not unreasonable with a local socket where you control both client and server), or if the whole thing is in a thread which doesn nothing else, then it may be ok to use waitForReadyRead method. But the event loop will remain blocked until data arrives, freezing GUI for example (if in GUI thread), and generally troublesome.
I am streaming data as a string over UDP, into a Socket class inside Unreal engine. This is threaded, and runs in the background.
My read function is:
float translate;
void FdataThread::ReceiveUDP()
{
uint32 Size;
TArray<uint8> ReceivedData;
if (ReceiverSocket->HasPendingData(Size))
{
int32 Read = 0;
ReceivedData.SetNumUninitialized(FMath::Min(Size, 65507u));
ReceiverSocket->RecvFrom(ReceivedData.GetData(), ReceivedData.Num(), Read, *targetAddr);
}
FString str = FString(bytesRead, UTF8_TO_TCHAR((const UTF8CHAR *)ReceivedData));
translate = FCString::Atof(*str);
}
I then call the translate variable from another class, on a Tick, or timer.
My test case sends an incrementing number from another application.
If I print this number from inside the above Read function, it looks as expected, counting up incrementally.
When i print it from the other thread, it is missing some of the numbers.
I believe this is because I call it on the Tick, so it misses out some data due to processing time.
My question is:
Is there a way to queue the incoming data, so that when i pull the value, it is the next incremental value and not the current one? What is the best way to go about this?
Thank you, please let me know if I have not been clear.
Is this the complete code? ReceivedData isn't used after it's filled with data from the socket. Instead, an (in this code) undefined variable 'buffer' is being used.
Also, it seems that the while loop could run multiple times, overwriting old data in the ReceivedData buffer. Add some debugging messages to see whether RecvFrom actually reads all bytes from the socket. I believe it reads only one 'packet'.
Finally, especially when you're using UDP sockets over the network, note that the UDP protocol isn't guaranteed to actually deliver its packets. However, I doubt this is causing your problems if you're using it on a single computer or a local network.
Your read loop doesn't make sense. You are reading and throwing away all datagrams but the last in any given sequence that happen to be in the socket receive buffer at the same time. The translate call should be inside the loop, and the loop should be while(true), or while (running), or similar.