I have used Qt quite a lot, but recently needed to debug the threads I have been creating and found many more threads then I was expecting.
So my program is a simple console only (no GUI) Qt application (linux).
Threads that I have created:
It has a main() (which executes the QtCoreApplication) - so that is the main thread.
A thread to process received data from the com port (using FTDI D2XX thirdparty code drivers)
And that is all. When I do ps -T... and find my application there are 7 threads. I have two classes that are QObjects using signals and slots, so maybe they need a thread each for message handling, that takes me to 4 threads... so I am at a loss as to why I might have 7 threads for my application.
Can anyone explain more about what is going on? can post code if needed. Note I only use new QThread once in my code (for the moment).
Qt doesn't create any per-QObject threads. It creates helper threads for some plaform-specific reasons, e.g. QProcess sometimes needs helper threads.
The FTDI D2XX unix driver uses libusb and that implementation is completely backwards and uses additional threads on top of the thread you've provided for it. Frankly said, you shouldn't be using the D2XX driver on Linux or OS X. Just use the kernel driver.
You should simply run the D2XX driver in a trivial non-Qt test application that opens the device and reads from it continuously and see how many threads it spawns. You'll be dismayed...
Related
It's something that seems deceptively simple, but comes with a lot of nasty details and compatibility problems. I have some code that kinda works on Linux and... sorta works on Windows but it's having various problems, for what seems like a common and simple problem. I know async is all the rage these days, but I have good reasons to want a process per connection.
I'm writing a server that hosts simulation processes. So each connection is long-running and CPU intensive. But more importantly, these simulators (Ngspice, Xyce) have global state and sometimes segfault or reach unrecoverable errors. So it is essential that each connection has its own process so they can run/crash in parallel and not mess with each other's state.
Another semi-important detail is that the protocol is based on Capnp RPC, which has a nice cross-platform async API, but not a blocking one. So what I do is have my own blocking accept loop that forks a new process and then starts the Capnp event loop in the new process.
So I started with a simple accept loop, added a ton of ifdefs to support windows, and then added fork to make it multiprocess and then added a SIGCHLD handler to try to avoid zombie processes. But Windows doesn't have fork, and if many clients disconnect simultaneously I still get zombies.
My current code lives here: https://github.com/NyanCAD/SimServer/blob/1ba47205904fe57196498653ece828c572579717/main.cpp
I'm fine with either some more ifdefs and hacks to make Windows work and avoid zombies, or some sort of library that either offers a ready made multiprocess socket server or functionality for writing such a thing. The important part is that it can accept a socket in a new process and pass the raw FD to the Capnp event loop.
I'm using blocking sockets API (waitFor* functions) for sending mail by smtp protocol (it's a DLL modue). All operations are synchronous: connect->waitForConnected(timeout)->login->waitForReadyRead(timeout)->sendMessage->waitForBytesWritten(timeout)->etc.
I'm using blocking API, because QCoreApplication absence is required (DLL used by different apps, incl. non-qt-based). Blocking functions don't require event loop and it works fine.
But how can I make a visual progress for long-term sending mail operations (with big attachment, for example)?
And how can organize callbacks for progress notifications in sendmail DLL?
ps: all blocking waitFor* functions marked as
functions, that may fail randomly on Windows. Consider using the event loop and the readyRead() signal if your software will run on Windows.
Why?
You can definitely have a QApplication instance when using DLLs (it must be QApplication not QCoreApplication since you want a widget-based gui). It integrates into the native message loop of the main thread. Remember that for Qt Gui to run you only need a native event loop - the code doesn't have to be stuck within QCoreApplication::exec. You need an instance of the application, and you need to prime it by invoking exec once, and ensuring that it returns (i.e. by using a zero-timeout timer), but that's all. Past that, the application's main thread's message pump will handle things for you.
Furthermore, to use networking APIs, you don't need to be stuck in the main thread - you can handle them in a separate thread.
Your DLL won't be compatible with console applications that don't run a message pump in the main thread, but then you can cheat: on Windows, and on Windows only, the QCoreApplication and its derived classes can be used in any thread :)
You definitely must either statically link your DLL with Qt, or use a dynamically-linked Qt that was put in a unique namespace. Remember that if the application you link with uses Qt, there are absolutely no guarantees that the Qt they built is binary-compatible with the Qt that you use. Even if it's the same version.
I am writing an QT desktop application that is going to display information received from a serial port. Therefore a class was created and packed into an DLL using standard Windows API features to communicate with the connected device (CreateFile, ReadFile, WriteFile, ...).
At the moment a timer calls the DLL at a predefined rate [< 200ms] and this leads the gui to freeze for short periods. Because of that i am thinking of using a thread to do the serial port stuff, that is also going to display everything.
Is is better to use threads for this problem or should i rewrite the class to do the work event based? The target is, that the gui doesnt freeze.
Edit:
I solved the problem using a QThread derived worker class with an overshadowed run() function, that handles the serial port communication in the background and updates the gui as new informations are available.
It is good practice in many use cases to do all blocking (synchronous) I/O on a separate thread, especially when a graphical user interface is involved. Here's a page I've referenced regarding the challenges with synchronous I/O (as opposed to asynchronous where your code doesn't block but is still single-threaded, or parallel as you're discussing). There are more issues than just what you brought up, for example:
What if there is no data available? Does the GUI block until there is data? For example, if the sender was off then there would be no data
What does the program do if the I/O device is no longer available? For example, if it is a USB-to-serial adapter, what happens if the adapter is unplugged?
I am working on a client/server application (using qt for tcp).
The clients have to send about 15 messages per second to the server.
The problem is this:
the messages from the clients are received in groups. What i mean:
when i get the readyRead() signal and i read the data from the socket, there are multiple messages in the buffer.
This of-course causes lag in the system.
I tried putting the incoming connections in separate threads (thread per connection) but there was no improvement.
I also tried to rise a thread each time i got a readyRead() signal, but again nothing...
BUT when i run a number of clients on the same pc as the server, everything seems ok. When using different pc's over the network, the lag occurs...
(the network used is 100Mbps LAN, the messages are <200KB, and ping between pc's is <5msec, so i don't believe it's a network issue)
On the client side, the code to write the data is pretty simple:
tcpSocket->write(message.toUtf8());
tcpSocket->waitForBytesWritten();
tcpSocket->flush();
I also tried it without flush() or waitForBytesWritten() but the same...
EDIT: Using Qt 4.8.4 and Windows 7 and XP
Anybody has any idea how to overcome this?
Thank you in advance!
The last time I ran into a similar problem was with the stdin/stdout communication of a QProcess of Qt3.3. It behaved completely different on Linux and Windows.
Finally we found out that on Linux it used select() to react asynchronously when data arrived (fast, in most cases only one line readable) while on Windows the existence of new data was polled via a QTimer from the Qt mainloop (large delay, several messages available). A workaround we tried was to reduce the timer period in the source of Qt, but at the end we switched to shared memory based on the native OS mechanisms.
Your description sounds like you are using a similar Qt version on a Windows OS.
I'm designing a networking framework which uses WSAEventSelect for asynchronous operations. I spawn one thread for every 64th socket due to the max 64 events per thread limitation, and everything works as expected except for one thing:
Threads keep getting spawned uncontrollably by Winsock during connect and disconnect, threads that won't go away.
With the current design of the framework, two threads should be running when only a few sockets are active. And as expected, two threads are running in total. However, when I connect with a few sockets (1-5 sockets), an additional 3 threads are spawn which persist until I close the application. Also, when I lose connection on any of the sockets, 2 more threads are spawned (also persisting until closure). That's 7 threads in total, 5 of which I have no idea what they are there for.
If they are required by Winsock for connecting or whatever and then disappeared, that would be fine. But it bothers me that they persist until I close my application.
Is there anyone who could shed some light on this? Possibly a solution to avoid these threads or force them to close when no connections are active?
(Application is written in C++ with Win32 and Winsock 2.2)
Information from Process Explorer:
Expected threads:
MyApp.exe!WinMainCRTStartup
MyApp.exe!Netfw::NetworkThread::ThreadProc
Unexpected threads:
ntdll.dll!RtlpUnWaitCriticalSection+0x2dc
mswsock.dll+0x7426
ntdll.dll!RtlGetCurrentPeb+0x155
ntdll.dll!RtlGetCurrentPeb+0x155
ntdll.dll!RtlGetCurrentPeb+0x155
All of the unexpected threads have call stacks with calls to functions such as ntkrnlpa.exe!IoSetCompletionRoutineEx+0x46e which probably means it is a part of the notification mechanism.
Download the sysinternals tool process explorer. Install the appropriate debugging tools for windows. In process explorer, set Options -> Symbols path to:
SRV*C:\Websymbols*http://msdl.microsoft.com/download/symbols
Where C:\Websymbols is just a place to store the symbol cache (I'd create a new empty directory for it.)
Now, you can inspect your program with process explorer. Double click the process, go to the threads tab, and it will show you where the threads started, how busy they are, and what their current callstack is.
That usually gives you a very good idea of what the threads are. If they're Winsock internal threads, I wouldn't worry about them, even if there are hundreds.
One direction to look in (just a guess): If these are TCP connections, these may be background threads to handle internal TCP-related timers. I don't know why they would use one thread per connection, but something has to do the background work there.