I'm using Live555 to realize a C++ RTPS client for IP cameras.
I'm using most of the testRTSPClient code.
I used Poco library and Poco::Thread class too.
In other words any client for each camera runs in a separate thread tha owns his instance of Live555 objects (as live555-devel suggests, any thread uses an instance with his UsageEnvironment and TaskScheduler.). This to avoid shared variables and synchronization stuff. It seems to works well and fast.
My runnable (following the Poco library requirements) object IPCamera has the run method as simple as:
void IPCamera::run()
{
openURL(_myEnv, "", _myRtspCommand.c_str(), *this); //taken from the testRTSPClient example
_myEnv->TaskScheduler().doEventLoop(&_watchEventLoopVariable);
//it runs until _watchEventLoopVariable change to a value != 0
//exit from the run;
}
When the run is finished I call join() to close the thread (by the way I figured that if I don't call myThread->join(), the memory is not freed totally).
Upon shutdown, following the requirements in Live555-devel I put in my code:
void IPCamera::shutdown()
{
...
_myEnv->reclaim();
delete _myScheduler;
}
Using Valgrind to detect memory leaks I saw a strange behaviour:
1) case: Run the program - Close the program with all the IPCameras that run in the proper manner.
a) At the end of the program all the destructors are invoked.
b) exit from doEventLoop().
c) join the thread (actually is terminated because it exits from run method.
d) destroy the _myEnv and _myScheduler as showed.
e) destroy all the others objects, including IPCamera and Thread associated.
-> no memory leaks are found by Valgrind. Ok
Now comes the problem.
2) case: I'm implementing a use case where a Poco::Timer checks every X seconds if the camera is alive using ICMP ping. It raises an event (using Poco events) in case it doesn't answer because the network is down and I do the follow:
IPCamera down :
a) put the _watchEventLoopVariable = 1 to exit from the run method;
b) shutdown the client associated to the IPCamera as showed
c) join the thread
I don't destroy the thread because I would like to reuse it when the network is up again and the camera works again.And in that case:
a)I set the _eventWatchVariable = 0.
b) Let start again the thread with: myThread->run()
Valgrind tells me that memory leaks are found: 60 bytes direct, 20.000 indirect bytes are lost in the thread, in the H264BufferdPackedFactory::createNewPacket(...), a class of Live555.
SOLVED
I found out that the problem was the tunneling over TCP. In LIVE55 you can select the kind of protocol.
If I select:
#define REQUEST_STREAMING_OVER_TCP false
I don't have any leak. I used many times Valgrind to be sure (it discovered the problem).
If I use TCP then the above problem is manifested.
Related
I have a very weird issue with a zombie/garbage thread crashing the process although this thread should have been joined a while ago.
Application written in C++, built for x86_64, runs on MacOS using Rosetta. Googletest runs app multiple times: initializes the engine, performs tests and uninitializes it for every test - no new processes are started - all happens within googletest process. In about one goggletest start of about ten, one of the threads that my application has not created during current run crashes with invalid resource access which triggers SIGABRT and crashes the complete process.
Threads in the pool are created like this during thread pool construction, done once during application initialization:
// threadsNum is a constant, e.g. 8
for (size_t i = 0; i < threadsNum; ++i){
// this->workers has type std::vector< std::thread > workers;
workers.emplace_back(
[this, i]
{
workerThreadFunction(i);
}
);
}
workerThreadFunction() sets thread name and enters an infinete loop for doing it's job while checking if it should stop in every iteration:
for (;;) {
// Mutex lock to access this->stop
std::unique_lock<std::mutex> lock(this->queue_mutex);
if (this->stop) {
break;
}
...
}
During thread pool destruction all threads are joined:
for(std::thread &worker: workers) {
worker.join();
}
To sum up:
Application starts 8 threads for a specfic task using a thread pool
Threads have their names for debugging purposes - e.g. "MyThread 1", "MyThread 2", etc.
I have verified that all threads are joined upon thread pool destruction (threadX.join() returns)
Application shutdown is clean after test iteration execution before the next test with the crash starts
In problematic runs with the crash there are more threads present than application has created - one of the threads is present two times (thread name duplicated). Crash dump also shows there are two threads with the same name and one of them has crashed.
That duplicated thread has corrupt stack and crashes due to invalid resource access (locking this->queue_mutex to be precise)
Additional main thread sleep for e.g. 100ms after engine uninitialization does not help - does not look like a timing issue
To me it looks like that thread has survived join() somehow and reappeared in the process. But I cannot imagine how could it be possible.
The question is, am I missing something here? Are there any tools to debug this issue besides what I have already done?
How should QLocalSocket/QDataStream be read?
I have a program that communicates with another via named pipes using QLocalSocket and QDataStream. The recieveMessage() slot below is connected to the QLocalSocket's readyRead() signal.
void MySceneClient::receiveMessage()
{
qint32 msglength;
(*m_stream) >> msglength;
char* msgdata = new char[msglength];
int read = 0;
while (read < msglength) {
read += m_stream->readRawData(&msgdata[read], msglength - read);
}
...
}
I find that the application sometimes hangs on readRawData(). That is, it succesfully reads the 4 byte header, but then never returns from readRawData().
If I add...
if (m_socket->bytesAvailable() < 5)
return;
...to the start of this function, the application works fine (with the short test message).
I am guessing then (the documentation is very sparse) that there is some sort of deadlock occurring, and that I must use the bytesAvailable() signal to gradually build up the buffer rather than blocking.
Why is this? And what is the correct approach to reading from QLocalSocket?
Your loop blocks the event loop, so you will never get data if all did not arrive pn first read, is what causes your problem I think.
Correct approach is to use signals and slots, readyRead-signal here, and just read the available data in your slot, and if there's not enough, buffer it and return, and read more when you get the next signal.
Be careful with this alternative approach: If you are absolutely sure all the data you expect is going to arrive promptly (perhaps not unreasonable with a local socket where you control both client and server), or if the whole thing is in a thread which doesn nothing else, then it may be ok to use waitForReadyRead method. But the event loop will remain blocked until data arrives, freezing GUI for example (if in GUI thread), and generally troublesome.
I have a problem with Microsoft's WaveOut API:
edit1: Added Link to sample project:
edit2: removed link, its not representative of the issue
After playing some audio, when I want to terminate a given playback stream, I call the function:
waveOutClose(hWaveOut_);
However, even after waveOutClose() is called, sometimes the library will still access memory previously passed to it by waveOutWrite(), causing an invalid memory access.
I then tried to ensure all the buffers are marked as done before freeing the buffer:
PcmPlayback::~PcmPlayback()
{
if(hWaveOut_ == nullptr)
return;
waveOutReset(hWaveOut_); // infinite-loops, never returns
for(auto it = buffers_.begin(); it != buffers_.end(); ++it)
waveOutUnprepareHeader(hWaveOut_, &it->wavehdr_, sizeof(WAVEHDR));
while( buffers_.empty() == false ) // infinite loops
removeCompletedBuffers();
waveOutClose(hWaveOut_);
//Unhandled exception at 0x75629E80 (msvcrt.dll) in app.exe:
// 0xC0000005: Access violation reading location 0xFEEEFEEE.
}
void PcmPlayback::removeCompletedBuffers()
{
for(auto it = buffers_.begin(); it != buffers_.end();)
{
if( it->wavehdr_.dwFlags & WHDR_DONE )
{
waveOutUnprepareHeader(hWaveOut_, &it->wavehdr_, sizeof(WAVEHDR));
it = buffers_.erase(it);
}
else
++it;
}
}
However, this situation never happens - the buffer never becomes empty. There will be 4-5 blocks remaining with wavehdr_.dwFlags == 18 (I believe this means the blocks are still marked as in playback)
How can I resolve this issue?
# Martin Schlott ("Can you provide the loop where you write the buffer to waveOutWrite?")
Its not quite a loop, instead I have a function that is called whenever I receive an audio packet over the network:
void PcmPlayback::addData(const std::vector<short> &rhs)
{
removeCompletedBuffers();
if(rhs.empty())
return;
// add new data
buffers_.push_back(Buffer());
Buffer & buffer = buffers_.back();
buffer.data_ = rhs;
ZeroMemory(&buffers_.back().wavehdr_, sizeof(WAVEHDR));
buffer.wavehdr_.dwBufferLength = buffer.data_.size() * sizeof(short);
buffer.wavehdr_.lpData = (char *)(buffer.data_.data());
waveOutPrepareHeader(hWaveOut_, &buffer.wavehdr_, sizeof(WAVEHDR)); // prepare block for playback
waveOutWrite(hWaveOut_, &buffer.wavehdr_, sizeof(WAVEHDR));
}
The described behavior can happen if you do not call
waveOutUnprepareHeader
to every buffer you used before you use
waveOutClose
The flagfield _dwFlags seems to indicate that the buffers are still enqueued (WHDR_INQUEUE | WHDR_PREPARED) try:
waveOutReset
before unprepare buffers.
After analyses your code, I found two problems/bugs which are not related to waveOut (funny, you use C++11 but the oldest media interface). You use a vector as buffer. During some calling operations, the vector is copied! One bug I found is:
typedef std::function<void(std::vector<short>)> CALLBACK_FN;
instead of:
typedef std::function<void(std::vector<short>&)> CALLBACK_FN;
which forces a copy of the vector.
Try to avoid using vectors if you expect to use it mostly as rawbuffer. Better use std::unique_pointer as buffer pointer.
Your callback in the recorder is not monitored by a mutex, nor does it check if a destructor was already called. The destructing happens during the callback (mostly) which leads to an exception.
For your test program, go back and use raw pointer and static callbacks before blaming waveOut. Your code is not bad, but the first bug already shows, that a small bug will lead to unpredictical errors. As you also organize your buffers in a std::array, I would search for bugs there. I guess, you make a unintentional copy of your whole buffer array, unpreparing the wrong buffers.
I did not have the time to dig deeper, but I guess those are the problems.
I managed to find my problem in the end, it was caused by multiple bugs and a deadlock. I will document what happened here so people can learn from this in the future
I was clued in to what was happening when I fixed the bugs in the sample:
call waveInStop() before waveInClose() in ~Recorder.cpp
wait for all buffers to have the WHDR_DONE flag before calling waveOutClose() in ~PcmPlayback.
After doing this, the sample worked fine and did not display the behavior of the WHDR_DONE flag never being marked.
In my main program, that behavior was caused by a deadlock that occurs in the following situation:
I have a vector of objects representing each peer I am streaming audio with
Each Object owns a Playback class
This vector is protected by a mutex
Recorder callback:
mutex.lock()
send audio packet to each peer.
Remove Peer:
mutex.lock()
~PcmPlayback
wait for WHDR_DONE flags to be marked
A deadlock occurs when I remove a peer, locking the mutex and the recorder callback tries to acquire a lock too.
Note that this will happen often because the playback buffer is usually (~4 * 20ms) while the recorder has a cadence of 20ms.
In ~PcmPlayback, the buffers will never be marked as WHDR_DONE and any calls to the WaveOut API will never return because the WaveOut API is waiting for the Recorder callback to complete, which is in turn waiting on mutex.lock(), causing a deadlock.
I have a little problem, I wrote a program, server role, doing an infinite loop waiting for client requests.
But I would like this program to also return his pid.
Thus, I think I should use multithreading.
Here's my main :
int main(int argc, char **argv) {
int pid = (int) getpid();
int port = 5555
ServerSoap *servsoap;
servsoap = new ServerSoap(port, false);
servsoap->StartServer(); //Here starts the infinite loop
return pid; //so it never executes this
}
If it was bash scripting I would add & to run it in background.
Shall I use pthread ? And how to do it please ?
Thanks.
eo
When a program returns (exits), all running threads terminate, so you can't have a background thread continue to run.
In addition, the int return value of main is (usually) truncated to a 7-bit value, so you don't have enough space to return a full pid.
It'd be better just to print the pid to stdout using printf.
If you put the infinite loop in a separate thread, and then return from main it will kill the whole process including your new thread. One solution, keeping to threads, is to make a detached thread. A better solution is probably to create a new process:
int main()
{
int pid = fork();
if (pid == -1)
perror("fork");
else if (pid == 0)
{
ServerSoap serversoap(5555, false);
serversoap.StartServer();
}
return pid;
}
Edit: Also note the limit to the return value from main as noted in the answer from ecatmur.
I have a feeling that you're trying to implement daemon.
To add to #ecatmur answer, if no error has happened program should always return 0 on termination.
PID is usually saved in some file, often times in /var/run/ directory. Some programs use /tmp/ directory.
Your main is attempting to do what your server should do. You're confusing a couple patterns here.
Pattern #1: Daemon
Think of the main as the program that, when on, accepts client requests and performs operations with them. The main has to wait for requests if this is the structure of the program. When a request is received, only then do you perform the requested operation. The main serves only to turn on or off this service. Normally this type of behavior is handled by default with threads. The listener activates a thread calling specific methods with information regarding the request, for instance. Unless you require threads for the work you need done, you shouldn't require threads for this.
Pattern #2: Tool
Alternatively, you could simply call this program as a tool. You'd still need a web service, but this program could be separate from that. Apart from what your tool should do, you shouldn't require threads for this.
In either case, I don't think what you're looking for is to implement threading. You're simply activating a server which does nothing. You should probably look into adding request handlers instead.
EDIT: I have now edited my code a bit to have a rough idea of "all" the code. Maybe this
might be helpful to identify the problem ;)
I have integrated the following simple code fragement which either cancels the timer if data
is read from the TCP socket or otherwise it cancels the data read from the socket
// file tcp.cpp
void CheckTCPSocket()
{
TRequestStatus iStatus;
TSockXfrLength len;
int timeout = 1000;
RTimer timer;
TRequestStatus timerstatus;
TPtr8 buff;
iSocket.RecvOneOrMore( buff, 0, iStatus, len );
timer.CreateLocal();
timer.After(timerstatus, timeout);
// Wait for two requests – if timer completes first, we have a
// timeout.
User::WaitForRequest(iStatus, timerstatus);
if(timerstatus.Int() != KRequestPending)
{
iSocket.CancelRead();
}
else
{
timer.Cancel();
}
timer.Close();
}
// file main.cpp
void TestActiveObject::RunL()
{
TUint Data;
MQueue.ReceiveBlocking(Data);
CheckTCPSocket();
SetActive();
}
This part is executed within active Object and since integrating the code piece above I always get the kernel panic:
E32User-CBase 46: This panic is raised by an active scheduler, a CActiveScheduler. It is caused by a stray signal.
I never had any problem with my code until now this piece of code is executed; code executes fine as data is read from the socket and
then the timer is canceled and closed. I do not understand how the timer object has here any influence on the AO.
Would be great if someone could point me to the right direction.
Thanks
This could be a problem with another active object completing (not one of these two), or SetActive() not being called. See Forum Nokia. Hard to say without seeing all your code!
BTW User::WaitForRequest() is nearly always a bad idea. See why here.
Never mix active objects and User::WaitForRequest().
(Well, almost never. When you know exactly what you are doing it can be ok, but the code you posted suggests you still have some learning to do.)
You get the stray signal panic when the thread request semaphore is signalled with RThread::RequestComplete() by the asynchronous service provider and the active scheduler that was waiting on the semaphore with User::WaitForAnyRequest() tries to look for an active object that was completed so that its RunL() could be called, but cannot find any in its list of active objects.
In this case you have two ongoing requests, neither of which is controlled by the active scheduler (for example, not using CActive::iStatus as the TRequestStatus; issuing SetActive() on an object where CActive::iStatus is not involved in an async request is another error in your code but not the reason for stray signal). You wait for either one of them to complete with WaitForRequest() but don't wait for the other to complete at all. The other request's completion signal will go to the active scheduler's WaitForAnyRequest(), resulting in stray signal. If you cancel a request, you will still need to wait on the thread request semaphore.
The best solution is to make the timeout timer an active object as well. Have a look at the CTimer class.
Another solution is just to add another WaitForRequest on the request not yet completed.
You are calling TestActiveObject::SetActive() but there is no call to any method that sets TestActiveObject::iStatus to KRequestPending. This will create the stray signal panic.
The only iStatus variable in your code is local to the CheckTCPSocket() method.