I am writing a program using threads in c++ in linux.
Currently, I am just keeping an array of threads, and every time one second has elapsed, I check to see which have finished, and restart them. Is this bad? I need to keep this program running for a long time. As it is now, I am getting a code 11 after so many loops of restarting threads (the 100th loop in the last trial). I figured that reusing threads and making sure I only have a small number of them running at any one time, that I would not hit the limit. The array I am using only has a size of 8 (of course, I am not starting 8 each time, just those that have stopped).
Any ideas?
My code is below:
if ( loop_times == 0 || pthread_kill(threads[t],0) != 0 )
{
rc = pthread_create(&threads[t], NULL, thread_stall, (void *)NULL);
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
thread_count++;
}
The loop_times variable is just so that I can get into the loop and start the threads the first time. Otherwise, I get a SEGFAULT because the threads haven't been started before.
Also, I have been wanting to see the value of PTHREAD_THREADS_MAX, but I can't print it (even when including limits.h)
If you want to use multiple threads...It better to go for thread pool.
Start a set of threads as detached ones and then through a queue you can send info to every thread so that it can work on that and wait for next input from you.
As it turns out, my problem was that I needed to pthread_join my thread before I restarted it each time. After this, I stopped getting a code 11 and stopped having "still reachable" memory when running it through Valgrind.
Related
The code below is my entire test program. Each time I press ENTER, the RAM that the process is using is increasing with 4k (it will keep increasing, without stopping; I am seeing it with task manager). What is wrong? The same things happens with _beginthread.
I am trying to write a server, and I want to process each connection with a thread. (Note that this means that I can't join the thread, because that will pause the main thread from accepting new connections.)
unsigned __stdcall thread_test(void *)
{
for(int i = 0; i < 10000; i++)
{
i+=1;
i-=1;
} //simulating processing
_endthreadex( 0 );
}
int main()
{
HANDLE hThread;
while(1)
{
getchar();
hThread = (HANDLE)_beginthreadex( NULL, 0, thread_test, 0, 0, NULL );
CloseHandle( hThread );
}
}
Compiled with code blocks and visual studio.
EDIT: I've made some tests, and the memory stops filling up once it reach around 133.000K (when the program starts, the memory is around 800k); but at this stage, the program runs like 4-5 times slower than it did in the beginning (higher the memory - slower the program runs), so it would not be good for my server to run like that.
EDIT 2: I've got Visual Studio 2013 and the problem gone.
EDIT 3: If I test the code above in Visual Studio 2013, it gives no leaks. But if I use beginthreadex with a small server code, it gives me leaks like before, each request giving 4k. Here is the server testcode(it does nothing, only to see that it leaks memory) that I use http://pastebin.com/EDmJXkZU . You can compile it and test it by typing your IP into the adress bar of the browser.
Task Manager is not showing RAM used by your program. For a better view use Task Manager's Resource Monitor and observe the private bytes indication. But all memory monitors show only "virtual memory," which is commonly retained by the runtime library instead of being freed back to Windows. You don't have a real problem.
as i describe in the header I would like to have in a thread an if statement which is checked every 1 minute and if it is true restart the whole programm.. Any suggestions?
void* checkThread(void* arg)
{
if(statement)
//restart procedure
sleep(60);
}
int main()
{
pthread_create(&thread1, NULL, checkThread, main_object);
pthread_create();
pthread_create();
}
If you are going for the nuke-it-from-orbit approach (i.e. you don't want to trust your code to do a controlled shutdown reliably), then having the kill-and-auto-relaunch mechanism inside the same process space as the other code is not a very robust approach. For example, if one of the other threads were to crash, it would take your auto-restart-thread down with it.
A more fail-safe approach would be to have your auto-restart-thread launch all of the other code in a sub-process (via fork(); calling exec() is allowable but not necessary in this case). After 60 seconds, the parent process can kill the child process it created (by calling kill() on the process ID that fork() returned) and then launch a new one.
The advantage of doing it this way is that the separating of memory spaces protects your relauncher-code from any bugs in the rest of the code, and the killing of the child process means that the OS will handle all the cleanup of memory and other resources for you, so there is less of a worry about things like memory or file-handle leaks.
If you want a "nice" way to do it, you set a flag, and then politely wait for the threads to finish, before relaunching everything.
main_thread() {
do {
kill_and_restart_everything = false;
// create your threads.
pthread_create(&thread1, NULL, checkThread, main_object);
pthread_create(&thread2, ...);
pthread_create(&thread3, ...);
// wait for your threads.
pthread_join(thread1, nullptr);
pthread_join(thread2, nullptr);
pthread_join(thread3, nullptr);
} while (kill_and_restart_everything);
}
void* checkThread(void* arg) {
while (! kill_and_restart_everything) {
if(statement)
kill_and_restart_everything = true;
else
sleep(60);
}
}
void* workerThread(void* arg) {
// do stuff. periodically check
if (kill_and_restart_everything) {
// terminate this thread early.
// do it cleanly too, release any resources, etc (RAII is your friend here).
return nullptr;
}
// do other stuff, remember to have that check happen fairly regularly.
}
This way, whenever if(statement) is true, it will set a boolean that can be used to tell each thread to shut down. Then the program waits for each thread to finish, and then starts it all over again.
Downsides: If you're using any global state, that data will not be cleaned up and can cause problems for you. If a thread doesn't check your signal, you could be waiting a looooong time.
If you want to kill everything (nuke it from orbit) and restart, you could simply wrap this program in a shell script (which can then detect whatever condition you want, kill -9 the program, and relaunch it).
Use the exec system call to restart the process from the start of the program.
you can do it in two parts:
Part1: one thread that checks for the statement and sets a boolean to true when you need to restart the program
This is the "checker" thread
Part2: one thread that computes what you want:
this will "relaunch" the program as long as needed
This "relaunch" consists in a big loop
In the loop:
creates a thread that will actually execute your programme (the task you want to be executed)
ends this taks when the boolean is set to true
creates another thread to replace then one that is terminated
The main of your program consists in launching the "checker" and the "relauncher"
Tell me if you have any questions/remarks I can detail or add some code
Every connection requires one thread for each, and for now, we're allowing only certain number of connections per period. So every time a user connects, we increment the counter if we're within certain period from the last time we set the check time.
1.get current_time = time(0)
2.if current_time is OUTSIDE certain period from check_time,
set counter = 0, and check_time = current_time.
3.(otherwise, just leave it the way it is)
4.if counter < LIMIT, counter++ and return TRUE
5.Otherwise return FALSE
But this is independent of actually how many threads we have running in the server, so I'm thinking of a way to allow connections depending on this number.
The problem is that we're actually using a third-party api for this, and we don't know exactly how long the connection will last. First I thought of creating a child thread and run ps on it to pass the result to the parent thread, but it seems like it's going to take more time since I'll have to parse the output result to get the total number of threads, etc. I'm actually not sure if I'm making any sense.. I'm using c++ by the way. Do you guys have any suggestions as to how I could implement the new checking method? It'll be very much appreciated.
There will be a /proc/[pid]/task (since Linux 2.6.0-test6) directory for every thread belonging to process [pid]. Look at man proc for documentation. Assuming you know the pid of your thread pool you could just count those directories.
You could use boost::filesystem to do that from c++, as described here:
How do I count the number of files in a directory using boost::filesystem?
I assumed you are using Linux.
Okay, if you know the TID of the thread in use by the connection then you can wait on that object in a separate thread which can then decrement the counter.
At least I know that you can do it with MSVC...
bool createConnection()
{
if( ConnectionMonitor::connectionsMaxed() )
{
LOG( "Connection Request failed due to over-subscription" );
return false;
}
ConnectionThread& connectionThread = ThreadFactory::createNewConnectionThread();
connectionThread.startConnection();
ThreadMonitorThread& monitor = ThreadFactory::createThreadMonitor(connectionThread);
monitor.monitor();
}
and in ThreadMonitorThread
ThreadMonitorThread( const Thread& thread )
{
this.thread = thread;
}
void monitor()
{
WaitForSingleObject( thread.getTid() );
ConnectionMonitor::decrementThreadCounter();
}
Of course ThreadMonitorThread will require some special privileges to call the decrement and the ThreadFactory will probably need the same to increment it.
You also need to worry about properly coding this up... who owns the objects and what about exceptions and errors etc...
I'm writing a program in C++ using WINAPI to monitor certain directory for new files arriving, and send them in certain order. The files are derived from a live video stream, so there are 2 files in a unit - audio file and video file, and units should be sent in sequence. a. k. a. (1.mp3, 1.avi); (2.mp3, 2.avi)... Architecture is:
1) detect a new file added to the folder, insert file name to the input queue
2) organize files into units, insert units into unit queue
3) send unit by unit
But since I have to use monitoring file directory for files added there, I need to make sure that file is complete, a. k. a. it is ready to send, since the signal appears when the file is created, but it has yet to be filled with info and closed. So I pop file name from a input queue either when queue has more than 1 file (a. k. a. signal came for next file created, that means that previous file is ready to send) or on timeout(10 sec) so for 10 seconds any file should be done.
So in general this program runs and works properly. But, if I assume that the send procedure will take too long time, so the unit queue will grow. And after some number of units buffered in a unit queue the bug appears.
time[END] = 0;
time[START] = clock();
HANDLE hIOMutex2= CreateMutex (NULL, FALSE, NULL);
WaitForSingleObject( hIOMutex2, INFINITE );
hTimer = CreateThread(NULL, 0, Timer, time, 0, &ThreadId1);
if(hTimer == NULL)
printf("Timer Error\n");
ReleaseMutex(hIOMutex2);
ReadDirectoryChangesW(hDir, szBuffer, sizeof(szBuffer) / sizeof(TCHAR), FALSE, FILE_NOTIFY_CHANGE_FILE_NAME, &dwBytes, NULL, NULL);
HANDLE hIOMutex= CreateMutex (NULL, FALSE, NULL);
WaitForSingleObject( hIOMutex, INFINITE );
time[END] = clock();
TerminateThread(hTimer, 0);
ReleaseMutex( hIOMutex);
After around 800 units buffered in a queue, my program gives me "Time Error" message, if I'm right that means that program can't allocate thread. But in this code program terminates timer thread exactly after the file was created in a directory. So I'm kind of confused with this bug. Also interesting is that even with this time error, my program continue to send units as usual, so that doesn't look like a OS mistake or something different, it is wrong thread declaration/termination, at least it seems like that to me.
Also providing Timer code below if it is helpful.
DWORD WINAPI Timer(LPVOID in){
clock_t* time = (clock_t*) in;
while(TRUE){
if(((clock() - time[START])/CLOCKS_PER_SEC >= 10) && (!time[END]) && (!output.empty())){
Send();
if(output.empty()){
ExitThread(0);
}
}
else if((output.empty()) || (time[END])){
break;
}
else{
Sleep(10);
}
}
ExitThread(0);
return 0;
}
Please could anyone here give me some advise how to solve this bug? Thanks in advance.
Using TerminateThread is a bad idea in many ways. In your case, it makes your program fail because it doesn't release the memory for the thread stack. Failure comes when your program has consumed all available virtual memory and CreateThread() cannot reserve enough memory for another thread. Only ever use TerminateThread while exiting a program.
You'll have to do this a smarter way. Either by asking a thread to exit nicely by signaling an event or by just not consuming such an expensive system resource only for handling a file. A simple timer and one thread can do this too.
cout<<"abcd";
sleep(100);
cout<<'\b';
If I want to print the string out and then get back one character ,
why a sleep() is needed here?
But when using printf in C ,it seems that it is not necessary, why?
char* a = "12345";
char* b = "67890";
threadA(){cout<<a;}
threadB(){cout<<b;}
beginthread (threadA);
sleep(100);
beginthread (threadB);
In the second pseudo code above ,is it right to use sleep()?
For calculating tomorrow date:
void get_tomorrow_date( struct timeval *date )
{
sleep( 86400 ); // 60 * 60 * 24
gettimeofday( date, 0 );
}
;)
There are two subtle issues that you need to understand:
Multi-threading
I/O and Buffering
I'll try to give you some idea:
Multi-threading and sleep
Having a sleep in a threaded environment makes sense. The sleep call makes you wait thereby giving the initial thread some scope to have completed its processing i.e. writing out the string abcd to the standard output before the other thread inserts the backspace character. If you didn't wait for the first thread to complete its processing, you'd have written the backspace character first, and then the string abcd and wouldn't notice any difference.
Buffered I/o
I/O typically happens in buffered, non-buffered and semi-buffered states. This can influence how long, if at all, you have to wait for the output to appear on the console.
Your implementation of cout is probably using a buffered model. Try adding a newline or the endl at the end of your cout statements to print a new line and have it flush immediately, or use cout << "abcd" << flush; to flush without printing a new line.
In the second case without the sleep there's a slim chance that the second thread could start working before the first, resulting in the output "6789012345".
However a "sleep" isn't really the way to handle synchronisation between threads. You'd normally use a semaphore or similar in threadA() which threadB() has to wait for before doing its work.
The reason that the call to sleep makes your code work is because you are using it to turn the potentially parallel execution of the two output stream actions into a single, sequential action. The call to sleep() will allow the scheduler to switch away from the main thread of execution and execute thread A.
If you don't put sleep() in, the order of thread execution is not guaranteed and thread B could well start executing/printing before thread A had a chance to do that.
I think you need to understand what sleep does in general, and understand why it might exist.
sleep does what it sounds like. It instructs the OS to put the requesting task (where a task is a thread of execution) to sleep by removing it from the list of currently running processes and putting it on some sort of wait queue.
Note that there are also times when the OS will put you to sleep whether you like it or not. An example would be any form of blocking I/O, like reading a file from disk. The OS does this so that other tasks may get the CPU's attention while you're off waiting for your data.
One would use sleep voluntarily for similar purposes that the OS would. For example, if you have multiple threads and they're waiting on the completion of some computation, you'll probably want to voluntarily relinquish the CPU so that the computation can complete. You may also voluntarily relinquish the CPU so that other threads have a chance to run. For example, if you have a tight loop that's highly CPU-bound, you'll want to sleep now and then to give other threads a chance to run.
What it looks like you're doing is sleeping for the sake of something being flushed to stdout so that some other thread won't write to stdout before you. This, however, isn't guaranteed to work. It might work incidentally, but it's certainly not what you'd want to do by design. You'd either want to explicitly flush your buffer and not sleep at all, or use some form of synchronization.
As for why printf doesn't exhibit those issues... well, it's a crapshoot. Both printf and cout use some form of buffered output, but the implementation of each may be different.
In summary, it's probably best to remember the following:
When you want to synchronize, use synchronization primitives.
When you want to give someone else a chance to run, sleep.
The OS is better at deciding whether an I/O operation is blocking or not.
if you're having problems seeing the "abcd" being printed, it's because you're not giving cout an endline character to flush the buffer.
if you put
cout << "abcd" << endl;
you would be able to see the characters, then it would beep. no sleep necessary.
while( true )
{
msgStack.Lock();
process( msgStack.pop_msg());
msgStack.Unlock();
sleep(0);
}
sleep in the first example is just to print message a little before you will see "backspace" action. In the second example sleep "can" help. But it is weird. You won't be able to synchronize console outs with sleep in some more complex case.
In the code that launches two threads:
beginthread (threadA);
sleep(100);
beginthread (threadB);
the sleep() waits for 100 ms and then continues. The programmer probably did this in order to give threadA a chance to start up before launching threadB. If you must wait for threadA to be initialized and running before starting threadB, then you need a mechanism that waits for threadA to start, but this is the wrong way to do it.
100 is a magic cookie, chosen arbitrarily, probably accompanying a thought like "it should never take threadA more than 100 ms to start up." Assumptions like this are faulty because you have no way of knowing how long it will take for threadA to start. If the machine is busy or if the implementation of threadA changes it could easily take longer than 100 ms for the thread to launch, run its startup code, and get to it's main loop (if it is that kind of thread).
Instead of sleeping for some arbitrary amount of time, threadA needs to tell the main thread when it is up & running. One common way of doing this is by signaling an event.
Sample code that illustrates how to do this:
#include "stdafx.h"
#include <windows.h>
#include <process.h>
struct ThreadParam
{
HANDLE running_;
HANDLE die_;
};
DWORD WINAPI threadA(void* pv)
{
ThreadParam* param = reinterpret_cast<ThreadParam*>(pv);
if( !param )
return 1;
// do some initialization
// : :
SetEvent(param->running_);
WaitForSingleObject(param->die_, INFINITE);
return 0;
}
DWORD WINAPI threadB(void* pv)
{
ThreadParam* param = reinterpret_cast<ThreadParam*>(pv);
if( !param )
return 1;
// do some initialization
// : :
SetEvent(param->running_);
WaitForSingleObject(param->die_, INFINITE);
return 0;
}
int main(int argc, char** argv)
{
ThreadParam
paramA = {CreateEvent(0, 1, 0, 0), CreateEvent(0, 1, 0, 0) },
paramB = {CreateEvent(0, 1, 0, 0), CreateEvent(0, 1, 0, 0) };
DWORD idA = 0, idB = 0;
// start thread A, wait for it to initialize
HANDLE a = CreateThread(0, 0, threadA, (void*)¶mA, 0, &idA);
WaitForSingleObject(paramA.running_, INFINITE);
// start thread B, wait for it to initi
HANDLE b = CreateThread(0, 0, threadB, (void*)¶mB, 0, &idB);
WaitForSingleObject(paramB.running_, INFINITE);
// tell both threads to die
SetEvent(paramA.die_);
SetEvent(paramB.die_);
CloseHandle(a);
CloseHandle(b);
return 0;
}
It's not needed - what output do you get if you omit it?
The only thing sleep does is pauses execution on the calling thread for the specified number of milliseconds. It in no way will affect the outcome of any printing you might do.
Sleep can be used to avoid a certain thread/process (yeah, i know they are different things) hogging the processor.
On the other hand, printf is thread safe. Cout is not. That may explain differences in their behaviour.