I'm writing a program in C++ using WINAPI to monitor certain directory for new files arriving, and send them in certain order. The files are derived from a live video stream, so there are 2 files in a unit - audio file and video file, and units should be sent in sequence. a. k. a. (1.mp3, 1.avi); (2.mp3, 2.avi)... Architecture is:
1) detect a new file added to the folder, insert file name to the input queue
2) organize files into units, insert units into unit queue
3) send unit by unit
But since I have to use monitoring file directory for files added there, I need to make sure that file is complete, a. k. a. it is ready to send, since the signal appears when the file is created, but it has yet to be filled with info and closed. So I pop file name from a input queue either when queue has more than 1 file (a. k. a. signal came for next file created, that means that previous file is ready to send) or on timeout(10 sec) so for 10 seconds any file should be done.
So in general this program runs and works properly. But, if I assume that the send procedure will take too long time, so the unit queue will grow. And after some number of units buffered in a unit queue the bug appears.
time[END] = 0;
time[START] = clock();
HANDLE hIOMutex2= CreateMutex (NULL, FALSE, NULL);
WaitForSingleObject( hIOMutex2, INFINITE );
hTimer = CreateThread(NULL, 0, Timer, time, 0, &ThreadId1);
if(hTimer == NULL)
printf("Timer Error\n");
ReleaseMutex(hIOMutex2);
ReadDirectoryChangesW(hDir, szBuffer, sizeof(szBuffer) / sizeof(TCHAR), FALSE, FILE_NOTIFY_CHANGE_FILE_NAME, &dwBytes, NULL, NULL);
HANDLE hIOMutex= CreateMutex (NULL, FALSE, NULL);
WaitForSingleObject( hIOMutex, INFINITE );
time[END] = clock();
TerminateThread(hTimer, 0);
ReleaseMutex( hIOMutex);
After around 800 units buffered in a queue, my program gives me "Time Error" message, if I'm right that means that program can't allocate thread. But in this code program terminates timer thread exactly after the file was created in a directory. So I'm kind of confused with this bug. Also interesting is that even with this time error, my program continue to send units as usual, so that doesn't look like a OS mistake or something different, it is wrong thread declaration/termination, at least it seems like that to me.
Also providing Timer code below if it is helpful.
DWORD WINAPI Timer(LPVOID in){
clock_t* time = (clock_t*) in;
while(TRUE){
if(((clock() - time[START])/CLOCKS_PER_SEC >= 10) && (!time[END]) && (!output.empty())){
Send();
if(output.empty()){
ExitThread(0);
}
}
else if((output.empty()) || (time[END])){
break;
}
else{
Sleep(10);
}
}
ExitThread(0);
return 0;
}
Please could anyone here give me some advise how to solve this bug? Thanks in advance.
Using TerminateThread is a bad idea in many ways. In your case, it makes your program fail because it doesn't release the memory for the thread stack. Failure comes when your program has consumed all available virtual memory and CreateThread() cannot reserve enough memory for another thread. Only ever use TerminateThread while exiting a program.
You'll have to do this a smarter way. Either by asking a thread to exit nicely by signaling an event or by just not consuming such an expensive system resource only for handling a file. A simple timer and one thread can do this too.
Related
as i describe in the header I would like to have in a thread an if statement which is checked every 1 minute and if it is true restart the whole programm.. Any suggestions?
void* checkThread(void* arg)
{
if(statement)
//restart procedure
sleep(60);
}
int main()
{
pthread_create(&thread1, NULL, checkThread, main_object);
pthread_create();
pthread_create();
}
If you are going for the nuke-it-from-orbit approach (i.e. you don't want to trust your code to do a controlled shutdown reliably), then having the kill-and-auto-relaunch mechanism inside the same process space as the other code is not a very robust approach. For example, if one of the other threads were to crash, it would take your auto-restart-thread down with it.
A more fail-safe approach would be to have your auto-restart-thread launch all of the other code in a sub-process (via fork(); calling exec() is allowable but not necessary in this case). After 60 seconds, the parent process can kill the child process it created (by calling kill() on the process ID that fork() returned) and then launch a new one.
The advantage of doing it this way is that the separating of memory spaces protects your relauncher-code from any bugs in the rest of the code, and the killing of the child process means that the OS will handle all the cleanup of memory and other resources for you, so there is less of a worry about things like memory or file-handle leaks.
If you want a "nice" way to do it, you set a flag, and then politely wait for the threads to finish, before relaunching everything.
main_thread() {
do {
kill_and_restart_everything = false;
// create your threads.
pthread_create(&thread1, NULL, checkThread, main_object);
pthread_create(&thread2, ...);
pthread_create(&thread3, ...);
// wait for your threads.
pthread_join(thread1, nullptr);
pthread_join(thread2, nullptr);
pthread_join(thread3, nullptr);
} while (kill_and_restart_everything);
}
void* checkThread(void* arg) {
while (! kill_and_restart_everything) {
if(statement)
kill_and_restart_everything = true;
else
sleep(60);
}
}
void* workerThread(void* arg) {
// do stuff. periodically check
if (kill_and_restart_everything) {
// terminate this thread early.
// do it cleanly too, release any resources, etc (RAII is your friend here).
return nullptr;
}
// do other stuff, remember to have that check happen fairly regularly.
}
This way, whenever if(statement) is true, it will set a boolean that can be used to tell each thread to shut down. Then the program waits for each thread to finish, and then starts it all over again.
Downsides: If you're using any global state, that data will not be cleaned up and can cause problems for you. If a thread doesn't check your signal, you could be waiting a looooong time.
If you want to kill everything (nuke it from orbit) and restart, you could simply wrap this program in a shell script (which can then detect whatever condition you want, kill -9 the program, and relaunch it).
Use the exec system call to restart the process from the start of the program.
you can do it in two parts:
Part1: one thread that checks for the statement and sets a boolean to true when you need to restart the program
This is the "checker" thread
Part2: one thread that computes what you want:
this will "relaunch" the program as long as needed
This "relaunch" consists in a big loop
In the loop:
creates a thread that will actually execute your programme (the task you want to be executed)
ends this taks when the boolean is set to true
creates another thread to replace then one that is terminated
The main of your program consists in launching the "checker" and the "relauncher"
Tell me if you have any questions/remarks I can detail or add some code
I am writing a program using threads in c++ in linux.
Currently, I am just keeping an array of threads, and every time one second has elapsed, I check to see which have finished, and restart them. Is this bad? I need to keep this program running for a long time. As it is now, I am getting a code 11 after so many loops of restarting threads (the 100th loop in the last trial). I figured that reusing threads and making sure I only have a small number of them running at any one time, that I would not hit the limit. The array I am using only has a size of 8 (of course, I am not starting 8 each time, just those that have stopped).
Any ideas?
My code is below:
if ( loop_times == 0 || pthread_kill(threads[t],0) != 0 )
{
rc = pthread_create(&threads[t], NULL, thread_stall, (void *)NULL);
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
exit(-1);
}
thread_count++;
}
The loop_times variable is just so that I can get into the loop and start the threads the first time. Otherwise, I get a SEGFAULT because the threads haven't been started before.
Also, I have been wanting to see the value of PTHREAD_THREADS_MAX, but I can't print it (even when including limits.h)
If you want to use multiple threads...It better to go for thread pool.
Start a set of threads as detached ones and then through a queue you can send info to every thread so that it can work on that and wait for next input from you.
As it turns out, my problem was that I needed to pthread_join my thread before I restarted it each time. After this, I stopped getting a code 11 and stopped having "still reachable" memory when running it through Valgrind.
is there a way for a forked child to examine another forked child so that, if the other forked child takes more time than usual to perform its chores, the first child may perform predefined steps?
if so, sample code will be greatly appreciated.
Yes. Simply fork the process to be watched, from the process to watch it.
if (fork() == 0) {
// we are the watcher
pid_t watchee_pid = fork();
if (watchee_pid != 0) {
// wait and/or handle timeout
int status;
waitpid(watchee_pid, &status, WNOHANG);
} else {
// we're being watched. do stuff
}
} else {
// original process
}
To emphasise: There are 3 processes. The original, the watcher process (that handles timeout etc.) and the actual watched process.
To do this, you'll need to use some form of IPC, and named shared memory segments makes perfect sense here. Your first child could read a value in a named segment which the other child will set once it has completed it's work. Your first child could set a time out and once that time out expires, check for the value - if the value is not set, then do what you need to do.
The code can vary greatly depending on C or C++, you need to select which. If C++, you can use boost::interprocess for this - which has lots of examples of shared memory usage. If C, then you'll have to put this together using native calls for your OS - again this should be fairly straightforward - start at shmget()
This is some orientative code that could help you to solve the problem in a Linux environment.
pid_t pid = fork();
if (pid == -1) {
printf("fork: %s", strerror(errno));
exit(1);
} else if (pid > 0) {
/* parent process */
int i = 0;
int secs = 60; /* 60 secs for the process to finish */
while(1) {
/* check if process with pid exists */
if (exist(pid) && i > secs) {
/* do something accordingly */
}
sleep(1);
i++;
}
} else {
/* child process */
/* child logic here */
exit(0);
}
... those 60 seconds are not very strict. you could better use a timer if you want more strict timing measurement. But if your system doesn't need critical real time processing should be just fine like this.
exist(pid) refers to a function that you should have code that looks into proc/pid where pid is the process id of the child process.
Optionally, you can implement the function exist(pid) using other libraries designed to extract information from the /proc directory like procps
The only processes you can wait on are your own direct child processes - not siblings, not your parent, not grandchildren, etc. Depending on your program's needs, Matt's solution may work for you. If not, here are some other alternatives:
Forget about waiting and use another form of IPC. For robustness, it needs to be something where unexpected termination of the process you're waiting on results in your receiving an event. The best one I can think of is opening a pipe which both processes share, and giving the writing end of the pipe to the process you want to wait for (make sure no other processes keep the writing end open!). When the process holding the writing end terminates, it will be closed, and the reading end will then indicate EOF (read will block on it until the writing end is closed, then return a zero-length read).
Forget about IPC and use threads. One advantage of threads is that the atomicity of a "process" is preserved. It's impossible for individual threads to be killed or otherwise terminate outside of the control of your program, so you don't have to worry about race conditions with process ids and shared resource allocation in the system-global namespace (IPC objects, filenames, sockets, etc.). All synchronization primitives exist purely within your process's address space.
How can I create a global counter-value that can be shared between multiple processes in c++? What I need is a way to "invalidate" multiple processes at once, signaling them to perform some operation (like reading from file). All processes would continuously poll (every 10ms) for current counter-value and compare it with internally stored last value. Mismatching values would indicate that some work is needed.
Edit: btw my processes are executing as different .exe:s, not created from some parent process. Operating system is windows.
What about a named semaphore? Posix supports it, not sure about windows.
Consider the way you want to distribute the information and potential overlaps - if it takes longer for any of the readers to finish reading than it takes for a refresh then you are going to get in trouble with the suggested approach.
The way I read your question, there are multiple readers, the writer doesn't know (or care in most part) how many readers there are at one time, but wants to notify the readers that something new is available to read.
Without knowing how many potential readers there are you can't use a simple mutex or semaphore to know when the readers are done, without knowing when everybody is done you don't have good info on when to reset an event to notify for the next read event.
MS Windows specific:
Shared Segments
One option is to place variables within a shared data segment. That means that the same variables can be read (and written to) by all exe's that have named the same segment or if you put it into a DLL - loaded the shared DLL.
See http://www.codeproject.com/KB/DLL/data_seg_share.aspx for more info.
// Note: Be very wary of using anything other than primitive types here!
#pragma data_seg(".mysegmentname")
HWND hWnd = NULL;
LONG nVersion = -1;
#pragma data_seg()
#pragma comment(linker, "/section:.mysegmentname,rws")
IPC - COM
Make your main app a com service where the workers can register with for events, push out the change to each event sink.
IPC - dual events
Assuming any 1 read cycle is much less than time between write events.
create 2 manual reset events, at any time at most 1 of those events will be signaled, alternate between events. signaling will immediatly release all the readers and once complete they will wait on the alternate event.
you can do this the easy way or the way
the easy way is to store shared values in registry or a file so that all processes agree to check it frequently.
the hard way is to use IPC(inter process communication, the most common method that i use is NamedPipes. its not too hard because you can find plenty of resources about IPC on the net.
If you are on *nix you could make the processes read from a named pipe (or sockets), and then write the specific msg there to tell the other processes that they should shutdown.
IPC performance: Named Pipe vs Socket
Windows NAmed Pipes alternative in Linux
Use a named event object with manual reset. The following solution doesn't use the CPU so much than busy waiting
Sending process:
Set event
Sleep 10 ms
Reset Event
Receiving processes:
All waiting processes pass when event is set
They read the file
Let them sleep for 20 ms, so say can't see the same event twice.
Wait again
Sleep( 10 ) might actually take longer than Sleep( 20 ) but this only results in another cycle (reading the unchanged file again).
As the name of the executable is known, I have another solution which I implemented (in C#) in a project just a few days ago:
Every reader process creates a named event "Global\someuniquestring_%u" with %u being it's process id. If the event is signaled, read the file and do the work.
The sender process has a list of event handles and sets them active if the file has changed and thus notifys all reader processes. From time to time, e.g. when the file has changed, it has to update the list of event handles:
Get all processes with name 'reader.exe' (e.g.)
For every process get it's id
Open a handle for the existing event "Global\someuniquestring_%u" if it's a new process.
Close all handles for no longer running processes.
Found one solution for monitoring folder changes (with "event_trigger"-event) and reading additional event information from file:
HANDLE event_trigger;
__int64 event_last_time;
vector<string> event_info_args;
string event_info_file = "event_info.ini";
// On init
event_trigger = FindFirstChangeNotification(".", false, FILE_NOTIFY_CHANGE_LAST_WRITE);
event_last_time = stat_mtime_force("event_info.ini");
// On tick
if (WaitForSingleObject(event_trigger, 0)==0)
{
ResetEventTrigger(event_trigger);
if (stat_mtime_changed("event_info.ini", event_last_time))
{
FILE* file = fopen_force("event_info.ini");
char buf[4096];
assert(fgets(buf, sizeof(buf), file));
split(buf, event_info_args, "\t\r\n");
fclose(file);
// Process event_info_args here...
HWND wnd = ...;
InvalidateRect(wnd,0,false);
}
}
// On event invokation
FILE* file = fopen("event_info.ini", "wt");
assert(file);
fprintf(file,"%s\t%s\t%d\n",
"par1", "par2", 1234);
fclose(file);
stat_mtime_changed("event_info.ini", event_last_time);
// Helper functions:
void ResetEventTrigger()
{
do
{
FindNextChangeNotification(evt);
}
while(WaitForSingleObject(evt, 0)==0);
}
FILE* fopen_force(const char* file);
{
FILE* f = fopen(file, "rt");
while(!f)
{
Sleep(10+(rand()%100));
f = fopen(f, "rt");
}
assert(f);
return f;
}
__int64 stat_mtime_force(const char* file)
{
struct stat stats;
int res = stat(file, &stats);
if(res!=0)
{
FILE* f = fopen(file, "wt");
fclose(f);
res = stat(file, &stats);
}
assert(res==0);
return stats.st_mtime;
}
bool stat_mtime_changed(const char* file, __int64& time);
{
__int64 newTime = stat_mtime(file);
if (newTime - time > 0)
{
time = newTime;
return true;
}
return false;
}
cout<<"abcd";
sleep(100);
cout<<'\b';
If I want to print the string out and then get back one character ,
why a sleep() is needed here?
But when using printf in C ,it seems that it is not necessary, why?
char* a = "12345";
char* b = "67890";
threadA(){cout<<a;}
threadB(){cout<<b;}
beginthread (threadA);
sleep(100);
beginthread (threadB);
In the second pseudo code above ,is it right to use sleep()?
For calculating tomorrow date:
void get_tomorrow_date( struct timeval *date )
{
sleep( 86400 ); // 60 * 60 * 24
gettimeofday( date, 0 );
}
;)
There are two subtle issues that you need to understand:
Multi-threading
I/O and Buffering
I'll try to give you some idea:
Multi-threading and sleep
Having a sleep in a threaded environment makes sense. The sleep call makes you wait thereby giving the initial thread some scope to have completed its processing i.e. writing out the string abcd to the standard output before the other thread inserts the backspace character. If you didn't wait for the first thread to complete its processing, you'd have written the backspace character first, and then the string abcd and wouldn't notice any difference.
Buffered I/o
I/O typically happens in buffered, non-buffered and semi-buffered states. This can influence how long, if at all, you have to wait for the output to appear on the console.
Your implementation of cout is probably using a buffered model. Try adding a newline or the endl at the end of your cout statements to print a new line and have it flush immediately, or use cout << "abcd" << flush; to flush without printing a new line.
In the second case without the sleep there's a slim chance that the second thread could start working before the first, resulting in the output "6789012345".
However a "sleep" isn't really the way to handle synchronisation between threads. You'd normally use a semaphore or similar in threadA() which threadB() has to wait for before doing its work.
The reason that the call to sleep makes your code work is because you are using it to turn the potentially parallel execution of the two output stream actions into a single, sequential action. The call to sleep() will allow the scheduler to switch away from the main thread of execution and execute thread A.
If you don't put sleep() in, the order of thread execution is not guaranteed and thread B could well start executing/printing before thread A had a chance to do that.
I think you need to understand what sleep does in general, and understand why it might exist.
sleep does what it sounds like. It instructs the OS to put the requesting task (where a task is a thread of execution) to sleep by removing it from the list of currently running processes and putting it on some sort of wait queue.
Note that there are also times when the OS will put you to sleep whether you like it or not. An example would be any form of blocking I/O, like reading a file from disk. The OS does this so that other tasks may get the CPU's attention while you're off waiting for your data.
One would use sleep voluntarily for similar purposes that the OS would. For example, if you have multiple threads and they're waiting on the completion of some computation, you'll probably want to voluntarily relinquish the CPU so that the computation can complete. You may also voluntarily relinquish the CPU so that other threads have a chance to run. For example, if you have a tight loop that's highly CPU-bound, you'll want to sleep now and then to give other threads a chance to run.
What it looks like you're doing is sleeping for the sake of something being flushed to stdout so that some other thread won't write to stdout before you. This, however, isn't guaranteed to work. It might work incidentally, but it's certainly not what you'd want to do by design. You'd either want to explicitly flush your buffer and not sleep at all, or use some form of synchronization.
As for why printf doesn't exhibit those issues... well, it's a crapshoot. Both printf and cout use some form of buffered output, but the implementation of each may be different.
In summary, it's probably best to remember the following:
When you want to synchronize, use synchronization primitives.
When you want to give someone else a chance to run, sleep.
The OS is better at deciding whether an I/O operation is blocking or not.
if you're having problems seeing the "abcd" being printed, it's because you're not giving cout an endline character to flush the buffer.
if you put
cout << "abcd" << endl;
you would be able to see the characters, then it would beep. no sleep necessary.
while( true )
{
msgStack.Lock();
process( msgStack.pop_msg());
msgStack.Unlock();
sleep(0);
}
sleep in the first example is just to print message a little before you will see "backspace" action. In the second example sleep "can" help. But it is weird. You won't be able to synchronize console outs with sleep in some more complex case.
In the code that launches two threads:
beginthread (threadA);
sleep(100);
beginthread (threadB);
the sleep() waits for 100 ms and then continues. The programmer probably did this in order to give threadA a chance to start up before launching threadB. If you must wait for threadA to be initialized and running before starting threadB, then you need a mechanism that waits for threadA to start, but this is the wrong way to do it.
100 is a magic cookie, chosen arbitrarily, probably accompanying a thought like "it should never take threadA more than 100 ms to start up." Assumptions like this are faulty because you have no way of knowing how long it will take for threadA to start. If the machine is busy or if the implementation of threadA changes it could easily take longer than 100 ms for the thread to launch, run its startup code, and get to it's main loop (if it is that kind of thread).
Instead of sleeping for some arbitrary amount of time, threadA needs to tell the main thread when it is up & running. One common way of doing this is by signaling an event.
Sample code that illustrates how to do this:
#include "stdafx.h"
#include <windows.h>
#include <process.h>
struct ThreadParam
{
HANDLE running_;
HANDLE die_;
};
DWORD WINAPI threadA(void* pv)
{
ThreadParam* param = reinterpret_cast<ThreadParam*>(pv);
if( !param )
return 1;
// do some initialization
// : :
SetEvent(param->running_);
WaitForSingleObject(param->die_, INFINITE);
return 0;
}
DWORD WINAPI threadB(void* pv)
{
ThreadParam* param = reinterpret_cast<ThreadParam*>(pv);
if( !param )
return 1;
// do some initialization
// : :
SetEvent(param->running_);
WaitForSingleObject(param->die_, INFINITE);
return 0;
}
int main(int argc, char** argv)
{
ThreadParam
paramA = {CreateEvent(0, 1, 0, 0), CreateEvent(0, 1, 0, 0) },
paramB = {CreateEvent(0, 1, 0, 0), CreateEvent(0, 1, 0, 0) };
DWORD idA = 0, idB = 0;
// start thread A, wait for it to initialize
HANDLE a = CreateThread(0, 0, threadA, (void*)¶mA, 0, &idA);
WaitForSingleObject(paramA.running_, INFINITE);
// start thread B, wait for it to initi
HANDLE b = CreateThread(0, 0, threadB, (void*)¶mB, 0, &idB);
WaitForSingleObject(paramB.running_, INFINITE);
// tell both threads to die
SetEvent(paramA.die_);
SetEvent(paramB.die_);
CloseHandle(a);
CloseHandle(b);
return 0;
}
It's not needed - what output do you get if you omit it?
The only thing sleep does is pauses execution on the calling thread for the specified number of milliseconds. It in no way will affect the outcome of any printing you might do.
Sleep can be used to avoid a certain thread/process (yeah, i know they are different things) hogging the processor.
On the other hand, printf is thread safe. Cout is not. That may explain differences in their behaviour.