My application needs to read files from a nfs mount point. I'm using QFile::read(BUFFER_SIZE); which works as it is supposed to. The problem is, when the nfs unmounts (patchy network issues), this QFile::read() never timesout resulting in application hang. Any sensible solutions please?
Thanks.
You can use QFile::bytesAvailable () inherited from QIODevice which returns the number of bytes that are available for reading. Wait in a while loop until the bytes are available or some seconds elapsed :
QFile file("file.dat");
file.open(QIODevice::ReadOnly);
QElapsedTimer timer;
timer.start();
while(file.bytesAvailable()<BUFFER_SIZE || timer.nsecsElapsed()<3e9)
qApp->processEvents();
QByteArray array
if(file.waitForReadyRead())
{
array = file.read(BUFFER_SIZE);
}
Here it will wait 3 seconds for the data to be ready. The elapsed time is measured by QElapsedTimer. If BUFFER_SIZE bytes are not available after the timeout then it will continue with out hanging.
I think this is due to how NFS works. The process is stuck in the OS system call, in uninterruptible sleep. So, I think you have two reliable options:
Move file reading to another thread. It and its event loop may get stuck, just code your main thread so that it won't matter for the whole application. There are a few ways to go about using threads for this, but if you are not reading arbitrary number of different files at the same time, it'd probably be best to launch a dedicated thread for reading the single file, and then let the thread finish once everything is read. For this, you might even just subclass QThread, and override run() method with your own reading loop (you can still emit signals for progress if you want, you just can't receive them queued without Qt's event loop).
Move file reading to another process. In other words, write a small tool program or script which does the actual reading, and sends the contents to your main process. That way you should be able to kill the reading process without killing the whole application, should that become necessary. Again you have to decide if you want a separate single-use process for each reading (so you can give file name as command line paramater, for example), or if you launch one child process and implement communication with it so you can request files (or parts of files). Qt's QProcess makes this approach easy, too.
You can use QIODevice::readChannelFinished() signal and connect it to a slot that close the file reader
Related
I have a file with several integers (each one in a new line). I want to read the file every 30ms and process each of these integers.
Software : C++
Present idea :
1) In the main(), use file input/ouput to continuously read from the file and sleep for 30ms everytime.
2) Everytime I read an integer, I create a new thread which processes this integer.
Will the main() be suspended till the new thread finishes its operation? or will it also run in parallel?
Is there any better approach to doing the same process?
The idea works in theory. You do create a lot of threads, which is inefficient. And if you don't have enough CPU cores, at a certain moment the main thread won't get scheduled every 30 ms anymore so you'll fall behind.
The better solution is to have a threadpool whose threads pick up numbers that the main thread has made available.
I have a C++ function called write() that is supposed to write a large number of bytes on the users HDD. This woud normally take more than 30 seconds, so I want to give the user the ability to Abort operation from a GUI (created with Qt).
So I have the button generate_button. When this button is clicked(), I want to have something to end the function, wherever its progress is.
I thought about threads, but I am not sure. Can you advice me please?
I would probably use a thread. It should be quite simple to check a variable to see if the operation has been canceled.
Use a mutex to lock access to your cancel variable. That will make sure it is read and written in a proper way for multiple threads. Another option is if you are using C++11 use an atomic variable.
Break your large write into blocks of smaller size. 8 to 64 kilobytes should work. After writing each block check your cancel variable and if set, exit the thread.
Place the code that actually does the writing in a worker thread. Have a shared variable (one that is either atomic, or protected by a mutex). Have the worker thread check its value each iteration. If the user presses the "Abort" button, set the value for the variable.
You should use threads if this is a long running operation.
Since you are using C++11, std::atomic<bool> would probably serve you well.
Threaded guarantees that you will have a responsive GUI. But there is a learning curve to using a thread in this manner.
A threadless way to do this is to have in your routine that writes to the harddrive in the GUI thread, but gives time to the GUI thread to stay responsive.
QObject::connect(my_cancel_button, SIGNAL(clicked()), file_writer, SLOT(setCanceled()));
// open file for writing
QFile file("filename.txt");
file.open(//... );//
while(still_have_data_to_write && !canceled)
{
write( <1 MB of data> ); // or some other denomination of data
qApp->processEvents();// allows the gui to respond to events such as clicks on buttons
// update a progress bar... using a counter as a ratio of the total file size
emit updateProgressBar(count++);
}
if( canceled )
{
file.close();
// delete the partial file using QDir
}
Hope that helps.
I'm developing an application over Qt.
In this application the main thread is a web server. Another thread sometimes read data from big files (250mb) and write them in a output file (~2gb).
This thread performs high I/O operation on file, and CPU iowait is around 70%.
My problem is that when writing into the file, the web server is not responding quickly. What i understood is that the server's qt socket (on Linux) is represented by a system socket connected to the poll or select event system. So Qt send signal to my application only when poll emit event.
What i think is that too huge io operation with file writing may block the poll system, so my qt server doesn't receive socket event. When the thread has finished to write its data, everything become normal.
The file writing look like this:
while(dataToRead){
// context has the list of files to read and current step
dataToRead = extractData(context, &pBuffer, &sizeBuf);
fwrite (pBuffer, 1, sizeBuf, pOutFile);
free(pBuffer);
pBuffer = NULL;
// usleep(100000);
}
If i add a break with usleep function, this help to avoid the problem but not completely if i don't use a big enough sleep. But too big sleep destroy the performance, and i was the file generated as fast as possible.
What i'm doing wrong? Is it safe to read/write into a file as fast as possible? Is a sleep is mandatory in the above function? But how can we know the good timeslice?
I'm working on Mint LMDE, Linux 3.2.0 64 bits with Intel Core i5 2500 and standard HDD drive.
Edit:
A sample program that reproduce the problem is available here: https://bugreports.qt-project.org/secure/attachment/30436/TestQtBlocked.zip. Need qt's qmake to compile it. If you run it, it will create an empty 3GB file, the worker thread will be launched at startup and will create the file during few seconds. During this time, if you try to connect to http:// localhost:8081/ and you run many F5 to refresh the page, you will see that sometime it's not responding quickly.
If could be helpful if someone can reproduce my problem with my sample program, and let me know.
If you are starving the main thread's select calls, create a separate thread to do the file I/O. When the event comes from Qt, trigger some kind IPC that wakes up your worker thread to do the big file I/O and return from your event handler immediately.
(This assumes that writing to the file asynchronously makes sense to your program logic. Only you can figure out if that is true.)
from the man page:
size_t fwrite(const void *ptr, size_t size, size_t nmemb,
FILE *stream);
You want to write sizeBuf , 1 element.
You may want to tune buffering with setvbuf.
setvbuf(pOutfile, NULL, _IONBF, 0) - to disable buffering.
Complete example at:
http://publib.boulder.ibm.com/infocenter/iseries/v7r1m0/index.jsp?topic=%2Frtref%2Fsetvbuf.htm
better switch to work with file descritors, not file streams.
Using file descriptors you can use sendfile and slice.
man sendfile
man slice
I have a C++ console app that uses open() [O_RDWR | O_NONBLOCK], write(), select(), read() and close() to work with device file. Also ioctl() can be called to cancel current operation. At any given time only one user can work with device.
I need to come up with C++ class having libsigc++ signals that get fired when data is available from device.
The problem: when calling select() application becomes unresponsive as it waits for the data. How to make it responsive - by calling select() in worker thread? If so - how will worker thread communicate with main thread? Maybe I should look into boost::asio?
How to make it responsive - by calling select() in worker thread
you can use dup(), this will duplicated your file descriptors... thus you can move entire read operations into another thread. thus your write thread and processing thread will be responsive, even when the read [select()] thread is in sleeping.
signal emitting overhead of libsigc++ is minimal, thus i think you can embedded code inside the read thread itself. slots can exist in different thread, this is where you will receive your signals...
I think Thrift source code [entirely boost based] might be of your interest, though thrift does not use libsigc++.
It sounds as though you've misunderstood select; the purpose of select (or poll, epoll, etc) is not "wait for data" but "wait for one or more events to occur on a series of file descriptors or a timer, or a signal to be raised".
What "responsiveness" is going missing while you're in your select call? You said it's a console app so you're not talking about a GUI loop, so presumably it is IO related? If so, then you need to refactor your select so that waiting for the data you're talking about is one element; that is, if you're using select, build FD_SETs of ALL file/socket descriptors (and stdin and stdout are file descriptors) that you want to wait on input for.
Or build a loop that periodically calls "select" with a short timeout to /test/ for any pending input and only try and read it when select tells you there is something to read.
It sounds like you have a producer-consumer style problem. There are various way to implement a solution to this problem, but most folks these days tend to use condition variable based approaches (see this C++11 based example).
There are also a number of design patterns that when implemented can help alleviate your concurrency problem, such as:
Half-Sync / Half-Async
A producer-consumer style pattern that introduces a queue between an asynchronous layer that fills the queue with events, and a synchronous layer that processes those events.
Leader / Followers
Multiple threads take turns handling events
A related discussion is available here.
I have this tool in which a single log-like file is written to by several processes.
What I want to achieve is to have the file truncated when it is first opened, and then have all writes done at the end by the several processes that have it open.
All writes are systematically flushed and mutex-protected so that I don't get jumbled output.
First, a process creates the file, then starts a sequence of other processes, one at a time, that then open the file and write to it (the master sometimes chimes in with additional content; the slave process may or may not be open and writing something).
I'd like, as much as possible, not to use more IPC that what already exists (all I'm doing now is writing to a popen-created pipe). I have no access to external libraries other that the CRT and Win32 API, and I would like not to start writing serialization code.
Here is some code that shows where I've gone:
// open the file. Truncate it if we're the 'master', append to it if we're a 'slave'
std::ofstream blah(filename, ios::out | (isClient ? ios:app : 0));
// do stuff...
// write stuff
myMutex.acquire();
blah << "stuff to write" << std::flush;
myMutex.release();
Well, this does not work: although the output of the slave process is ordered as expected, what the master writes is either bunched together or at the wrong place, when it exists at all.
I have two questions: is the flag combination given to the ofstream's constructor the right one ? Am I going the right way anyway ?
If you'll be writing a lot of data to the log from multiple threads, you'll need to rethink the design, since all threads will block on trying to acquire the mutex, and in general you don't want your threads blocked from doing work so they can log. In that case, you'd want to write your worker thread to log entries to queue (which just requires moving stuff around in memory), and have a dedicated thread to pull entries off the queue and write them to the output. That way your worker threads are blocked for as short a time as possible.
You can do even better than this by using async I/O, but that gets a bit more tricky.
As suggested by reinier, the problem was not in the way I use the files but in the way the programs behave.
The fstreams do just fine.
What I missed out is the synchronization between the master and the slave (the former was assuming a particular operation was synchronous where it was not).
edit: Oh well, there still was a problem with the open flags. The process that opened the file with ios::out did not move the file pointer as needed (erasing text other processes were writing), and using seekp() completely screwed the output when writing to cout as another part of the code uses cerr.
My final solution is to keep the mutex and the flush, and, for the master process, open the file in ios::out mode (to create or truncate the file), close it and reopen it using ios::app.
I made a 'lil log system that has it's own process and will handle the writing process, the idea is quite simeple. The proccesses that uses the logs just send them to a pending queue which the log process will try to write to a file. It's like batch procesing in any realtime rendering app. This way you'll grt rid of too much open/close file operations. If I can I'll add the sample code.
How do you create that mutex?
For this to work this needs to be a named mutex so that both processes actually lock on the same thing.
You can check that your mutex is actually working correctly with a small piece of code that lock it in one process and another process which tries to acquire it.
I suggest blocking such that the text is completely written to the file before releasing the mutex. I've had instances where the text from one task is interrupted by text from a higher priority thread; doesn't look very pretty.
Also, put the format into Comma Separated format, or some format that can be easily loaded into a spreadsheet. Include thread ID and timestamp. The interlacing of the text lines shows how the threads are interacting. The ID parameter allows you to sort by thread. Timestamps can be used to show sequential access as well as duration. Writing in a spreadsheet friendly format will allow you to analyze the log file with an external tool without writing any conversion utilities. This has helped me greatly.
One option is to use ACE::logging. It has an efficient implementation of concurrent logging.