I have a multi-threaded application(OpenMP), in which each thread do some calculations and find the value of _pre.
Then a file with name "file_pre.txt" is opened and write the part of the string starting from index _sub and _end.
It should not happen that two threads having the same file name open the file for writing.
Hence the writing part I wrote in critical section but it's expensive because I would like multiple threads having different file name to be able to write in the respective files simultaneously, and only lock them out when they have same value in _pre (same file name) (the updating thread could wait for the other threads to finish).
Does anyone have a solution?
PART OF CODE:
ofstream outfile;
#pragma omp parallel for num_threads(8) schedule(static) private(it) shared(array)
for (it = 0; it < array.size(); ++it)
{
int _pre, _sub = 0, _end;
if(---)
{
_pre = xyz;
_end = abc;
ostringstream strCounter;
strCounter << _pre;
string result = "file" + strCounter.str() + ".txt";
#pragma omp critical(outfile)
{
outfile.open(result.c_str(), std::ios_base::app);
if (outfile.is_open())
{
for (int count = _sub; count < _end; count++)
{
outfile << buffer[count];
}
outfile << "\n";
outfile.close();
} else cout << "Unable to open file";
_sub = _abc;
}
}
else
{
_pre = _xyz_;
_end = _abc_;
ostringstream strCounter;
strCounter << _pre;
string result = "file" + strCounter.str() + ".txt";
#pragma omp critical(outfile)
{
outfile.open(result.c_str(), std::ios_base::app);
if (outfile.is_open())
{
for (int count = _sub; count < _end; count++)
{
outfile << buffer[count];
}
outfile << "\n";
outfile.close();
} else cout << "Unable to open file";
_sub = _abc;
}
}
}
Related
I have 2 functions that the main run them as a thread. first one reads from a file of messages and pushes every line into a queue and the second function writes into a file the messages of the queue.
from some reason, the queue in the second function doesn't update and the function writes nothing to the output file
here is the first func:
void MessagesSender::saveMessages()
{
std::string line = "";
std::ifstream file;
file.open(DATA_FILE);
if (file.is_open())
{
while (getline(file, line))
{
m.lock();
messages.push(line);
m.unlock();
}
file.close();
file.open(DATA_FILE, std::ofstream::out | std::ofstream::trunc);//clear file
file.close();
}
}
Here is the second:
void MessagesSender::sendMessages()
{
std::ofstream file(OUTPUT_FILE);
m.lock();
std::queue<std::string> temp_messages = messages;//problem - "messages" is empty
m.unlock();
if (file.is_open())
{
int i = 0;
while (!temp_messages.empty())
{
while (i < userName.size())
{
file << userName[i] << ": " << temp_messages.front() << std::endl;
i++;
}
temp_messages.pop();
i = 0;
}
file.close();
}
else std::cout << "Unable to open file";
}
int main()
{
MessagesSender mailingList;
while (mailingList.menu()){}//shows to the user the menu
std::thread t1(&MessagesSender::saveMessages, mailingList);
std::thread t2(&MessagesSender::sendMessages, mailingList);
t1.join();
t2.join();
}
why "messages" is empty and how can I fixe it?
I'm writing a program to read data from .txt file and using it in output .txt file. I'm using two threads; The first thread is for reading the data from the .txt file, and the second one is for writing it in the output file. I'm a beginner in the programming with mutex and condition_variable and for somehow my program handles exceptions... The exception is
abort() has been called.
These are the two threads methods:
void MessagesSender::readsData()
{
ifstream data;
data.open("data.txt");
string buffer;
bool toEmpty = false;
std::unique_lock<mutex> locker(mtx, std::defer_lock);
if (data.is_open())
{
while (std::getline(data, buffer)) //reads line to the buffer.
{
locker.lock();
this->messages.push(buffer); //push the message to the queue.
locker.unlock();
cond.notify_one();
}
data.close();
toEmpty = true;
}
else
{
cout << "Error opening file... " << endl;
}
if (toEmpty) //empty the data file.
{
ofstream emptyFile;
emptyFile.open("data.txt", ofstream::out | ofstream::trunc);
emptyFile.close();
}
}
void MessagesSender::sendsData()
{
ofstream output;
output.open("output.txt");
string tempString;
string tempMessage;
if (output.is_open())
{
std::unique_lock<mutex> locker(mtx, std::defer_lock);
locker.lock();
cond.wait(locker);
while (!(this->messages.empty()))
{
tempMessage = this->messages.front();
this->messages.pop();
locker.unlock();
for (std::vector<string>::iterator it = this->userNames.begin(); it != this->userNames.end(); ++it)
{
tempString = *it;
tempString += ": ";
tempString += tempMessage;
tempString += "\n";
output << tempString;
}
}
output.close();
}
else
{
cout << "Error opening file... " << endl;
}
}
Why is the program handling exception?
One possible error is that you are repeatedly unlocking the mutex in your while-loop, even though the mutex is not locked:
if (output.is_open())
{
std::unique_lock<mutex> locker(mtx, std::defer_lock);
locker.lock();
cond.wait(locker);
while (!(this->messages.empty()))
{
tempMessage = this->messages.front();
this->messages.pop();
// if multiple messages are in the queue, you unlock multiple times
// even though the mutex is not locked
locker.unlock();
for (std::vector<string>::iterator it = this->userNames.begin(); it != this->userNames.end(); ++it)
{
tempString = *it;
tempString += ": ";
tempString += tempMessage;
tempString += "\n";
output << tempString;
}
}
output.close();
}
According to unique_lock::unlock this throws a std::system_error
i'm trying to multithread a program wich take word from file then hash these and write to another.
If i make it without multithreading it's really fast, it's able to use only 15-20% of the CPU and has out like 300.000line/s
But when i tryied with multithreading it only slow down and start to hash only at 17000lines/s, can you please help me?
Thanks
#include <iostream> // std::cout, std::streambuf, std::streamsize
#include <fstream> // std::ifstream
#include <string>
#include <thread>
#include "md5.h"
using namespace std;
static const int num_threads = 10;
void call_from_thread(int tid) {
cout << "Launched by thread " << tid << std::endl;
int cl = 0;
int uscita = 0;
int parole = 0;
char* contents;
ifstream istr("test.txt");
if (istr) {
streambuf * pbuf = istr.rdbuf();
streamsize size = pbuf->pubseekoff(0, istr.end);
pbuf->pubseekoff(0, istr.beg); // rewind
contents = new char[size];
pbuf->sgetn(contents, size);
istr.close();
ofstream myfile;
myfile.open("out.txt");
do {
string prova("");
uscita = 0;
do {
if (contents[cl] == '\n') {
uscita = 1;
}
prova += contents[cl];
cl += 1;
} while (uscita != 1);
parole += 1;
//cout << prova << ":" << md5(prova) << endl;
myfile << prova << ":" << md5(prova) << endl;
} while (parole != 9586054);
myfile.close();
}
}
int main()
{
thread t[num_threads];
//Launch a group of threads
for (int i = 0; i < num_threads; ++i) {
t[i] = thread(call_from_thread, i);
}
cout << "Launched from the main\n";
//Join the threads with the main thread
for (int i = 0; i < num_threads; ++i) {
t[i].join();
}
return 0;
}
What you see here is that all the threads are reading the same file and doing the same work. This results in that the file has to be read num_threads times, probably causing a slowdown of about num_threads (since file I/O probably is on the critical path) or maybe even more because of more cache misses.
You could try reading the entire file into memory, splitting it up between your threads and then having each thread process a subset of that files content.
Currently I've written a program (in Visual Studio C++) based on the available example code of "libsndfile" in which I've just added a subtraction-function to subtract one (channel of a) WAV-file with another.
The program worked perfectly for files with a limit of 4 channels.
The problem is that as soon as I upload a file with as many as 5 channels or more, the program refuses to execute the function. This is independent of the frame-size as I've managed to process 4-channel files which were many times larger than some 5/6-channel-files which I have used.
Has anyone encountered this or a similar problem before? If requested I will provide the source code.
My apologies for the bad structure, I don't really optimize until I get the whole deal working.
Code:
#include <sndfile.hh>
#include <tinyxml2.h>
#include <iostream>
#include <fstream>
#define BLOCK_SIZE 32
using TiXmlDocument = tinyxml2::XMLDocument;
using CStdStringA = std::string;
int main(int argc, char* argv[])
{
SNDFILE *infile = NULL;
SNDFILE *infile2 = NULL;
SNDFILE *outfile = NULL;
SF_INFO sfinfo;
SF_INFO sfinfo2;
SF_INFO sfinfoOut;
sf_count_t readcount;
//int filetype = SF_FORMAT_WAV | SF_FORMAT_PCM_24; // 24-bit Wav-Files
TiXmlDocument iDoc; // Init tinyXML
int i; // Iteration-var;
short BufferIn[BLOCK_SIZE]; // Buffer1
short BufferIn2[BLOCK_SIZE]; // Buffer2
short BufferOut[BLOCK_SIZE];
CStdStringA FileLoc, FileName, FileLocWav, FileLocTxt,FileLocRaw; // OutputFile locationstring
CStdStringA WavName, WavName2, FileIn, FileIn2;
FileLoc = "C:\\Testfiles\\";
//TextTitle(FileLoc); // Print console-header
printf("Name Wavefile 1:\n");
std::cin >> WavName;
FileIn = FileLoc + WavName + ".wav";
if((infile = sf_open(FileIn.c_str(),SFM_READ,&sfinfo)) == NULL) // Open Wav-file 2 instance
{
printf("Not able to open input file 1\n");
printf("\n\nPress any key to exit."); // Exit-text if file not present
puts(sf_strerror(NULL)) ;
return 1;
}
std::cout << "Wav file 1 succesfully opened." << std::endl;
std::cout << "\nChannels: " << sfinfo.channels << "\nFrames: " << sfinfo.frames << std::endl; // Print Channels & Frames
std::cout << "Samplerate: " << sfinfo.samplerate << "\n\n" << std::endl;// Print Samplerate
printf("Name Wavefile 2:\n");
std::cin >> WavName2;
FileIn2 = FileLoc + WavName2 + ".wav";
if((infile2 = sf_open(FileIn2.c_str(),SFM_READ,&sfinfo2)) == NULL) // Open Wav-file 2 instance
{
printf("Not able to open input file 2\n");
printf("\n\nPress any key to exit."); // Exit-text if file not present
puts(sf_strerror(NULL)) ;
return 1;
}
std::cout << "Wav file 2 succesfully opened." << std::endl;
std::cout << "\nChannels: " << sfinfo2.channels << "\nFrames: " << sfinfo2.frames << std::endl; // Print Channels & Frames
std::cout << "Samplerate: " << sfinfo2.samplerate << "\n\n" << std::endl;// Print Samplerate
//{
//std::cout << "Format differs from eachother, abort program.";
//printf("\n\nPress any key to exit."); // Exit-text
//return 1;
//}
if(sfinfo.channels != sfinfo2.channels) // Abort if channels not the same
{
std::cout << "Channelammount differs from eachother, abort program.";
printf("\n\nPress any key to exit."); // Exit-text
return 1;
}
if(sfinfo.samplerate != sfinfo2.samplerate) // Abort if samplerate not the same
{
std::cout << "Samplerate differs from eachother, abort program.";
printf("\n\nPress any key to exit."); // Exit-text
return 1;
}
std::cout << "Give a filename for Txt- & Wav-file: ";
std::cin >> FileName;
FileLoc += FileName; // Fuse Filelocation with given name
FileLocTxt = FileLoc + ".txt";
FileLocWav = FileLoc + ".wav";
FileLocRaw = FileLoc + "Raw.txt";
sfinfoOut.channels = sfinfo.channels;
sfinfoOut.format = sfinfo.format;
sfinfoOut.samplerate = sfinfo.samplerate;
if((outfile = sf_open(FileLocWav.c_str(),SFM_WRITE,&sfinfoOut)) == NULL) // Open Wav-file 2 instance
{
printf("Not able to create output file \n");
printf("\n\nPress any key to exit."); // Exit-text if file not present
puts(sf_strerror(NULL)) ;
return 1;
}
std::ofstream myfile;
myfile.open(FileLocTxt.c_str(),std::ios::app);
std::ofstream myfileRaw;
myfileRaw.open(FileLocRaw.c_str(),std::ios::app);
while((readcount = sf_read_short(infile, BufferIn, BLOCK_SIZE))) // While there are still frames to be processed
{
//process_data (data, readcount, sfinfo.channels) ;
auto readcount2 = sf_read_short(infile2, BufferIn2, BLOCK_SIZE);
for(i = 0; i < BLOCK_SIZE; i++) // BLOCK_SIZE decides the chunk-size
{
BufferOut[i] = BufferIn[i] - BufferIn2[i];
myfileRaw << BufferOut[i];
}
sf_write_short(outfile, BufferOut, BLOCK_SIZE) ; // Write the data to a new file
}
sf_close(infile); // Close Wav-file handlers
sf_close(infile2);
sf_close(outfile);
myfile.close(); // Close text-file handlers
printf("\n\nPress any key to exit."); // Exit-text
return 0;
}
The documentation states:
File Read Functions (Items)
sf_count_t sf_read_short (SNDFILE *sndfile, short *ptr, sf_count_t items) ;
sf_count_t sf_read_int (SNDFILE *sndfile, int *ptr, sf_count_t items) ;
sf_count_t sf_read_float (SNDFILE *sndfile, float *ptr, sf_count_t items) ;
sf_count_t sf_read_double (SNDFILE *sndfile, double *ptr, sf_count_t items) ;
The file read items functions fill the array pointed to by ptr with the requested number of items. The items parameter must be an integer product of the number of channels or an error will occur
There's your problem, since BLOCK_SIZE is a multiple of 1,2,4 but not of 3 and 5 - so that would fail.
About the problem that program completes as if calculations had taken place: add appropriate error handling.
As input, I have a list of gzipped files. As shown here, I use gzstream to handle them. For practical reasons, I want to open each file and record each stream into a vector. It seems pretty straightforward but I don't manage to make it work. Here is the minimal code:
#include <cstdlib>
#include <iostream>
#include <vector>
using namespace std;
#include <gzstream.h>
int main (int argc, char ** argv)
{
size_t i;
vector<string> vInFiles;
vector<igzstream *> vStreams;
string line;
// create the dummy input files
system ("rm -f infile*.gz; for i in {1..2}; do echo \"toto\"${i} | gzip > infile${i}.gz; done");
vInFiles.push_back ("infile1.gz");
vInFiles.push_back ("infile2.gz");
// open each input file
for (i = 0; i < vInFiles.size(); ++i)
{
igzstream inStream;
inStream.open (vInFiles[i].c_str());
if (! inStream.good())
{
cerr << "ERROR: can't open file " << vInFiles[i] << endl;
exit (1);
}
vStreams.push_back (&inStream);
}
// manipulate each input file
for (i = 0; i < vInFiles.size(); ++i)
{
cout << "read first line of file " << vInFiles[i] << endl;
getline (*(vStreams[i]), line);
if (line.empty())
{
cerr << "empty line" << endl;
exit (1);
}
cout << line << endl;
}
// close each input file
for (i = 0; i < vInFiles.size(); ++i)
{
vStreams[i]->close();
}
vStreams.clear();
return 0;
}
This code compiles properly:
$ gcc -Wall test.cpp -lstdc++ -lgzstream -lz
And although it run smoothly, it doesn't read the files properly:
$ ./a.out
read first line of file infile1.gz
empty line
Your stream pointers are invalid after the iteration ends, as the automatic stream object is destroyed then. If you really need that you need to allocate them on the free store (or make igzstream movable).
// std::vector<boost::shared_ptr<igzstream>> for C++03
std::vector<std::unique_ptr<igzstream>> vStreams;
// ...
for (size_t i = 0; i < vInFiles.size(); ++i) {
// boost::shared_ptr<igzstream> inStream = boost::make_shared<igzstream>();
auto inStream = std::unique_ptr<igzstream>(new igzstream);
inStream->open(...);
// ...
vStreams.push_back(inStream);
}
// ...
This is broken; you store a vector of pointers to streams, but you initialize it with a pointer to a locally scoped automatic instance of the stream (inside the for-loop). Once each iteration of the loop completes, that instance is out of scope, and you have a pointer to some crap.
You then use that crap later, and you get crap out.
Use a smart pointer, e.g.
std::vector<boost::shared_ptr<igzstream> > vStreams;
// to initialize
for (i = 0; i < vInFiles.size(); ++i)
{
boost::shared_ptr<igzstream> inStream(new igzstream(vInFiles[i].c_str());
if (!inStream->good())
{
cerr << "ERROR: can't open file " << vInFiles[i] << endl;
exit (1);
}
vStreams.push_back (inStream); // save the smart pointer
}
As mentioned in the comments, I would prefer not to use Boost and I only have gcc 4.1.2. Thus, here is the solution using the free store, thanks to the suggestion of Cat Plus Plus:
// open each input file
for (i = 0; i < vInFiles.size(); ++i)
{
igzstream * pt_inStream = new igzstream;
pt_inStream->open (vInFiles[i].c_str());
if (! pt_inStream->good())
{
cerr << "ERROR: can't open file " << vInFiles[i] << endl;
exit (1);
}
vStreams.push_back (pt_inStream);
}
And:
// close each input file
for (i = 0; i < vInFiles.size(); ++i)
{
vStreams[i]->close();
delete vStreams[i];
}