I know that construction of a string in a shared memory needs an allocator.
That's fine, but I can't find out how can I do that, because all examples are using a Managed Shared Memory which has a method of get_segment_manager() which has to be used as allocator (if I'm not wrong).
Let's see this example copied from here: https://www.boost.org/doc/libs/1_77_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.conditions.conditions_anonymous_example
doc_anonymous_condition_shared_data.hpp
#include <boost/interprocess/sync/interprocess_mutex.hpp>
#include <boost/interprocess/sync/interprocess_condition.hpp>
struct trace_queue
{
enum { LineSize = 100 };
trace_queue()
: message_in(false)
{}
//Mutex to protect access to the queue
boost::interprocess::interprocess_mutex mutex;
//Condition to wait when the queue is empty
boost::interprocess::interprocess_condition cond_empty;
//Condition to wait when the queue is full
boost::interprocess::interprocess_condition cond_full;
//Items to fill
char items[LineSize];
//Is there any message
bool message_in;
};
Main process
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>
#include <iostream>
#include <cstdio>
#include "doc_anonymous_condition_shared_data.hpp"
using namespace boost::interprocess;
int main ()
{
//Erase previous shared memory and schedule erasure on exit
struct shm_remove
{
shm_remove() { shared_memory_object::remove("MySharedMemory"); }
~shm_remove(){ shared_memory_object::remove("MySharedMemory"); }
} remover;
//Create a shared memory object.
shared_memory_object shm
(create_only //only create
,"MySharedMemory" //name
,read_write //read-write mode
);
try{
//Set size
shm.truncate(sizeof(trace_queue));
//Map the whole shared memory in this process
mapped_region region
(shm //What to map
,read_write //Map it as read-write
);
//Get the address of the mapped region
void * addr = region.get_address();
//Construct the shared structure in memory
trace_queue * data = new (addr) trace_queue;
const int NumMsg = 100;
for(int i = 0; i < NumMsg; ++i){
scoped_lock<interprocess_mutex> lock(data->mutex);
if(data->message_in){
data->cond_full.wait(lock);
}
if(i == (NumMsg-1))
std::sprintf(data->items, "%s", "last message");
else
std::sprintf(data->items, "%s_%d", "my_trace", i);
//Notify to the other process that there is a message
data->cond_empty.notify_one();
//Mark message buffer as full
data->message_in = true;
}
}
catch(interprocess_exception &ex){
std::cout << ex.what() << std::endl;
return 1;
}
return 0;
}
Second process:
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>
#include <iostream>
#include <cstring>
#include "doc_anonymous_condition_shared_data.hpp"
using namespace boost::interprocess;
int main ()
{
//Create a shared memory object.
shared_memory_object shm
(open_only //only create
,"MySharedMemory" //name
,read_write //read-write mode
);
try{
//Map the whole shared memory in this process
mapped_region region
(shm //What to map
,read_write //Map it as read-write
);
//Get the address of the mapped region
void * addr = region.get_address();
//Obtain a pointer to the shared structure
trace_queue * data = static_cast<trace_queue*>(addr);
//Print messages until the other process marks the end
bool end_loop = false;
do{
scoped_lock<interprocess_mutex> lock(data->mutex);
if(!data->message_in){
data->cond_empty.wait(lock);
}
if(std::strcmp(data->items, "last message") == 0){
end_loop = true;
}
else{
//Print the message
std::cout << data->items << std::endl;
//Notify the other process that the buffer is empty
data->message_in = false;
data->cond_full.notify_one();
}
}
while(!end_loop);
}
catch(interprocess_exception &ex){
std::cout << ex.what() << std::endl;
return 1;
}
return 0;
}
I'd like to replace char items[LineSize]; to a more convenient string in the trace_queue struct.
How can I do that without the Managed Shared Memory?
Or this is somewhat completely not recommended to do without the managed Boost libraries?
Or this is somewhat completely not recommended to do without the managed Boost libraries?
I cannot recommend it. It's fine to do it unmanaged, but I'd 100% suggest the exact approach they gave with the fixed char array. What's wrong with that?
You cannot have your cake and eat it. You can't wish for "highlevel dynamic strings" and "no heap management overhead" magically at the same time.
That said, you may be able to find some trade-offs. Specifically, you might want to emulate something like a polymorphic memory resource in such a shared byte array. Then you could use std::pmr::string on top of that. Tragedy has it that memory_resource isn't shared-memory safe.
SIMPLIFY
However, I suppose all you need is some nice abstraction, where the interface is using C++ vocabulary types. Why not simplfy the entire deal to that point?
Here's a quick draft:
struct trace_queue {
private:
bip::interprocess_mutex mutex;
bip::interprocess_condition cond;
std::array<char, 300> buffer{};
bool message_in{false}; // Is there any message
auto wait(bool state) {
bip::scoped_lock lock(mutex);
cond.wait(lock, [=,this] { return message_in == state; });
return lock;
}
public:
void send(std::string_view msg) {
auto lock = wait(false); // !message_in
auto n = std::min(buffer.size(), msg.size());
std::fill(buffer.begin(), buffer.end(), '\0');
std::copy_n(msg.data(), n, buffer.begin());
message_in = true;
cond.notify_one();
}
std::string receive() {
auto lock = wait(true); // message_in
std::string msg(buffer.data(), strnlen(buffer.data(), buffer.size()));
message_in = false;
cond.notify_one();
return msg;
}
};
In my opinion the code is already easier to read. And it's certainly easier to use! The entire server side:
// Create a shared memory object.
bip::shared_memory_object shm(bip::create_only, "MySharedMemory",
bip::read_write);
shm.truncate(sizeof(trace_queue));
// Map the whole shared memory in this process
bip::mapped_region region(shm, bip::read_write);
trace_queue& data = *new (region.get_address()) trace_queue;
for (int i = 0; i < 99; ++i)
data.send("my_trace_" + std::to_string(i));
data.send("TEARDOWN");
And the client side:
bip::shared_memory_object shm(bip::open_only, "MySharedMemory",
bip::read_write);
bip::mapped_region region(shm, bip::read_write);
trace_queue& data = *static_cast<trace_queue*>(region.get_address());
while (true) {
auto msg = data.receive();
if (msg == "TEARDOWN")
break;
std::cout << msg << "\n";
};
See it Live On Coliru
#include <array>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/sync/interprocess_condition.hpp>
#include <boost/interprocess/sync/interprocess_mutex.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>
#include <iostream>
namespace bip = boost::interprocess;
struct trace_queue {
private:
bip::interprocess_mutex mutex;
bip::interprocess_condition cond;
std::array<char, 300> buffer{};
bool message_in{false}; // Is there any message
auto wait(bool state) {
bip::scoped_lock lock(mutex);
cond.wait(lock, [=,this] { return message_in == state; });
return lock;
}
public:
void send(std::string_view msg) {
auto lock = wait(false); // !message_in
auto n = std::min(buffer.size(), msg.size());
std::fill(buffer.begin(), buffer.end(), '\0');
std::copy_n(msg.data(), n, buffer.begin());
message_in = true;
cond.notify_one();
}
std::string receive() {
auto lock = wait(true); // message_in
std::string msg(buffer.data(), strnlen(buffer.data(), buffer.size()));
message_in = false;
cond.notify_one();
return msg;
}
};
int main(int argc, char**) {
try {
if (argc < 2) {
// Erase previous shared memory and schedule erasure on exit
struct shm_remove {
shm_remove() { bip::shared_memory_object::remove("MySharedMemory"); }
~shm_remove() { bip::shared_memory_object::remove("MySharedMemory"); }
} remover;
// Create a shared memory object.
bip::shared_memory_object shm(bip::create_only, "MySharedMemory",
bip::read_write);
shm.truncate(sizeof(trace_queue));
// Map the whole shared memory in this process
bip::mapped_region region(shm, bip::read_write);
trace_queue& data = *new (region.get_address()) trace_queue;
for (int i = 0; i < 99; ++i)
data.send("my_trace_" + std::to_string(i));
data.send("TEARDOWN");
} else {
bip::shared_memory_object shm(bip::open_only, "MySharedMemory",
bip::read_write);
bip::mapped_region region(shm, bip::read_write);
trace_queue& data = *static_cast<trace_queue*>(region.get_address());
while (true) {
auto msg = data.receive();
if (msg == "TEARDOWN")
break;
std::cout << msg << "\n";
};
}
} catch (std::exception const& ex) {
std::cout << ex.what() << std::endl;
return 1;
}
}
Output, as expected:
Related
In a previous question, I asked how to implement asynchronous I/O. This code now works, except that at the end it never stops. It seems that aio_read reads starting at offset, for length, and if it is past the end of the file, the operation succeeds? This code builds and runs on Ubuntu 20.04LTS and successfully reads blocks 1-5, each 512 bytes, then when it runs out of file it keeps oscillating between block 4 and 5. It never terminates.
Here is the code:
#include <aio.h>
#include <fcntl.h>
#include <signal.h>
#include <unistd.h>
#include <condition_variable>
#include <cstring>
#include <iostream>
#include <thread>
using namespace std;
using namespace std::chrono_literals;
constexpr uint32_t blockSize = 512;
mutex readMutex;
bool readReady = false;
condition_variable cv;
bool operation_completed = false;
int fh;
int bytesRead;
void process(char* buf, uint32_t bytesRead) {
cout << "processing..." << endl;
usleep(100000);
}
void aio_completion_handler(sigval_t sigval) {
struct aiocb* req = (struct aiocb*)sigval.sival_ptr;
// check whether asynch operation is complete
int status;
if ((status = aio_error(req)) != 0) {
cout << "Error: " << status << '\n';
return;
}
int ret = aio_return(req);
bytesRead = req->aio_nbytes;
cout << "ret == " << ret << endl;
cout << (char*)req->aio_buf << endl;
unique_lock<mutex> readLock(readMutex);
operation_completed = true;
cv.notify_one();
}
void thready() {
char* buf1 = new char[blockSize];
char* buf2 = new char[blockSize];
aiocb cb;
char* processbuf = buf1;
char* readbuf = buf2;
fh = open("smallfile.dat", O_RDONLY);
if (fh < 0) {
throw std::runtime_error("cannot open file!");
}
memset(&cb, 0, sizeof(aiocb));
cb.aio_fildes = fh;
cb.aio_nbytes = blockSize;
cb.aio_offset = 0;
// Fill in callback information
/*
Using SIGEV_THREAD to request a thread callback function as a notification
method
*/
cb.aio_sigevent.sigev_notify_attributes = nullptr;
cb.aio_sigevent.sigev_notify = SIGEV_THREAD;
cb.aio_sigevent.sigev_notify_function = aio_completion_handler;
/*
The context to be transmitted is loaded into the handler (in this case, a
reference to the aiocb request itself). In this handler, we simply refer to
the arrived sigval pointer and use the AIO function to verify that the request
has been completed.
*/
cb.aio_sigevent.sigev_value.sival_ptr = &cb;
int cursor = 0;
int currentBytesRead = read(fh, buf1, blockSize); // read the 1st block
while (true) {
cb.aio_buf = readbuf;
operation_completed = false; // set predicate to true and wait until asynch changes it
cb.aio_offset = cursor;
aio_read(&cb); // each next block is read asynchronously
process(processbuf, currentBytesRead); // process while waiting
{
unique_lock<mutex> readLock(readMutex);
cv.wait( readLock, []{ return operation_completed; } );
}
if (!operation_completed)
break;
currentBytesRead = bytesRead; // make local copy of global modified by the asynch code
cursor += bytesRead;
if (currentBytesRead < blockSize) {
break; // last time, get out
}
cout << "back from wait" << endl;
swap(processbuf, readbuf); // switch to other buffer for next time
currentBytesRead = bytesRead; // create local copy
}
delete[] buf1;
delete[] buf2;
}
int main() {
try {
thready();
} catch (std::exception& e) {
cerr << e.what() << '\n';
}
return 0;
}
First, is the above code an appropriate way to do this to get the length of the file and figure out exactly how many reads to do?
Second, if this is so, fine, but how can aio_read just return success if I try to read past the end of file? Error status is always zero. I am confused about what it is supposed to do.
with 512 bytes of each of 1,2,3,4,5
In a previous questionTrying to write asynchronous I/O in C++ using locks and condition variables. This code calls terminate on the first lock() why?
,
we tried to use two mutexes to have asynchronous code that reads one block of a file into memory, then asynchronously tries to read the next block while processing the current one. Someone made a comment that using read was not the best way to do that. This is an attempt to use POSIX aio_read, but we are trying to wait on a condition_variable and do a notify on the condition variable in the callback after the I/O completes, and it's not working -- in the debugger we can see it blows right past the wait.
#include <aio.h>
#include <fcntl.h>
#include <signal.h>
#include <unistd.h>
#include <condition_variable>
#include <cstring>
#include <iostream>
#include <thread>
using namespace std;
using namespace std::chrono_literals;
constexpr uint32_t blockSize = 512;
mutex readMutex;
mutex procMutex;
condition_variable cv;
int fh;
int bytesRead;
void process(char* buf, uint32_t bytesRead) {
cout << "processing..." << endl;
usleep(100000);
}
void aio_completion_handler(sigval_t sigval) {
struct aiocb* req = (struct aiocb*)sigval.sival_ptr;
// check whether asynch operation is complete
if (aio_error(req) == 0) {
int ret = aio_return(req);
cout << "ret == " << ret << endl;
cout << (char*)req->aio_buf << endl;
}
cv.notify_one();
}
void thready() {
char* buf1 = new char[blockSize];
char* buf2 = new char[blockSize];
aiocb cb;
char* processbuf = buf1;
char* readbuf = buf2;
fh = open("smallfile.dat", O_RDONLY);
if (fh < 0) {
throw std::runtime_error("cannot open file!");
}
memset(&cb, 0, sizeof(aiocb));
cb.aio_fildes = fh;
cb.aio_nbytes = blockSize;
cb.aio_offset = 0;
// Fill in callback information
/*
Using SIGEV_THREAD to request a thread callback function as a notification
method
*/
cb.aio_sigevent.sigev_notify_attributes = nullptr;
cb.aio_sigevent.sigev_notify = SIGEV_THREAD;
cb.aio_sigevent.sigev_notify_function = aio_completion_handler;
/*
The context to be transmitted is loaded into the handler (in this case, a
reference to the aiocb request itself). In this handler, we simply refer to
the arrived sigval pointer and use the AIO function to verify that the request
has been completed.
*/
cb.aio_sigevent.sigev_value.sival_ptr = &cb;
int currentBytesRead = read(fh, buf1, blockSize); // read the 1st block
unique_lock<mutex> readLock(readMutex);
while (true) {
cb.aio_buf = readbuf;
aio_read(&cb); // each next block is read asynchronously
process(processbuf, currentBytesRead); // process while waiting
cv.wait(readLock);
if (currentBytesRead < blockSize) {
break; // last time, get out
}
cout << "back from wait" << endl;
swap(processbuf, readbuf); // switch to other buffer for next time
currentBytesRead = bytesRead; // create local copy
}
delete[] buf1;
delete[] buf2;
}
int main() {
try {
thready();
} catch (std::exception& e) {
cerr << e.what() << '\n';
}
return 0;
}
We were trying to create C++ code that would read a block from a file, and start a thread to asynchronously read the next block while processing the first block.
We started with condition_variable, but it was crashing so we went with straight locks.
The program dies on the first readLock.lock()
Ok, as some comments explained, the following code is wrong because we are not using RAII. Corrected code follows after, with delays put in for each function to force bugs. It still crashes though not on the first lock.
#include <fcntl.h>
#include <unistd.h>
#include <condition_variable>
#include <iostream>
#include <thread>
using namespace std;
using namespace std::chrono_literals;
constexpr uint32_t blockSize = 32768;
char* buf1;
char* buf2;
char* processbuf = buf1;
char* readbuf = buf2;
mutex readMutex;
mutex procMutex;
//condition_variable readCV;
//condition_variable procCV;
int fh;
int bytesRead;
std::unique_lock<std::mutex> readLock;
std::unique_lock<std::mutex> procLock;
void nextRead() {
while (true) {
readLock.lock();
bytesRead = read(fh, readbuf, blockSize);
for (int i = 0; i < 5; i++) {
cerr << "reading..." << endl;
usleep(100000);
readLock.unlock();
}
cerr << "notifying..." << endl;
readLock.unlock();
if (bytesRead != blockSize) // last time, end here!
return;
procLock.lock();
procLock.unlock();
}
}
void process(char* buf, uint32_t bytesRead) { cout << "hit it!" << endl; }
int thready() {
buf1 = new char[blockSize];
buf2 = new char[blockSize];
fh = open("bigfile.dat", O_RDONLY);
if (fh < 0) {
throw std::runtime_error("cannot open file!");
}
int currentBytesRead = read(fh, buf1, blockSize);
thread reader(nextRead);
process(processbuf, currentBytesRead);
while (true) {
readLock.lock();
cout << "back from wait" << endl;
swap(processbuf, readbuf); // switch to other buffer for next time
currentBytesRead =
bytesRead; // copy locally so thread can do the other one
// TODO: is the above a problem? what if
readLock.unlock();
procLock.lock();
process(processbuf, currentBytesRead);
procLock.unlock();
}
reader.join();
delete[] buf1;
delete[] buf2;
}
int main() {
try {
thready();
}catch(std::exception& e) {
cerr << e.what() << '\n';
}
return 0;
}
Here is the corrected code using RAII which still does not work. Now it terminates on the 3rd read? The file is 1Mb of zeros.
I am considering changing block size to 1 byte and having the file be "123456789" for purposes of easy testing
:
#include <fcntl.h>
#include <unistd.h>
#include <condition_variable>
#include <iostream>
#include <thread>
using namespace std;
using namespace std::chrono_literals;
constexpr uint32_t blockSize = 32768;
char* buf1;
char* buf2;
char* processbuf = buf1;
char* readbuf = buf2;
mutex readMutex;
mutex procMutex;
//condition_variable readCV;
//condition_variable procCV;
int fh;
int bytesRead;
void nextRead() {
while (true) {
{
unique_lock<mutex> readLock(readMutex);
bytesRead = read(fh, readbuf, blockSize);
for (int i = 0; i < 5; i++) {
cerr << "reading..." << endl;
usleep(100000);
readLock.unlock();
}
cerr << "notifying..." << endl;
}
if (bytesRead != blockSize) // last time, end here!
return;
// wait for process to complete
unique_lock<mutex> procLock(procMutex);
}
}
void process(char* buf, uint32_t bytesRead) {
cout << "processing..." << endl;
usleep(100000);
}
int thready() {
buf1 = new char[blockSize];
buf2 = new char[blockSize];
fh = open("bigfile.dat", O_RDONLY);
if (fh < 0) {
throw std::runtime_error("cannot open file!");
}
int currentBytesRead = read(fh, buf1, blockSize);
thread reader(nextRead);
process(processbuf, currentBytesRead);
while (true) {
{
unique_lock<mutex> readLock(readMutex);
cout << "back from wait" << endl;
swap(processbuf, readbuf); // switch to other buffer for next time
currentBytesRead = bytesRead; // create local copy
// TODO: is the above a problem? what if
}
unique_lock<mutex> procLock(procMutex);
process(processbuf, currentBytesRead);
}
reader.join();
delete[] buf1;
delete[] buf2;
}
int main() {
try {
thready();
} catch(std::exception& e) {
cerr << e.what() << '\n';
}
return 0;
}
The output is:
processing...
reading...
reading...back from wait
processing...
back from wait
processing...
terminate called after throwing an instance of 'std::system_error'
what(): Operation not permitted
Aborted (core dumped)
Your mutexes are shared between threads as globals. That's fine. But your locks need to be local variables owned by a single thread. Use the fact that unique_lock auto locks the mutex in its constructor and auto-releases it its destructor. Here's a modificaton of your code:
void nextRead() {
while (true) {
{
// acquire the read lock
std::unique_lock<std::mutex> readLock(readMutex);
bytesRead = read(fh, readbuf, blockSize);
for (int i = 0; i < 5; i++) {
cerr << "reading..." << endl;
usleep(100000);
}
// when readLock goes out of scope, the mutex associated with it is unlocked
}
cerr << "notifying..." << endl;
{
// acquire the process lock
std::unique_lock<std::mutex> procLock(procMutex);
if (bytesRead != blockSize) // last time, end here!
return;
// procMutex gets unlocked inmplicitly as procLock goes out of scope
}
}
You'll need to make similar changes in your thready function.
I have the program to count all words in all .log files in given directory using N threads.
I wrote something like this.
ThreadPool.h
#ifndef THREAD_POOL_H
#define THREAD_POOL_H
#include <boost/thread/condition_variable.hpp>
#include <boost/thread.hpp>
#include <future> // I don't how to work with boost future
#include <queue>
#include <vector>
#include <functional>
class ThreadPool
{
public:
using Task = std::function<void()>; // Our task
explicit ThreadPool(int num_threads)
{
start(num_threads);
}
~ThreadPool()
{
stop();
}
template<class T>
auto enqueue(T task)->std::future<decltype(task())>
{
// packaged_task wraps any Callable target
auto wrapper = std::make_shared<std::packaged_task<decltype(task()) ()>>(std::move(task));
{
boost::unique_lock<boost::mutex> lock{ mutex_p };
tasks_p.emplace([=] {
(*wrapper)();
});
}
event_p.notify_one();
return wrapper->get_future();
}
/*void enqueue(Task task)
{
{
boost::unique_lock<boost::mutex> lock { mutex_p };
tasks_p.emplace(std::move(task));
event_p.notify_one();
}
}*/
private:
std::vector<boost::thread> threads_p; // num of threads
std::queue<Task> tasks_p; // Tasks to make
boost::condition_variable event_p;
boost::mutex mutex_p;
bool isStop = false;
void start(int num_threads)
{
for (int i = 0; i < num_threads; ++i)
{
// Add to the end our thread
threads_p.emplace_back([=] {
while (true)
{
// Task to do
Task task;
{
boost::unique_lock<boost::mutex> lock(mutex_p);
event_p.wait(lock, [=] { return isStop || !tasks_p.empty(); });
// If we make all tasks
if (isStop && tasks_p.empty())
break;
// Take new task from queue
task = std::move(tasks_p.front());
tasks_p.pop();
}
// Execute our task
task();
}
});
}
}
void stop() noexcept
{
{
boost::unique_lock<boost::mutex> lock(mutex_p);
isStop = true;
}
event_p.notify_all();
for (auto& thread : threads_p)
{
thread.join();
}
}
};
#endif
main.cpp
#include "ThreadPool.h"
#include <iostream>
#include <iomanip>
#include <Windows.h>
#include <chrono>
#include <vector>
#include <map>
#include <boost/filesystem.hpp>
#include <boost/thread.hpp>
#include <locale.h>
namespace bfs = boost::filesystem;
//int count_words(boost::filesystem::ifstream& file)
//{
// int counter = 0;
// std::string buffer;
// while (file >> buffer)
// {
// ++counter;
// }
//
// return counter;
//}
//
int count_words(boost::filesystem::path filename)
{
boost::filesystem::ifstream ifs(filename);
return std::distance(std::istream_iterator<std::string>(ifs), std::istream_iterator<std::string>());
}
int main(int argc, const char* argv[])
{
std::cin.tie(0);
std::ios_base::sync_with_stdio(false);
bfs::path path = argv[1];
// If this path is exist and if this is dir
if (bfs::exists(path) && bfs::is_directory(path))
{
// Number of threads. Default = 4
int n = (argc == 3 ? atoi(argv[2]) : 4);
ThreadPool pool(n);
// Container to store all filenames and number of words inside them
//std::map<bfs::path, std::future<int>> all_files_and_sums;
std::vector<std::future<int>> futures;
auto start = std::chrono::high_resolution_clock::now();
// Iterate all files in dir
for (auto& p : bfs::directory_iterator(path)) {
// Takes only .txt files
if (p.path().extension() == ".log") {
// Future for taking value from here
auto fut = pool.enqueue([p]() {
// In this lambda function I count all words in file and return this value
int result = count_words(p.path());
static int count = 0;
++count;
std::ostringstream oss;
oss << count << ". TID, " << GetCurrentThreadId() << "\n";
std::cout << oss.str();
return result;
});
// "filename = words in this .txt file"
futures.emplace_back(std::move(fut));
}
}
int result = 0;
for (auto& f : futures)
{
result += f.get();
}
auto stop = std::chrono::high_resolution_clock::now();
auto duration = std::chrono::duration_cast<std::chrono::seconds>(stop - start);
std::cout << "Result: " << result << "\n";
std::cout << duration.count() << '\n';
}
else
std::perror("Dir is not exist");
}
Variable N is 4(Number of threads). I've 320 .log files in my directory and I need count words in this files. Everything works fine but when variable "count" is 180 - the program stops for a while and then continues but much slower.
What could be the reason?
CPU - Xeon e5430 (I have tested this program on another CPU - the result is the same).
It depends on how you measure "slow" but basically you are using one of the worst models possible:
one task queue shared between all threads.
The problem with this approach is blocking in each thread on the shared queue.
A much better model is something like
task stealing - you can try creating a task queue pro thread and then use try_lock (which doesnt block) with enabling each thread "stealing" work from some other thread's tasks if it has nothing else to do.
This is very nice explained in excellent Sean Parent Talk about Concurrency.
I see an example in boost ipc (inter process communication)
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>
#include <iostream>
#include <cstdio>
#include "doc_anonymous_condition_shared_data.hpp"
using namespace boost::interprocess;
int main ()
{
//Erase previous shared memory and schedule erasure on exit
struct shm_remove
{
shm_remove() { shared_memory_object::remove("MySharedMemory"); }
~shm_remove(){ shared_memory_object::remove("MySharedMemory"); }
} remover;
//Create a shared memory object.
shared_memory_object shm
(create_only //only create
,"MySharedMemory" //name
,read_write //read-write mode
);
try{
//Set size
shm.truncate(sizeof(trace_queue));
//Map the whole shared memory in this process
mapped_region region
(shm //What to map
,read_write //Map it as read-write
);
//Get the address of the mapped region
void * addr = region.get_address();
//Construct the shared structure in memory
trace_queue * data = new (addr) trace_queue;
const int NumMsg = 100;
for(int i = 0; i < NumMsg; ++i){
scoped_lock<interprocess_mutex> lock(data->mutex);
if(data->message_in){
data->cond_full.wait(lock);
}
if(i == (NumMsg-1))
std::sprintf(data->items, "%s", "last message");
else
std::sprintf(data->items, "%s_%d", "my_trace", i);
//Notify to the other process that there is a message
data->cond_empty.notify_one();
//Mark message buffer as full
data->message_in = true;
}
}
catch(interprocess_exception &ex){
std::cout << ex.what() << std::endl;
return 1;
}
return 0;
}
There is no delete operator in the example. Probably new operator used in memory region place and it can not use with delete operator. If I need to call destructor, I should simply call directly:
data->~trace_queue();
Am I right?
Yes, you are right as Joachim commented.
However, I'd suggest using managed_shared_memory which has the find<T>, find_or_construct<T> or construct<T> to make your life easier.
While you're at it, if you need to store many object of the same type, consider using a std::vector (or boost::container::vector) of that type, with boost::interprocess::allocator.