taken from here:
http://xerces.apache.org/xerces-c/faq-parse-3.html#faq-6
When two or more parser instances exist in a process, the instances can be used
concurrently, without external synchronization. That is, in an application containing two
parsers and two threads, one parser can be running within the first thread concurrently with
the second parser running within the second thread.
But the below code fails whenever the QMutex is commented and does not whenever it is used.
bool reports::validateSchema( QString fileName )
{
// QMutexLocker lock( &xercesMutex );
try
{
XMLPlatformUtils::Initialize();
}
catch(...)
{
this->throw_report_exception(__FILE__,__LINE__,__TIME__,__DATE__,"reports::validateSchema",
"unable to initialize Xerces Plateform");
return false;
}
const char* const xsd = "full absloute path to .xsd ==> hard written";
XercesDOMParser* parser = new XercesDOMParser();
try
{
parser->setValidationSchemaFullChecking(true);
parser->setDoSchema(true);
parser->setDoNamespaces(true);
parser->setValidationConstraintFatal(true);
parser->setValidationScheme(XercesDOMParser::Val_Auto);
ParserErrorHandler errHandler;
parser->setErrorHandler(&errHandler);
parser->cacheGrammarFromParse(true);
parser->loadGrammar(xsd,Grammar::SchemaGrammarType,true);
parser->parse(fileName.toStdString().c_str());
std::cout << parser->getErrorCount() << std::endl;
if(parser->getErrorCount()!=0)
{
return false;
}
}
catch (const XMLException& toCatch)
{
char* message = XMLString::transcode(toCatch.getMessage());
std::cout << "Exception message is: \n"
<< message << "\n";
XMLString::release(&message);
return false;
}
catch (const DOMException& toCatch)
{
char* message = XMLString::transcode(toCatch.msg);
std::cout << "Exception message is: \n"
<< message << "\n";
XMLString::release(&message);
return false;
}
catch (...)
{
std::cout << "Unexpected Exception \n" ;
return false;
}
delete parser;
XMLPlatformUtils::Terminate();
return true;
}
What am I missing ?
The functions get executed hundreds of times and at some point, I'm getting a segfault from either:
XercesDOMParser* parser = new XercesDOMParser();
or
parser->loadGrammar(xsd,Grammar::SchemaGrammarType,true);
A quote from the same FAQ answer you refer to:
The application also needs to guarantee that the XMLPlatformUtils::Initialize() and
XMLPlatformUtils::Terminate() methods are called from the same thread (usually the
initial thread executing main()) or proper synchronization is performed by the application
if multiple threads call XMLPlatformUtils::Initialize() and XMLPlatformUtils::Terminate()
concurrently.
Related
I created a C++ wrapper to access my Python modules. everything is working until I try to use threads in my application.
On my Python module there is a method which reads from a webcam (so its uses an infinite loop) and I send callbacks from C++ to get the image and other needed information from it.
Since we have a blocking method here, I decided to use threads.
The threading on Python part seems not to be working on the C++ side that is if I call the async counter part of the webcam_feed loop, none of my callbacks are actually executed (on python part the routines are all executed however, it seems it doesn't reach to C++ section somehow. I don't get any feedback in C++ side, however, on Python part, those routines responsible for executing the callbacks save the info to the disk so I know for sure they are executed).
I asked a separate question for it here.
Therefore I decided to use the threading inside C++ client. However, whenever I execute the code (given below), I get an access violation whenever I want to use any methods after the thread is started.
Here are the sample callbacks I have for now:
void default_callback(bool status, std::string id, py::array_t<uint8_t>& img)
{
auto rows = img.shape(0);
auto cols = img.shape(1);
auto type = CV_8UC3;
cv::Mat img1(rows, cols, type, img.mutable_data());
cv::imshow("from callback", img1);
cv::waitKey(1);
auto timenow = std::chrono::system_clock::to_time_t(std::chrono::system_clock::now());
std::cout << "\narg1: " << status << " arg2: " << id << " arg3: " << typeid(img).name() << " " << ctime(&timenow) << std::endl;
}
void default_c_callback_temporary(bool status, char* message)
{
std::cout << "status is: " << status << " id/name: " << message << " ptr:" << "" << std::endl;
std::ofstream myfile;
myfile.open("example.txt");
myfile << "Writing this to a file: " << status << message << std::endl;
myfile.close();
}
And this is the actual test
void thread_test_start(Core* core)
{
try
{
core->SetCpuAffinity(2);
core->AddCallback(default_callback);
core->AddCallback_C_tmp(default_c_callback_temporary);
//set true to run the async version (implemented in python)
core->Start(false);
}
catch (const std::exception& ex)
{
std::cout << ex.what() << std::endl;
}
}
int main()
{
Core* core = new Core(false);
std::thread t(thread_test_start, core);
py::print(core->GetCallbacks());
std::cout << "\nGet C Callbacks:\n";
py::print(core->GetCallbacks_C_tmp());
std::cout << "\nEverything done. press Enter to Exit";
t.join();
std::getchar();
return 0;
}
The call to core->GetCallbacks() causes the memory access violation:
Exception thrown at 0x000000006FCC6D80 (python36.dll) in TestDLL.exe: 0xC0000005: Access violation reading location 0x0000000000000010.
And here is a snapshot showing the access violation error inside VS2019:
Doing something like this is also the same :
void thread_test_start2()
{
try
{
Core* core = new Core(false);
core->SetCpuAffinity(2);
core->AddCallback(default_callback);
core->AddCallback_C_tmp(default_c_callback_temporary);
std::thread t(&Core::Start, core, false);
py::print(core->GetCallbacks());
std::cout << "\nGet C Callbacks:\n";
py::print(core->GetCallbacks_C_tmp());
t.join();
}
catch (const std::exception& ex)
{
std::cout << ex.what() << std::endl;
}
}
results in :
Exception thrown at 0x000000006FCC0CDF (python36.dll) in TestDLL.exe: 0xC0000005: Access violation writing location 0x0000000000000020.
like the former one.
Why am I getting this error ? Can we not use threading with Pybind11? What am I missing here?
Here is a sample project to re-create this issue : https://workupload.com/file/6LmfRtbztHK
The reason for memory access violations were due to trying to run methods using different threads. That is, all Pybind11 related methods (methods that use Pybind11) need to be executed under the very same thread it seems.
Therefore executing some portion of the code under one thread and trying to execute some other methods in the main thread will result in memory access violation.
In order to get around this, I ended up implementing a simple dispatcher in one callback where any method that needs to be run, first sets a flag, then each time the callback is run, the flag is checked and the corresponding method is run.
int flag=0;
void callback(...)
{
switch(flag)
{
case 1: //e.g. stop
core->stop();
break;
case 2: // e.g. get_callbacks()
core->get_callbacks();
break;
case 3:
//some other op
break;
....
}
//reset flag
flag = 0;
}
I am trying to run some function in asynchronous manner. For this purpose I wrote class called Core where I use std::async to run function in different thread and std::shared_future<int> to wait for this thread and possibly to get future result. This is code of test program:
#include <iostream>
#include <future>
class Core : public std::enable_shared_from_this<Core>
{
public:
Core()
: isRunning_(false) {
};
~Core() {
isRunning_ = false;
if (f_.valid())
{
f_.wait();
std::cout << "Result is: " << f_.get() << std::endl;
}
};
void Start() {
isRunning_ = true;
auto self(shared_from_this());
f_ = std::async(std::launch::async, [self, this]() {
try {
while (true) {
if (!isRunning_)
break;
std::cout << "Boom" << std::endl; // Error occurs here
std::this_thread::sleep_for(std::chrono::seconds(1));
}
}
catch (const std::exception& e) {
std::cerr << "Loop error:" << e.what();
}
return 999;
});
}
private:
std::shared_future<int> f_;
std::atomic<bool> isRunning_;
};
int main()
{
try {
std::shared_ptr<Core> load(new Core);
load->Start();
throw std::runtime_error("Generate error"); // Added in order to generate error
}
catch (const std::exception& e) {
std::cout << "Error occurred: " << e.what();
}
return 0;
}
Each time when I start this program it crashes at this line:
std::cout << "Boom" << std::endl; // Error occurs here
with this error:
That is debugger error and call stack which I managed to get during debugging:
Looks like Core destructor function doesn't call at all. Why is it happens? weird!!!
Could you tell me where is my mistake? Thanks.
When main thread returns from main() it starts tearing down the environment before terminating the whole process. All this while background thread is accessing objects there are being destroyed or have been destroyed already.
I am not sure what you are triying to achieve, but you are doing something wrong:
Your lambda should execute some work and return immediately after it is done e.g. you should never loop forever.
Your main thread should wait for your future to complete by calling std::future<T>::get().
I have my application main loop control where I do start a thread to handle asio work as follows:
void AsioThread::Run()
{
try
{
/*
* Start the working thread to io_service.run()
*/
boost::asio::io_service::work work(ioService);
boost::thread t(boost::bind(&boost::asio::io_service::run, &ioService));
t.detach();
while (true)
{
// Process stuff related to protocol, calling
// connect_async, send_async and receive_async
}
}
catch (std::runtime_error &ex)
{
std::cout << "ERROR IN FTP PROTOCOL: " << ex.what() << std::endl;
}
catch (...)
{
std::cout << "UNKNOWN EXCEPTION." << std::endl;
}
During the async operation, the handlers are called and sometimes I do throw exception on those handlers, like:
void AsioThread::ReceiveDataHandler(const boost::system::error_code& errorCode, std::size_t bytesTransferred)
{
std::cout << "Receive data handler called. " << bytesTransferred << " bytes received." << std::endl;
if (errorCode)
{
std::cout << "Error receiving data from server." << std::endl;
}
rxBufferSize = bytesTransferred;
/*
* Check for response
*/
std::string msg(rxBuffer);
if (msg.substr(0, 3) != "220")
throw std::runtime_error("Error connecting to FTP Server");
}
My problem is that the exception thrown inside the async handler (AsioThread::ReceiveDataHandler) is not catched by the main processing loop try...catch block in AsioThread::Run. Naturally this happens because the working thread t is on another thread, detached, and at runtime that leads to an execution error.
How can I receive exceptions from the detached boost::asio::io_service::work thread ? How can I structure my code to make this logic work ?
Thanks for helping.
You can catch exceptions in the worker thread, save them into a queue variable that is shared by the two threads, and check that queue periodically in the main thread.
To use a queue, you need to first convert your exceptions to a common type. You can use std::exception or string or whatever is the best for your situation. If you absolutely need to keep information of the original exception class, you can use boost::exception_ptr.
Variables you need (these could be members of AsioThread):
boost::mutex queueMutex;
std::queue<exceptionType> exceptionQueue;
Run this function in the worker thread:
void AsioThread::RunIoService(){
try{
ioService.run();
}
catch(const exceptionType& e){
boost::lock_guard<boost::mutex> queueMutex;
exceptionQueue.push(e);
}
catch(...){
boost::lock_guard<boost::mutex> queueMutex;
exceptionQueue.push(exceptionType("unknown exception"));
}
}
Launch the worker thread like this:
boost::thread t(boost::bind(&AsioThread::RunIoService, this));
t.detach();
In the main thread:
while(true){
// Do something
// Handle exceptions from the worker thread
bool hasException = false;
exceptionType newestException;
{
boost::lock_guard<boost::mutex> queueMutex;
if(!exceptionQueue.empty()){
hasException = true;
newestException = exceptionQueue.front();
exceptionQueue.pop();
}
}
if(hasException){
// Do something with the exception
}
}
This blog post implements a thread-safe queue, which you can use to simplify saving exceptions; in that case you would not need a separate mutex because that would be inside the queue class.
Building a SignalR C++ client using Visual Studio 2013, I am starting with the working sample code from NuGet Package Microsoft.AspNet.SignalR.Client.Cpp.v120.WinDesktop, source here
Reviewing the library source it seems to me the event handling processes are based on the Concurrency Runtime (pplx::task) which relies on C++11 features
void chat(const utility::string_t& name)
{
signalr::hub_connection connection{ U("https://testsite") };
auto proxy = connection.create_hub_proxy(U("ChatHub"));
proxy.on(U("broadcastMessage"), [](const web::json::value& m)
{
ucout << std::endl << m.at(0).as_string() << U(" wrote:") << m.at(1).as_string() << std::endl << U("Enter your message: ");
});
connection.start()
.then([proxy, name]()
{
for (;;)
{
utility::string_t message;
std::getline(ucin, message);
if (message == U(":q"))
{
break;
}
send_message(proxy, name, message);
}
})
.then([&connection]() // fine to capture by reference - we are blocking so it is guaranteed to be valid
{
return connection.stop();
})
.then([](pplx::task<void> stop_task)
{
try
{
stop_task.get();
ucout << U("connection stopped successfully") << std::endl;
}
catch (const std::exception &e)
{
ucout << U("exception when starting or stopping connection: ") << e.what() << std::endl;
}
}).get();
}
I want to eliminate the "user input" component; and instead quit loop when a particular "broadcastMessage" has been received.
If I replace the for loop with a sleep statement, the broadcastMessage event stops firing.
If I use the for loop without the getline, set bComplete to true when done, it works the way I want but causes high CPU usage (obviously)
for (;;)
{
if (bComplete) break;
}
Ideally I want connection to start, and then just wait until the broadcastMessage events signals to close the connection.
In addition the "chat" function shouldn't return until connection has closed.
I can see in your answer that you've already discovered Windows event objects; however, if you were looking for a C++11 platform-independent solution, consider std::condition_variable!
unsigned int accountAmount;
std::mutex mx;
std::condition_variable cv;
void depositMoney()
{
// go to the bank etc...
// wait in line...
{
std::unique_lock<std::mutex> lock(mx);
std::cout << "Depositing money" << std::endl;
accountAmount += 5000;
}
// Notify others we're finished
cv.notify_all();
}
void withdrawMoney()
{
std::unique_lock<std::mutex> lock(mx);
// Wait until we know the money is there
cv.wait(lock);
std::cout << "Withdrawing money" << std::endl;
accountAmount -= 2000;
}
int main()
{
accountAmount = 0;
std::thread deposit(&depositMoney);
std::thread withdraw(&withdrawMoney);
deposit.join();
withdraw.join();
std::cout << "All transactions processed. Final amount: " << accountAmount << std::endl;
return 0;
}
In this example we make two threads: one to deposit money into the account and one to withdraw money. Because it's possible for the thread to withdraw the money to run first, especially because there's more processing involved with depositMoney(), we need to wait until we know the money is there. We lock our thread before accessing the money, and then tell the condition_variable what we are waiting for. The condition_variable will unlock the thread, and once the money has been deposited and notify_all() is called we'll be re-awoken to finish processing our logic.
Note that it's possible to do the exact same using the Windows event objects. Instead of std::condition_variable::wait() and std::condition_variable::notify_all() you'd use SetEvent() and WaitForSingleObject(). This is platform-independent though.
I got this working using WinAPI WaitForSingleObject:
HANDLE hEvent;
void chat(const utility::string_t& name)
{
signalr::hub_connection connection{ U("https://testsite") };
auto proxy = connection.create_hub_proxy(U("ChatHub"));
proxy.on(U("broadcastMessage"), [](const web::json::value& m)
{
ucout << std::endl << m.at(0).as_string() << U(" wrote:") << m.at(1).as_string() << std::endl;
if (m.at(1).as_string() == L"quit")
{
SetEvent(hEvent);
}
});
hEvent = CreateEvent(0, TRUE, FALSE, 0);
connection.start()
.then([proxy, name]()
{
WaitForSingleObject(hEvent, INFINITE);
})
.then([&connection]() // fine to capture by reference - we are blocking so it is guaranteed to be valid
{
return connection.stop();
})
.then([](pplx::task<void> stop_task)
{
try
{
stop_task.get();
ucout << U("connection stopped successfully") << std::endl;
}
catch (const std::exception &e)
{
ucout << U("exception when starting or stopping connection: ") << e.what() << std::endl;
}`enter code here`
}).get();
}
I am referring to Chat Client
My write Operation is:
void CSession::beginWrite(const Buffer & message)
{
//Check if the socket is open or not?
bool writeInProgress = !writeQueue_.empty();
writeQueue_.push_back(message);
if (!writeInProgress) //Exception Thrown here
{
asio::async_write(socket_, asio::buffer(writeQueue_.front().received_, writeQueue_.front().buffsize),
std::bind(&CSession::handle_write, this,
std::placeholders::_1, std::placeholders::_2));
}
}
void CSession::handle_write(const asio::error_code& error /*error*/, size_t bytes_transferred /*bytes_transferred*/)
{
//std::cout << "CSession::handle_write() Called" << "(" << __FILE__ << " : " << __LINE__ << ")" << std::endl;
if (!error)
{
//std::cout << bytes_transferred << " bytes written to the socket." << std::endl;
writeQueue_.pop_front();
if (!writeQueue_.empty())
{
asio::async_write(socket_, asio::buffer(writeQueue_.front().received_, writeQueue_.front().buffsize),
std::bind(&CSession::handle_write, this,
std::placeholders::_1, std::placeholders::_2));
}
}
else
{
std::cout << "Write Error Detected" << std::endl;
std::cout << error.message() << std::endl;
state_ = false;
doClose();
return;
}
}
It works fine. Then I tried load testing by making client write message Client 2 to the server continuously for 11 minutes as shown below:
bool flag = false;
void setFlag(const asio::error_code& /*e*/)
{
flag = true;
}
void Client(std::string IP, std::string port)
{
CSession Session(IP, port);
Session.initSession();
asio::thread t(boost::bind(&asio::io_service::run, &(*CIOService::fetchIOService().getIO())));
asio::deadline_timer timer(*CIOService::fetchIOService().getIO(), boost::posix_time::seconds(675));
timer.async_wait(&setFlag);
while (!flag)
{
Session.write("Client 2");
}
Session.close();
t.join();
}
void main()
{
Client("localhost", "8974");
system("Pause");
}
After 2-3 minutes of successful write operation, the code throws exception Unhandled exception at 0x75B7C42D in NetworkComponentsClient.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x026DE87C. at line
if (!writeInProgress) //Exception Thrown here
{
asio::async_write(socket_, asio::buffer(writeQueue_.front().received_, writeQueue_.front().buffsize),
std::bind(&CSession::handle_write, this,
std::placeholders::_1, std::placeholders::_2));
}
Debug shows:
- writeQueue_ { size=16777215 } std::deque<channel::Buffer,std::allocator<channel::Buffer> >
+ [0] {received_=0x052a0ac8 "Client 2" } channel::Buffer
+ [1] {received_=0x052a0b28 "Client 2" } channel::Buffer
+ [2] {received_=0x052a0b88 "Client 2" } channel::Buffer
....
....
I can see size of writeQueue_ { size=16777215 } which is very large and hence std::bad_alloc.
Why such behaviour? I can see the code popping messages from deque as below:
if (!error)
{
writeQueue_.pop_front();
if (!writeQueue_.empty())
{
asio::async_write(socket_, asio::buffer(writeQueue_.front().received_, writeQueue_.front().buffsize),
std::bind(&CSession::handle_write, this,
std::placeholders::_1, std::placeholders::_2));
}
}
So write deque should not have grown so large.
My client is supposed to run for days and should be involved large continuous data write. How do I ensure smooth long write operations?
Your consumer (CSession) is far slower than your producer (Client).
Your producer is doing a denial of service attack by producing messages as fast as it can. This is a good test.
Your consumer should (at least one, ideally all):
detect that the work is accumulating and set up a policy when such things happen, like "ignore new", "drop oldest"
Limit the consumption lag from happening by setting an active filter on incoming messages
Improve the performance of incoming messages handling.
My client is supposed to run for days and should be involved large
continuous data write. How do I ensure smooth long write operations?
Then you need a much better code than an example found online.