Windows Named Semaphore Not Getting Locked - c++

I am developing C++ class with calls to Windows API C libraries.
I am using the Semaphores for a task, let's say I have two processes:
ProcessA has two semaphores:
Global\processA_receiving_semaphore
Global\processA_waiting_semaphore
ProcessB has two semaphores:
Global\processB_receiving_semaphore
Global\processB_waiting_semaphore
I have two threads in each process:
Sending thread in processA:
Wait on "Global\processB_waiting_semaphore"
// do something
Signal "Global\processB_receiving_semaphore"
Receiving thread on processB:
Wait on "Global\processB_receiving_semaphore"
// do something
Signal "Global\processB_waiting_semaphore
I removed ALL code that Releases "Global\processB_waiting_semaphore" but it can still be acquired. Calling WaitForSingleObject on that semaphore always returns successful wait and immediately. I tried setting the timeout period to 0 and it still acquires the semaphore while NOTHING is releasing it.
The receiving semaphore has initial count = 0 and max count = 1 while the waiting semaphore has initial count = 1 and max count = 1.
Calling WaitForSingleObject on the receiving semaphore works great and blocks until it is released by the other process. The problem is with the waiting semaphore and I cannot figure out why. The code is very big and I have made sure the names of the semaphores are set correctly.
Is this a common issue? If you need more explanation please comment and I will modify the post.
EDIT: CODE ADDED:
Receiver semaphores:
bool intr_process_comm::create_rcvr_semaphores()
{
std::cout << "\n Creating semaphore: " << "Global\\" << this_name << "_rcvr_sem";
rcvr_sem = CreateSemaphore(NULL, 0, 1, ("Global\\" + this_name + "_rcvr_sem").c_str());
std::cout << "\n Creating semaphore: " << "Global\\" << this_name << "_wait_sem";
wait_sem = CreateSemaphore(NULL, 1, 1, ("Global\\" + this_name + "_wait_sem").c_str());
return (rcvr_sem && wait_sem);
}
Sender semaphores:
// this sender connects to the wait semaphore in the target process
sndr_sem = OpenSemaphore(SEMAPHORE_MODIFY_STATE, FALSE, ("Global\\" + target_name + "_wait_sem").c_str());
// this target connects to the receiver semaphore in the target process
trgt_sem = OpenSemaphore(SEMAPHORE_MODIFY_STATE, FALSE, ("Global\\" + target_name + "_rcvr_sem").c_str());
DWORD intr_process_locking::wait(unsigned long period)
{
return WaitForSingleObject(sndr_sem, period);
}
void intr_process_locking::signal()
{
ReleaseSemaphore(trgt_sem, 1, 0);
}
Receiving thread function:
void intr_process_comm::rcvr_thread_proc()
{
while (conn_state == intr_process_comm::opened) {
try {
// wait on rcvr_semaphore for an infinite time
WaitForSingleObject(rcvr_sem, INFINITE);
if (inner_release) // if the semaphore was released within this process
return;
// once signaled by another process, get the message
std::string msg_str((LPCSTR)hmf_mapview);
// signal one of the waiters that want to put messages
// in this process's memory area
//
// this doesn't change ANYTHING in execution, commented or not..
//ReleaseSemaphore(wait_sem, 1, 0);
// put this message in this process's queue
Msg msg = Msg::from_xml(msg_str);
if (msg.command == "connection")
process_connection_message(msg);
in_messages.enQ(msg);
//std::cout << "\n Message: \n"<< msg << "\n";
}
catch (std::exception e) {
std::cout << "\n Ran into trouble getting the message. Details: " << e.what();
}
}
}
Sending thread function:
void intr_process_comm::sndr_thread_proc()
{
while (conn_state == intr_process_comm::opened ||
(conn_state == intr_process_comm::closing && out_messages.size() > 0)
) {
// pull a message out of the queue
Msg msg = out_messages.deQ();
if (connections.find(msg.destination) == connections.end())
connections[msg.destination].connect(msg.destination);
if (connections[msg.destination].connect(msg.destination)
!= intr_process_locking::state::opened) {
blocked_messages[msg.destination].push_back(msg);
continue;
}
// THIS ALWAYS GETS GETS WAIT_OBJECT_0 RESULT
DWORD wait_result = connections[msg.destination].wait(wait_timeout);
if (wait_result == WAIT_TIMEOUT) { // <---- THIS IS NEVER TRUE
out_messages.enQ(msg);
continue;
}
// do things here
// release the receiver semaphore in the other process
connections[msg.destination].signal();
}
}
To clarify some things:
trgt_sem in a sender is the rcvr_sem in the receiver.
`sndr_sem' in the sender is the 'wait_sem" in the receiver.

for call WaitForSingleObject with some handle:
The handle must have the SYNCHRONIZE access right.
but you open semaphore with SEMAPHORE_MODIFY_STATE access only. with this access possible call ReleaseSemaphore (This handle must have the SEMAPHORE_MODIFY_STATE access right) but call to WaitForSingleObject fail with result WAIT_FAILED. call to GetLastError() after this must return ERROR_ACCESS_DENIED.
so if we want call both ReleaseSemaphore and any wait function - we need have SEMAPHORE_MODIFY_STATE | SYNCHRONIZE access on handle. so need open with code
OpenSemaphore(SEMAPHORE_MODIFY_STATE|SYNCHRONIZE, )
and of course always checking api return values and error codes can save a lot of time

If you set the timeout to 0 WaitForSingleObject will always return immediately, a successful WaitForSingleObject will return WAIT_OBJECT_0 (which happens to have the value 0), WFSO is not like most APIs where success is indicated by a non-zero return.

Related

why GRPC AsyncClient throws Segfault when waiting for the Next result in the completion queue

I am using version 1.23.1 of the GRPC library.
I have an asynchronous RPC c++ Client class, which initiates each RPC with the following method:
void Client::SendTaskAsync(const Task& task) {
unique_lock<mutex> lock(mtx_);
cout << "Sending task with id " << task.id() << endl;
ClientContext context;
Status status;
unique_ptr<ClientAsyncResponseReader<Result>> rpc(
stub_->PrepareAsyncSendTask(&context, task, &queue_));
rpc->StartCall();
// Allocating memory to store result from RPC
Result* result = &results_.emplace_back();
int* tag = new int(results_.size() - 1);
rpc->Finish(result, &status, static_cast<void*>(tag));
}
In the main thread I call SendTaskAsync five times in a loop.
The Client class has a background thread informing when each RPC has returned a Result:
while (true) {
void* tag;
bool ok = false;
{
unique_lock<mutex> lock(mtx_);
cout << "Waiting the for next result" << endl;
const time_point<system_clock> deadline =
system_clock::now() + milliseconds(1000);
// SEGFAULT HERE, WHY?
GPR_ASSERT(queue_.AsyncNext(&tag, &ok, deadline));
}
if (ok) {
int index = *static_cast<int*>(tag);
cout << "Got result with tag " << index << endl;
} else {
cout << "Sleeping" << endl;
sleep_for(milliseconds(1000));
}
}
If I start my client, the following log is observed:
BACKGROUND: Waiting for the next result
MAIN THREAD: Sending task with id 0
BACKGROUND: Sleeping
MAIN THREAD: Sending task with id 1
MAIN THREAD: Sending task with id 2
MAIN THREAD: Sending task with id 3
MAIN THREAD: Sending task with id 4
BACKGROUND: Waiting for the next result
BACKGROUND: Segmentation fault (core dumped)
What happens is that
Background thread checks if a queue_ contains a result, there is none yet, so it goes to sleep;
Main thread makes 5 RPC that at the end should populate the queue_ with results;
Background thread wakes up and checks if a queue_ contains a result, AND CRASHES.
Any ideas why?
The code in the question is written according to this tutorial, which sends only one request and waits for a reply in the same thread.
If you want to use multiple threads, follow the client example here.

Strange behaviour of GetQueuedCompletionStatus when used from thread pool worker threads

I've been testing to combine the IO Completion Ports with the worker threads from the Thread Pool and stumbled on a behaviour I can't explain. In particular, while the following code:
int data;
for (int i = 0; i < NUM; ++i)
PostQueuedCompletionStatus(cp, 1, NULL, reinterpret_cast<LPOVERLAPPED>(&data));
{
std::thread t([&] ()
{
LPOVERLAPPED aux;
DWORD cmd;
ULONG_PTR key;
for (int i = 0; i < NUM; ++i)
{
if (!GetQueuedCompletionStatus(cp, &cmd, &key, &aux, 0))
break;
++count;
}
});
t.join();
}
works perfectly fine and receives NUM status notifications (with NUM being large number, 100000 or more), the similar code that uses the thread pool work object that reads one status notification per work item and repost the work item after reading it, fails after reading couple of hundred status notifications. Having the following global variables (please don't mind the names):
HANDLE cport;
PTP_POOL pool;
TP_CALLBACK_ENVIRON env;
PTP_WORK work;
std::size_t num_calls;
std::mutex mutex;
std::condition_variable cv;
bool job_done;
and the callback function:
static VOID CALLBACK callback(PTP_CALLBACK_INSTANCE instance_, PVOID pv_, PTP_WORK work_)
{
LPOVERLAPPED aux;
DWORD cmd;
ULONG_PTR key;
if (GetQueuedCompletionStatus(cport, &cmd, &key, &aux, 0))
{
++num_calls;
SubmitThreadpoolWork(work);
}
else
{
std::unique_lock<std::mutex> l(mutex);
std::cout << "No work after " << num_calls << " calls.\n";
job_done = true;
cv.notify_one();
}
}
the following code:
{
job_done = false;
std::unique_lock<std::mutex> l(mutex);
num_calls = 0;
cport = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 1);
pool = CreateThreadpool(nullptr);
InitializeThreadpoolEnvironment(&env);
SetThreadpoolCallbackPool(&env, pool);
work = CreateThreadpoolWork(callback, nullptr, &env);
for (int i = 0; i < NUM; ++i)
PostQueuedCompletionStatus(cport, 1, NULL, reinterpret_cast<LPOVERLAPPED>(&data));
SubmitThreadpoolWork(work);
cv.wait_for(l, std::chrono::milliseconds(10000), [] { return job_done; } );
}
would report "No more work after ..." after 250 or so calls to GetQueuedCompletionStatus although the NUM was set to 1000000. Even more curious is that setting the wait from 0 to, way, 10 milliseconds would increase the number of successful calls to couple of hundred thousand and would occasionally read all 1000000 notifications. Which I don't really understand since all status notifications were posted before submitting the work object for the first time.
Is it possible that there really is a problem with combining completion ports and a thread pool or is there something wrong in my code? Please don't go into why would I want to do this - I was investigating the possibilities and stumbled on this. In my view it should work and can't figure put what's wrong. Thank you.
I've tried running this code, the issue seems to be the NumberOfConcurrentThreads parameters supplied to CreateIoCompletionPort. Passing 1 means that the first pool thread that executes callback becomes associated with io completion port but since thread pool may execute callback using different thread GetQueuedCompletionStatus will fail when this happens. From documentation:
The most important property of an I/O completion port to consider carefully is the concurrency value. The concurrency value of a completion port is specified when it is created with CreateIoCompletionPort via the NumberOfConcurrentThreads parameter. This value limits the number of runnable threads associated with the completion port. When the total number of runnable threads associated with the completion port reaches the concurrency value, the system blocks the execution of any subsequent threads associated with that completion port until the number of runnable threads drops below the concurrency value.
Although any number of threads can call GetQueuedCompletionStatus for a specified I/O completion port, when a specified thread calls GetQueuedCompletionStatus the first time, it becomes associated with the specified I/O completion port until one of three things occurs: The thread exits, specifies a different I/O completion port, or closes the I/O completion port. In other words, a single thread can be associated with, at most, one I/O completion port.
So to use io completion with thread pool you need to set the number of concurrent threads to the size of the thread pool (that you can set using SetThreadpoolThreadMaximum).
::DWORD const threads_count{1};
cport = ::CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, threads_count);
...
pool = ::CreateThreadpool(nullptr);
::SetThreadpoolThreadMaximum(pool, threads_count);

Boost scoped lock assert fails

I'm using Boost 1.41 in a linux app that receives data on one thread and sticks it in a queue, another thread pops it off the queue and processes it. To make it thread safe I'm using scoped locks.
My problem is that very infrequently the lock function fails in the read function with the message:
void boost::mutex::lock() Assertion '!pthread_mutext_lock(&m)' failed
It is very infrequent, on last run, it took 36 hours (~425M transactions) before it failed. The read and write functions are listed below, its always in the read function that the Assert arises
Write to queue
void PacketForwarder::Enqueue(const byte_string& newPacket, long sequenceId)
{
try
{
boost::mutex::scoped_lock theScopedLock(pktQueueLock);
queueItem itm(newPacket,sequenceId);
packetQueue.push(itm);
if (IsConnecting() && packetQueue.size() > MaximumQueueSize)
{
// Reached maximum queue size while client unavailable; popping.
packetQueue.pop();
}
}
catch(...)
{
std::cout << name << " Exception was caught:" << std::endl;
}
}
Read from queue
while ( shouldRun )
{
try
{
if (clientSetsHaveChanged)
{
tryConnect();
}
size_t size = packetQueue.size();
if (size > 0)
{
byte_string packet;
boost::mutex::scoped_lock theQLock(pktQueueLock);
queueItem itm = packetQueue.front();
packet = itm.data;
packetQueue.pop();
BytesSent += packet.size();
trySend(packet);
}
else
{
boost::this_thread::sleep(boost::posix_time::milliseconds(50));
}
}
catch (...)
{
cout << name << " Other exception in send packet" << endl;
}
I've googled and found a few problems when destroying scoped_locks but nothing on failing to get a lock. I have also had a search through boost release notes and Trac logs to see if this has been identified as an issue by anyone else. I thought my code was about as simple as it gets but obviously something is up. Any thoughts?
TIA
Paul
There is one thread-safety issue in your program, in this piece of code:
size_t size = packetQueue.size();
if (size > 0)
{
byte_string packet;
boost::mutex::scoped_lock theQLock(pktQueueLock);
queueItem itm = packetQueue.front();
packet = itm.data;
packetQueue.pop();
// ...
}
The issue here is that between the time you checked the queue size and the time you got the lock some other reader thread might take the last item out of the queue, which will cause front() and pop() to fail. Unless you have only one reader thread, you need the size check to be under the lock as well.
I do not know if this is the reason of the assertion failure though. The assertion means the call to pthread_mutex_lock returned a non-zero value signaling an error. Unfortunately, Boost does not show which exactly of possible pthread_mutex_lock errors has happened.

RegisterWaitForSingleObjectEx() & several SetEvent()

I find a tricky behavior of multiple SetEvents with RegisterWaitForSingleObjectEx().
#include <windows.h>
#include <iostream>
using namespace System;
using namespace System::Drawing;
using namespace System::Threading;
VOID CALLBACK Callback(PVOID lpParameter, BOOLEAN TimerOrWaitFired)
{
String^ string = gcnew String("");
Monitor::Enter(string->GetType());
//wait for 2 seconds
for(int i=1; i<=2;i++) {
Sleep(1000);
cout << i << " seconds \n";
}
Monitor::Exit(string->GetType());
}
void main()
{
HANDLE eventhandle = CreateEvent(
NULL, // default security attributes
FALSE, // manual-reset event
FALSE, // initial state is nonsignaled
TEXT("WriteEvent") // object name
);
//register the callback for the event
RegisterWaitForSingleObjectEx(eventhandle, Callback, nullptr, -1, WT_EXECUTELONGFUNCTION);
BOOL bEvented[3];
bEvented[0] = SetEvent(eventhandle);
//Sleep(10);
bEvented[1] = SetEvent(eventhandle);
//Sleep(10);
bEvented[2] = SetEvent(eventhandle);
cout << "event0 = " << bEvented[0] << ", event1 = " << bEvented[1] << ", event2 = " << bEvented[2] << " \n";
}
I set the Event 3 times. So, I expect the callback to be called 3 times (please correct me if I am wrong).
But I get only 2 callbacks.
If I uncomment the lines //Sleep(10); , I get 3 callbacks.
What is happening here?
I am using Win7 64bit
UPDATE:
Can you please give an example about how to achieve this using semaphore?
Actual scenario:
I have a third-party library where I have to register a HANDLE to get notified about the occurrence of an event. Most of the times, I am able to get the notification (signalling on the HANDLE). Sometimes, I am not getting the correct "number of signalling", as expected.
I am passing the HANDLE created using CreateEvent() and registered a callback for the HANDLE using RegisterWaitForSingleObjectEx().
I suspect that this race condition is the reason for the behavior.
How to overcome this?
SetEvent on an event that's already signalled is a no-op. You have a race condition between the main thread that calls SetEvent, and the worker thread that waits on it (and resets it automatically when the wait is satisfied).
Most likely, you manage to call SetEvent twice while the worker is still running the first callback.

C++ How to exit out of a while loop recvfrom()

I'm trying to create a UDP broadcast program to check for local game servers, but I'm having some trouble with the receiving end. Since the amount of servers alive is unknown at all times, you must have a loop that only exits when you stop it. So in this bit of code here:
while(1) // start a while loop
{
if(recvfrom(sd,buff,BUFFSZ,0,(struct sockaddr *)&peer,&psz) < 0) // recvfrom() function call
{
cout << red << "Fatal: Failed to receive data" << white << endl;
return;
}
else
{
cout << green << "Found Server :: " << white;
cout << yellow << inet_ntoa(peer.sin_addr), htons(peer.sin_port);
cout << endl;
}
}
I wish to run this recvfrom() function until I press Ctrl + C. I've tried setting up handlers and such (from related questions), but they're all either too complicated for me, or it's a simple function that just exits the program as a demonstration. Here's my problem:
The program hangs on recvfrom until it receives a connection (my guess), so, there's never a chance for it to specifically wait for input. How can I set up an event that will work into this nicely?
Thanks!
In the CTRL-C handler, set a flag, and use that flag as condition in the while loop.
Oh, and if you're not on a POSIX systems where system-calls can be interrupted by signals, you might want to make the socket non-blocking and use e.g. select (with a small timeout) to poll for data.
Windows have a couple of problems with a scheme like this. The major problem is that functions calls can not be interrupted by the CTRL-C handler. Instead you have to poll if there is anything to receive in the loop, while also checking the "exit loop" flag.
It could be done something like this:
bool ExitRecvLoop = false;
BOOL CtrlHandler(DWORD type)
{
if (type == CTRL_C_EVENT)
{
ExitRecvLoop = true;
return TRUE;
}
return FALSE; // Call next handler
}
// ...
SetConsoleCtrlHandler((PHANDLER_ROUTINE) CtrlHandler, TRUE);
while (!ExitRecvLoop)
{
fd_set rs;
FD_ZERO(&rs);
FD_SET(sd, &rs);
timeval timeout = { 0, 1000 }; // One millisecond
if (select(sd + 1, &rs, NULL, NULL, &timeout) < 0)
{
// Handle error
}
else
{
if (FD_ISSET(sd, &rs))
{
// Data to receive, call `recvfrom`
}
}
}
You might have to make the socket non-blocking for this to work (see the ioctlsocket function for how to).
Thread off your recvFrom() loop so that your main thread can wait for user input. When user requests stop, close the fd from the main thread and the recvFrom() will return immediately with an error, so allowing your recvFrom() thread to exit.