Multi source BFS multithreading - c++

Question

I have a graph represented with an adjacency matrix arr. And a vector source for the multiple starting vertices.
My idea is to split the source vector into "equal" pieces depending on the number of threads(if it doesn't split equally I add the remaining on the last piece). And create threads that run this function. bool used[] is a global array
I am trying to get (I think its called) "liner" scaling. I assume the number of starting vertices is at least equal to the number of threads.
If I use a mutex to synchronise the threads it is very inefficient.
And If I don't some vertices get traversed more then once.
Question is there a data structure that would let me remove the mutex?
or another way to implement this algorithm?
mutex m;
void msBFS(bool** arr, int n, vector<int> s, atomic<bool>* used) //s is a different
// piece of the original source
{
queue<int> que;
for(auto i = 0; i < s.size(); ++i)
{
que.push(s[i]);
used[s[i]] = true;
}
while (!que.empty())
{
int curr = que.front();
que.pop();
cout << curr << " ";
for (auto i = 0; i < n; ++i)
{
lock_guard<mutex> guard(m);
if (arr[curr][i] == 1 && !used[i] && curr != i)
{
que.push(i);
used[i] = true;
}
}
}
}```

Answer 1

With the atomic<bool> I think you are almost there. The only piece missing is the atomic exchange operation. It allows you to read-modify-write as an atomic operation. For bool atomic type there is usually a piece of hardware that supports it.
void msBFS(bool** arr, int n, vector<int> s, atomic<bool>* used) //s is a different
// piece of the original source
{
//used[i] initialized to 'false' for all i
queue<int> que;
for(auto i = 0; i < s.size(); ++i)
{
que.push(s[i]);
//we don't change used just yet!
}
while (!que.empty())
{
int curr = que.front();
que.pop();
bool alreadyUsed = used[curr].exchange(true);
if(alreadyUsed) {
continue; //some other thread already processing it
}
cout << curr << " ";
for (auto i = 0; i < n; ++i) {
if (arr[curr][i] == 1 && !used[i] && curr != i) {
que.push(i);
}
}
}
}
Note, there is one logical change: the used[i] is set to true when a thread starts processing the node, and not when it is added to the queue.
At the first attempt to process a node, when used[i] is set to true, alreadyUsed will hold the previous value (false) indicating that noone else started processing the node earlier. At subsequent attempts to process the node, alreadyUsed will be already set to true and the processing will skip.
The above approach is not ideal: it is possible for a node to be added many times to a queue before it is processed. Depending on the shape of your graph it may or may not be a problem.
If this is a problem - I would suggest using a three-value used state: not_visited, queued and processed.
static constexpr int not_visited = 0;
static constexpr int queued = 1;
static constexpr int processed = 2;
Now, every time we try to push onto the que, and every time we try to process the node, we advance the state accordingly. These advancements need to be performed atomically, via compare_exchange_strong so that each change may happen exactly once. The call to compare_exchange_strong returns true if it succeeded (i.e. the previously contained value actually matched the expected one)
void msBFS(bool** arr, int n, vector<int> s, atomic<int>* used) //s is a different
// piece of the original source
{
//used[i] initialized to '0' for all i
queue<int> que;
int empty = 0;
for(auto i = 0; i < s.size(); ++i) {
int expected = not_visited;
//we check it even here, because one thread may be seriously lagging behind others which already entered the while loop
if(used[s[i]].compare_exchange_strong(expected, queued)) {
que.push(s[i]);
}
}
while (!que.empty())
{
int curr = que.front();
que.pop();
int expected = queued;
if(!used[curr].compare_exchange_strong(expected, processed)) {
continue;
}
cout << curr << " ";
for (auto i = 0; i < n; ++i) {
if (arr[curr][i] == 1 && curr != i) {
int expected = not_visited;
if(used[i].compare_exchange_strong(expected, queued)) {
que.push(i);
}
}
}
}
}
Check the performance. There are many atomic operations, but those are generally cheaper than a mutex. Internally, mutex also performs atomic operations similar to these, but in addition it may completely block a thread. The code I have shown never blocks (the thread is never put on a halt), all synchronization is done on the atomic variables only.
Edit: Some possible optimisations for second approach:
I realized, that if the transition not_visited->queued is guaranteed to occur exactly once, the other transition does not even have to be performed, because the node is present exactly once in exactly one queue anyway. So, you may save up on few atomic operations and use bool again. Since it is rare, though, I don't think it will have that much of an impact.
When iterating over the neighbors - there is an if statement: if (arr[curr][i] == 1 && curr != i) -- this does not check if the neighbor was visited or not. It is checked only later, through the atomic exchange. However, you may want to see if checking within that if anyway will help. If you find early that used[i] is already in a queue or processed, you skip over the branch, and over the no-longer-needed atomic compare-and-swap.
If you want to squeeze every tick from your processor, consider using bitfields, instead of bools for your adjacency matrix and used array. I think the iteration over neighbors, and the conditions can be evaluated with bitwise operations, for 32 bits/neighbors at once. There is even std::atomic<uint32_t>::fetch_or to aid you with updating 32 used at once.
Edit2: Possible optimisation for first approach:
Each thread can hold its own localUsed array, which would be checked and set to true when pushing to the queue (similarly how you did in your original code). This would be local for the thread, so no atomics, mutexes etc. With this simple check you have a guarantee that a given node appears in a queue of each thread at most once. So, at most, a node will appear N times, where N is the number of threads.
I think this is a compromise worth considering between scalability and the memory footprint, and may perform better than the second approach.