Mixing user threads and OpenCV built-in multithreading - c++

Question

I develop a C++ application that needs to process different images at the same time. The processing algorithm is built on top of OpenCV and uses parallelism functionalities.
The application works in the following way: for each image it has, it spawns a thread to execute the processing algorithm. Unfortunately it seems that this scheme does not work well with OpenCV internal multithreading.
Minimal example:
#include <iostream>
#include <thread>
#include <chrono>
#include <opencv2/core.hpp>
void run(int thread_id, cv::Mat& mat)
{
auto start = std::chrono::steady_clock::now();
// multithreaded operation on mat
mat.forEach<float>([](float& pixel, int const* position) {
std::this_thread::sleep_for(std::chrono::milliseconds(1));
});
auto end = std::chrono::steady_clock::now();
std::cout << "thread " << thread_id << " took "
<< (end - start).count() * 1e-9 << " sec"
<< std::endl;
}
int main()
{
cv::Mat mat1(100, 100, CV_32F), mat2(100, 100, CV_32F);
std::thread t1(run, 1, std::ref(mat1));
std::thread t2(run, 2, std::ref(mat2));
t1.join();
t2.join();
return 0;
}
Output on my machine:
thread 1 took 1.42477 sec
thread 2 took 12.1963 sec
It seems that the second operation is not taking advantage of multithreading. Looking at my CPU usage, I have the feeling that OpenCV assigns all its internal threads to the first operation and, when the second one arrives, there is no internal thread left. Thus, the second operation is executed sequentially in the application thread body.
Firstly, I would appreciate if someone that already faced similar issues with OpenCV can confirm that my hypothesis is correct.
Secondly, is there a way to dispatch internal OpenCV resources more intelligently ? For example, by assigning half of the threads to the first operation and half to the second one ?
Multithreading objective
After writing my question, I realize that the purpose of doing multithreading at the application level might be unclear. Some people may argue that it suffices to run the two operations sequentially at the application level to take full advantage of internal OpenCV multithreading. This is true for the minimal example I posted here, but typically not all parts of processing algorithms can be run in parallel.
The idea behind multithreading at application level is to try to run a maximum of 'unparallelisable' operations at the same time:
Operations 1 and 2 sequentially:
[-----seq 1----][-par 2 (full power)-][-----seq 2----][-par 2 (full power)-]
Operations 1 and 2 in parallel:
[-----seq 1----][------------par 2 (half power)------------]
[-----seq 2----][------------par 2 (half power)------------]
seq X = sequential task of operation X
par X = parallelisable task of operation X
We can see that application level multithreading reduce the total computation time, because sequential parts of different operations are run concurrently.

Answer 1

I think your approach to multi threading is correct. I ran the code you provided and here's my output:
thread 1 took 2.30654 sec
thread 2 took 2.63872 sec
Maybe you should check the number of available threads for your program?
std::cout << std::thread::hardware_concurrency() << std::endl;