performance difference of pthreads - c++

Question

I am programming performance-sensitive code. I implement a simple scheduler to distribute workloads and master thread takes charge of the scheduler.
cpu_set_t cpus;
pthread_attr_t attr;
pthread_attr_init(&attr);
for(int i_group =0; i_group<n_groups; i_group++){
std::cout << i_t<< "\t"<<i_group << "th group of cpu" <<std::endl;
for(int i =index ; i < index+group_size[i_group]; i++){
struct timeval start, end;
double spent_time;
gettimeofday(&start, NULL);
arguments[i].i_t=i_t;
arguments[i].F_x=F_xs[i_t];
arguments[i].F_y=F_ys[i_t];
arguments[i].F_z=F_zs[i_t];
CPU_ZERO(&cpus);
CPU_SET(arguments[i].thread_id, &cpus);
int err= pthread_attr_setaffinity_np(&attr, sizeof(cpu_set_t), &cpus);
if(err!=0){
std::cout << err <<std::endl;
exit(-1);
}
arguments[i].i_t=i_t;
pthread_create( &threads[i], &attr, &cpu_work, &arguments[i]);
gettimeofday(&end, NULL);
spent_time = ((end.tv_sec - start.tv_sec) * 1000000u + end.tv_usec - start.tv_usec) / 1.e6;
std::cout <<"create: " << spent_time << "s " << std::endl;
}
i_t++;
cpu_count++;
arr_finish[i_group]=false;
}
}
like above the master thread create. For the simple explanation, i will assume i_group=1. The child threads divide and conquer a bunch of matrix-matrix multiplications. Here rank means thread_id.
int local_first = size[2]*( rank -1 )/n_compute_thread ;
int local_end = size[2] * rank/n_compute_thread-1;
//mkl_set_num_threads_local(10);
gettimeofday(&start, NULL);
for(int i_z=local_first; i_z<=local_end; i_z++ ){
cblas_dgemm( CblasColMajor, CblasNoTrans, CblasNoTrans,
size[0], size[1], size[0], 1.0, F_x, size[0],
rho[i_z], size[1], 0.0, T_gamma[i_z], size[1] );
}
for(int i_z=local_first; i_z<=local_end; i_z++ ){
cblas_dgemm( CblasColMajor, CblasNoTrans, CblasNoTrans,
size[0], size[1], size[1], 1.0, T_gamma[i_z], size[0],
F_y, size[1], 0.0, T_gamma2[i_z], size[0] );
}
gettimeofday(&end, NULL);
std::cout <<i_t <<"\t"<< arg->thread_id <<"\t"<< sched_getcpu()<< "\t" << "compute: " <<spent_time << "s" <<std::endl;
Even though workload fairly distributed, the performance of each thread vary too much. see the result below
5 65 4 4 compute: 0.270229s
5 64 1 1 compute: 0.284958s
5 65 2 2 compute: 0.741197s
5 65 3 3 compute: 0.76302s
second column shows how many matrix-matrix multiplications are done in a particular thread. last column shows consumed time.
When I saw this result firstly, I thought that it related to the affinity of threads. Thus, I added several lines to control the binding of threads. However, it did not change the trends of last column.
My computer has 20 physical cores and 20 virtual core. I made only 4 child threads to test. Of course, it was tested in a Linux machine.
Why does the performance of the threads vary so much? and How to fix it?

Answer 1

First, are you actually creating a scheduler? Your code sample suggests that you are using Linux scheduler and setting thread attributes object, and thread affinity parameters, etc. This difference is relevant in choosing how to approach the problem.
In any case, the question is big, and there are several additional questions/topics that could be raised to help clarify the conditions, and move closer to a real answer. To start, here are some things to consider:
1 - Length of benchmark test. Sub-second evaluation of thread performance within a pool of threads seems inadequate. Stretch the evaluation time to allow the scheduler time to settle. Perhaps several minutes.
( For an example of typical duration times used in an existing benchmarking utility, read this )
2 - Thread priorities. Your threads are not the only ones. Is it possible the kernel scheduler may be periodically moving the benchmark around as threads belonging to other processes (other than the ones you have created) that have a higher priority? (and therefore are displacing yours, resulting in skewing the task completion time)
3 - Size of task. Is the number of operations required to complete each task small enough to fit within the time-slice allocated by the scheduler? This could possibly contribute to the perception of thread to thread performance issues, especially if there are any differences in the number of operations between each task. ( Processes that exceed the allotted
CPU time slice are automatically moved down to a lower “tier,” while processes that make I/O requests or block will be moved to higher “tiers.” )
4 - Equality of tasks - You mention divide and conquer a bunch of matrix-matrix multiplications. But are the matrices identical in size and similar in content. i.e., are you certain the count of operations in each task is equivalent the the count of operations in all the other tasks? The time slice allocated by the scheduler to each equally prioritized thread will ensure that over time a task having an operation count larger than that can be completed in a single time-slice will be more susceptible to longer completion times ( context switching due to higher priority of other OS processes) than those having few enough operations to fit comfortable within one time slice.
5 - Other processes. I have mentioned this in other items above, but it deserves its own number. In order to use multiple cores, multiple threads are required at the same time. But the inverse is not true. A single core is not limited to a single thread. The OS can at any time preemptively interrupt one of your processes (threads) on a particular core with a higher priority process, (while not interrupting any other core) possibly skewing your time measurement. Again, a longer benchmarking time will serve to reduce the impact of thread to thread differences caused by this particular phenomena.