Auto Parallelization with VS - c++

Question

I am trying to understand how the auto-parallelization works to speed up the execution of a program I am writing. I have created a simpler example:
#include <iostream>
#include <vector>
#include <chrono>
using namespace std;
using namespace std::chrono;
class matrix
{
public:
matrix(int size, double value)
{
A.resize(size, vector<double>(size, value));
B.resize(size, vector<double>(size, value));
};
void prodScal(double valore)
{
for (int m = 0; m < A.size(); m++)
for (int n = 0; n < A.size(); n++)
{
B[m][n] = A[m][n] * valore;
};
};
double elemento(int riga, int column) { return B[riga][column]; }
protected:
vector<vector<double>> A, B;
};
void main()
{
matrix* M;
M = new matrix(1000, 174.9);
high_resolution_clock::time_point t1 = high_resolution_clock::now();
#pragma loop(hint_parallel(4))
for (int i = 0; i < 1000; i++)
M->prodScal(567.3);
high_resolution_clock::time_point t2 = high_resolution_clock::now();
auto duration = duration_cast<milliseconds>(t2 - t1).count();
cout << "execution time [ms]: " << duration << endl;
}
When I try to compile this code using cl main.cpp /O2 /Qpar /Qpar-report:2, I get the following message:
c:\users\utente\documents\visual studio 2017\projects\parallel\parallel\main.cpp(39) : info C5012: ciclo non parallelizzato a causa del motivo '500'
c:\users\utente\documents\visual studio 2017\projects\parallel\parallel\main.cpp(39) : info C5012: ciclo non parallelizzato a causa del motivo '500'
c:\users\utente\documents\visual studio 2017\projects\parallel\parallel\main.cpp(38) : info C5012: ciclo non parallelizzato a causa del motivo '1000'
Can you help me with the correct way to parallelize this loop?
Thanks.

Answer 1

Auto-parallelisation efforts ( or Beliefs? ) are a rather dual-edge sword :
A machine can "guess" an intent only to a certain degree ( and can give-up, whenever such intent was not clear to a pre-wired set of transformation strategies ), so rather do not expect any bright tricks on a large scale of different approaches possible. Marketing people will beat all their drums and blow all their whistles to sell auto-"thinking"-PRODUCTs, but the reality is different. Even the best of the bests admit, that best performance comes from instruction-level profiling and sometimes they even avoid superscalar pipelined processor-knit tricks, so as to gain the last few nanoseconds, lost in parallelised-code performance at the very last level of CPU uop instruction flow. So, better never expect such expertise to happen just by using a #pragma code-section in a belief, the "machine"-will-invent a smartest way ahead.
So, test it ( always and thoroughly ):
An attempt to "parallelise" an outermost for(){...} is not the best step to start with. Both performance-wise and resources-wise. Let's tackle the case from a different side, the calculation itself:
#include <iostream> // https://stackoverflow.com/questions/48033769/auto-parallelization-with-vs
#include <vector>
#include <chrono> // g++ FLAGS.ADD: -std=c++11
#include <omp.h> // g++ FLAGS.ADD: -fopenmp -lm
#define OMP_NUM_OF_THREADS 4
using namespace std;
using namespace std::chrono;
class matrix {
public:
matrix( int size, double value ) {
A.resize( size, vector<double>( size, value ) );
B.resize( size, vector<double>( size, value ) );
}
void prodScal( double aScalarVALORE ) {
// #pragma loop( hint_parallel(4) ) // matrix_(hint_parallel(4)).cpp:18:0: warning: ignoring #pragma loop [-Wunknown-pragmas]
#pragma omp parallel num_threads( OMP_NUM_OF_THREADS ) // _____ YET, AGNOSTIC TO ANY BETTER CACHE-LINE RE-USE POLICY
for ( unsigned int m = 0; m < A.size(); m++ )
for ( unsigned int n = 0; n < A.size(); n++ )
B[m][n] = A[m][n] * aScalarVALORE;
}
double elemento( int riga, int column ) { return B[riga][column]; }
protected:
vector<vector<double>> A, B;
};
int main() { // matrix_(hint_parallel(4)).cpp:31:11: error: ‘::main’ must return ‘int’
matrix* M;
M = new matrix( 1000, 174.9 );
high_resolution_clock::time_point t1 = high_resolution_clock::now();
// *******************
// DEFINITELY NOT HERE
// *******************
// #pragma loop(hint_parallel(4)) // JUST A TEST EXECUTION, NOT ANY PARALLELISATION BENEFIT FOR A PROCESS-PER-SE PERFORMANCE
for ( int i = 0; i < 1000; i++ )
M->prodScal( 567.3 );
high_resolution_clock::time_point t2 = high_resolution_clock::now();
auto duration = duration_cast<milliseconds>( t2 - t1 ).count();
cout << "execution time [ms]: " << duration << endl;
/*
* execution time [ms]: 21601
------------------
(program exited with code: 0)
* */
return 0;
}
Once having a working code, the performance tweaking, to gain max, is the next hurdle.
A better stepping through a for(){...} can dramatically improve the sum of costs of all the MEM-fetches ( paying ~ +100 [ns] for each non-cached reference ) v/s CACHE-re-use ( paying just ~ +1.5 [ns] for any cache-re-use ).
It depends on the global size of the matrices, on the L3, L2 and L1 cache-sizes and on the cache-lines lengths / associativity, not to mention an additional performance skew(s) if the code is to be run on a virtual device.
The static sizing and an approximate NUMA-topology could be depicted using lstopo ( lscpu in an absence of the smart hwloc service ).
Here, you can read the cache capacities, that can hold the matrix cells for any potential speedup from a smart re-use ( obeying the cache-line striding of the for(){...} indexes ).
Best performance could be gained from tuning the for()-loop stepping, best near the CPU-hardware available ILP-level ( using another degree of parallelism possible from CPU Instruction-Level-Parallelism, permitted for co-executed micro-instruction chains ( ref. Intel CPU publications on these details ) and best tested on the target platform ( cross-compilation would fail to allow such optimisation without performance benchmarks on the target CPU-architecture, best on the target platform in-vivo ).
Details are way beyond the limited scope of this media format here, on StackOverflow, but definitely if interested in performance tuning, you will find both sources and your own experimentation hands-on experience will govern your further steps. To just somehow sense the powers, 've made a large-matrix linear algebra project to finish a few [TB] matrix processing from ~ 126 hours down to a few minutes ( not counting the loading phase, to get the matrix data into the RAM ), right by the very careful parallel code-design, so indeed worth doing the design "right".
For even higher performance, one will have to also avoid O/S from evicting the expensively pre-fetched data, so even more efforts are needed for ultimate performance, than just to rely on an automated "auto-parallelisation"-toy.
Epilogue:
If still in doubts, if that were indeed possible, why would HPC centers still take care for and nourish the HPC-experts for designing ultimately performant code, if the "auto-parallelisation"-toys would do it better or at least the same as these expert nerdy geeks?? They would not, if they indeed could.