NTE_BAD_DATA in CryptSetKeyParam while setting KP_P in wincrypt - c++

Question

I am having the below code. I am setting a prime for diffie-hellman algorithm using char *.
I am getting bad data after i set the prime. Where am i doing wrong?
I followed the same example in this link.
https://msdn.microsoft.com/en-us/library/aa381969(VS.85).aspx#exchanging_diffie-hellman_keys
What is the correct way to set prime in diffie-hellman using wincrypt?
#define DHKEYSIZE 1024
int fld_sz = 256;
BYTE* g_rgbPrime = new BYTE[DHKEYSIZE/8];
char * prime = "A1BD60EBD2D43C53FA78D938C1EF8C9AD231F9862FC402739302DEF1B6BEB01E5BE59848A04C48B0069A8FB56143688678F7CC1097B921EA3E13E1EF9B9EB5381BEFDE7BBF614C13827493A1CA31DA76B4083B62C5073451D6B1F06A2F1049C291464AC68CBB2F69474470BBAD374073392696B6447C82BF55F20B2D015EB97B";
string s_prime(prime, fld_sz);
vector<std::string> res;
// split the string two charactes for converting into hex format
for (size_t i = 0; i < fld_sz; i += 2)
res.push_back(s_prime.substr(i, 2));
for(int i = 0; i < res.size(); i++) {
BYTE b = static_cast<BYTE>(std::stoi(res[i], 0, 16));
g_rgbPrime[i] = b;
}
BYTE g_rgbGenerator[128] =
{
0x02
};
BOOL fReturn;
HCRYPTPROV hProvParty1 = NULL;
HCRYPTPROV hProvParty2 = NULL;
CRYPT_DATA_BLOB P;
CRYPT_DATA_BLOB G;
HCRYPTKEY hPrivateKey1 = NULL;
HCRYPTKEY hPrivateKey2 = NULL;
PBYTE pbKeyBlob1 = NULL;
PBYTE pbKeyBlob2 = NULL;
HCRYPTKEY hSessionKey1 = NULL;
HCRYPTKEY hSessionKey2 = NULL;
PBYTE pbData = NULL;
/************************
Construct data BLOBs for the prime and generator. The P and G
values, represented by the g_rgbPrime and g_rgbGenerator arrays
respectively, are shared values that have been agreed to by both
parties.
************************/
P.cbData = DHKEYSIZE / 8;
P.pbData = (BYTE*)(g_rgbPrime);
G.cbData = DHKEYSIZE / 8;
G.pbData = (BYTE*)(g_rgbGenerator);
/************************
Create the private Diffie-Hellman key for party 1.
************************/
// Acquire a provider handle for party 1.
fReturn = CryptAcquireContext(
&hProvParty1,
NULL,
MS_ENH_DSS_DH_PROV,
PROV_DSS_DH,
CRYPT_VERIFYCONTEXT);
if(!fReturn)
{
goto ErrorExit;
}
// Create an ephemeral private key for party 1.
fReturn = CryptGenKey(
hProvParty1,
CALG_DH_EPHEM,
DHKEYSIZE << 16 | CRYPT_EXPORTABLE | CRYPT_PREGEN,
&hPrivateKey1);
if(!fReturn)
{
goto ErrorExit;
}
// Set the prime for party 1's private key.
fReturn = CryptSetKeyParam(
hPrivateKey1,
KP_P,
(PBYTE)&P,
0);
if(!fReturn)
{
std::cout << GetLastError() << endl;
goto ErrorExit;
}
// Set the generator for party 1's private key.
fReturn = CryptSetKeyParam(
hPrivateKey1,
KP_G,
(PBYTE)&G,
0);
if(!fReturn)
{
std::cout << GetLastError() << endl;
goto ErrorExit;
}
Thanks in advance.
Update 1:
Thanks to #RbMm I was able to set the prime. The problem was with DHKEYSize. However i am getting an error in while setting KP_X. updated the code above to reflect the new code.
Here i converted the string to hex bytes array.

Answer 1

size of prime KP_P (and KP_G) and DH key size hard connected. must be cbKey == 8*cbP. look for example Diffie-Hellman Client Code for Creating the Master Key:
as key size if used cbP * 8 where cbP size of prime P. in your link also P.cbData = DHKEYSIZE/8;
also in code instead hard-code size of P (and G) you can get it in runtime:
ULONG dwDataLen;
CryptGetKeyParam(hPrivateKey1, KP_P, 0, &(dwDataLen = 0), 0);
CryptGetKeyParam(hPrivateKey1, KP_G, 0, &(dwDataLen = 0), 0);
and you can sure that dwDataLen == DHKEYSIZE / 8 where DHKEYSIZE is key size.
because you use 512 as key size, the length of data for P and G must be 512/8=64. but you use 256 (for P) and 1 (for G). as result and error.