Verify OpenPGP based RSA signature with WinCrypt/CryptoAPI - c++

Question

I have code that parses OpenPGP packets and I have n, e of the public key packet as well as s of the signature packet as byte arrays.
In order to verify a signature I first initialize CryptAcquireContext (I also tried with PROV_RSA_FULL instead of PROV_RSA_AES)
HCRYPTPROV hCryptProv;
CryptAcquireContext(&hCryptProv, nullptr, nullptr, PROV_RSA_AES, CRYPT_VERIFYCONTEXT);
then create a hash
HCRYPTHASH hHash;
CryptCreateHash(hCryptProv, CALG_SHA1, 0, 0, &hHash); // as the digest algorithm of the signature was 2 => SHA1
and populate it using CryptHashData. This works so far as well as parsing and importing the public key using CryptImportKey.
typedef struct _RSAKEY
{
BLOBHEADER blobheader;
RSAPUBKEY rsapubkey;
BYTE n[4096 / 8];
} RSAKEY;
static int verify_signature_rsa(HCRYPTPROV hCryptProv, HCRYPTHASH hHash, public_key_t &p_pkey, signature_packet_t &p_sig)
{
int i_n_len = mpi_len(p_pkey.key.sig.rsa.n); // = 512; p_pkey.key.sig.rsa.n is of type uint8_t n[2 + 4096 / 8];
int i_s_len = mpi_len(p_sig.algo_specific.rsa.s); // = 256; p_sig.algo_specific.rsa.s is of type uint8_t s[2 + 4096 / 8]
HCRYPTKEY hPubKey;
RSAKEY rsakey;
rsakey.blobheader.bType = PUBLICKEYBLOB; // 0x06
rsakey.blobheader.bVersion = CUR_BLOB_VERSION; // 0x02
rsakey.blobheader.reserved = 0;
rsakey.blobheader.aiKeyAlg = CALG_RSA_KEYX;
rsakey.rsapubkey.magic = 0x31415352;// ASCII for RSA1
rsakey.rsapubkey.bitlen = i_n_len * 8; // = 4096
rsakey.rsapubkey.pubexp = 65537;
memcpy(rsakey.n, p_pkey.key.sig.rsa.n + 2, i_n_len); // skip first two byte which are MPI length
std::reverse(rsakey.n, rsakey.n + i_n_len); // need to convert to little endian for WinCrypt
CryptImportKey(hCryptProv, (BYTE*)&rsakey, sizeof(BLOBHEADER) + sizeof(RSAPUBKEY) + i_n_len, 0, 0, &hPubKey); // no error
std::unique_ptr<BYTE[]> pSig(new BYTE[i_s_len]);
memcpy(pSig.get(), p_sig.algo_specific.rsa.s + 2, i_s_len); // skip first two byte which are MPI length
std::reverse(p_sig.algo_specific.rsa.s, p_sig.algo_specific.rsa.s + i_s_len); // need to convert to little endian for WinCrypt
if (!CryptVerifySignature(hHash, pSig.get(), i_s_len, hPubKey, nullptr, 0))
{
DWORD err = GetLastError(); // err=2148073478 -> INVALID_SIGNATURE
CryptDestroyKey(hPubKey);
return -1;
}
CryptDestroyKey(hPubKey);
return 0;
}
CryptVerifySignature fails with GetLastError() decoding to INVALID_SIGNATURE.
On https://www.rfc-editor.org/rfc/rfc4880#section-5.2.2 I read
With RSA signatures, the hash value is encoded using PKCS#1 encoding
type EMSA-PKCS1-v1_5 as described in Section 9.2 of RFC 3447. This
requires inserting the hash value as an octet string into an ASN.1
structure.
Is that needed or is that automatically done by CryptVerifySignature? If not, how to do that?

Answer 1

The PKCS#1 padding is not likely to be the problem. The hint that it uses an OID for the hash algorithm by default is pointing to PKCS#1 v1.5 type of signatures, so I think you can rest assured that the right padding is used.
More confirmation can be found in the CryptSignHash documentation:
By default, the Microsoft RSA providers use the PKCS #1 padding method for the signature. The hash OID in the DigestInfo element of the signature is automatically set to the algorithm OID associated with the hash object. Using the CRYPT_NOHASHOID flag will cause this OID to be omitted from the signature.
Looking through the API documentation, the following caught my eye:
The native cryptography API uses little-endian byte order while the .NET Framework API uses big-endian byte order. If you are verifying a signature generated by using a .NET Framework API, you must swap the order of signature bytes before calling the CryptVerifySignature function to verify the signature.
This does mean that the API is not PKCS#1 v1.5 compliant as the byte order is explicitly specified therein. This is therefore certainly something to be aware of and could be part of a solution.

Answer 2

The error was in this line
std::reverse(p_sig.algo_specific.rsa.s, p_sig.algo_specific.rsa.s + i_s_len); // need to convert to little endian for WinCrypt
which should read
std::reverse(pSig.get(), pSig.get() + i_s_len); // need to convert to little endian for WinCrypt
because converting the source of the bytes from big to little endian does not convert another buffer after a copy.