OpenSSL signature length is different for a different message - c++

Question

I have been struggling with a weird problem with RSA_verify. I am trying to RSA_sign using C and RSA_verify using C++. I have generated the private key and certificate using OpenSSL commands.
message = "1.2.0:08:00:27:2c:88:77"
When I use the message above, generate a hash and use RSA_sign to sign the digest, I get a signature of length 256 (strlen(signature)) and also the length returned from RSA_sign is 256. I use this length to verify and verification succeeds.
But when I use a message = "1.2.0:08:00:27:2c:88:08", the signature length is 60 and RSA_sign returns 256. When I use this length 60 to verify it fails. It fails to verify with length 256 as well. Also for some messages (1.2.0:08:00:27:2c:88:12) the signature generated is zero.
I am using SHA256 to hash the message and NID_SHA256 to RSA_sign and RSA_verify this digest. I have used -sha256 while generating the keys using the OpenSSL command.
I am forming the message by parsing an XML file reading some of the tags using some string operation.
Kindly suggest.
Below is the code used to sign.
int main(void)
{
int ret;
RSA *prikey;
char *data ;
unsigned char* signature;
int slen = 0;
FILE * fp_priv = NULL;
char* privfilepath = "priv.pem";
unsigned char* sign = NULL;
ERR_load_crypto_strings();
data = generate_hash();
printf("Message after generate hash %s: %d\n", data, strlen(data));
fp_priv = fopen(privfilepath, "r");
if (fp_priv == NULL)
{
printf("Private key path not found..");
return 1;
}
prikey = RSA_new();
prikey = PEM_read_RSAPrivateKey(fp_priv, &prikey, NULL, NULL);
if (prikey == NULL)
{
printf("Private key returned is NULL\n");
return 1;
}
signature = (unsigned char*)malloc(RSA_size(prikey));
if( signature == NULL )
return 1;
if(RSA_sign(NID_sha256, (unsigned char*)data, strlen(data),
signature, &slen, prikey) != 1) {
ERR_print_errors_fp(stdout);
return 1;
}
printf("Signature length while signing... %d : %d : %d ",
strlen(signature), slen, strlen(data));
FILE * sig_bin = fopen("sig_bin", "w");
fprintf(sig_bin, "%s", signature);
fclose(sig_bin);
system("xxd -p -c256 sig_bin sig_hex");
RSA_free(prikey);
if(signature)
free(signature);
return 0;
}

Answer 1

One very, very important thing to learn about C is it has two distinct types with the same name.
char*: This represents the beginning of a character string. You can do things like strstr or strlen.
You should never strstr or strlen, but rather strnstr and strnlen, but that's a different problem.
char*: This represents the beginning of a data blob (aka byte array, aka octet string), you can't meaningfully apply strlen/etc to it.
RSA_sign uses the latter. It returns "data", not "a message". So, in your snippet
printf("Signature length while signing... %d : %d : %d ",
strlen(signature), slen, strlen(data));
FILE * sig_bin = fopen("sig_bin", "w");
fprintf(sig_bin, "%s", signature);
fclose(sig_bin);
data came from a function called generate_hash(); it's probably non-textual, so strlen doesn't apply. signature definitely is data, so strlen doesn't apply. fprintf also doesn't apply, for the same reasons. These functions identify the end of the character string by the first occurrence of a zero-byte (0x00, '\0', etc). But 0x00 is perfectly legal to have in a signature, or a hash, or lots of "data".
The length of the output of RSA_sign is written into the address passed into the 5th parameter. You passed &slen (address-of slen), so once the function exits (successfully) slen is the length of the signature. Note that it will only very rarely match strlen(signature).
To write your signature as binary, you should use fwrite, such as fwrite(sig_bin, sizeof(char), signature, slen);. If you want it as text, you should Base-64 encode your data.