OpenSSL SHA256 generates different sum for the same file - c++

Question

I have never worked with SHA256 before. Recently, I've been trying to implement a SHA256 checksum in order to see if a library has been tampered with.
The funny thing is that OpenSSL SHA256 generates a different sum for the exact same library depending on its location. If it's located in another folder, the sum is different.
Is there anything I could do to get the same sum no matter where the file is located? I provided the code snippets and the sums I get.
unsigned char* getsum( char* filename ) {
std::ifstream pFile( filename, std::ios::binary );
SHA256_CTX sContext;
char pBuffer[ 1024*16 ];
unsigned char pSum[SHA256_DIGEST_LENGTH];
SHA256_Init( &sContext );
while( pFile.good() ) {
pFile.read( pBuffer, sizeof(pBuffer) );
SHA256_Update( &sContext, pBuffer, pFile.gcount() );
}
SHA256_Final( pSum, &sContext );
return pSum;
}
...
char* cl_sum = new char[256];
sprintf( cl_sum, "%02x", getsum("library.dll") );
MessageBoxA( NULL, cl_sum , NULL, NULL );
delete[] cl_sum;
exit( -1 );
I also tried using the SHA256() function instead of the whole SHA256 context, SHA256_Init(), Update & Final thing, but still the same result.

Answer 1

Your code has lots of basic mistakes:
sprintf( cl_sum, "%02x", getsum("library.dll") );
Format string says "print me int value", getsum returns char *! This is wrong and if you use compiler warnings it should explain issue.
Return value is wrong too:
unsigned char* getsum( char* filename ) {
...
unsigned char pSum[SHA256_DIGEST_LENGTH];
...
return pSum;
}
You are returning a pointer to local object which lifetime ends when function returns.
Here is my tweak to your code
using Sha256Hash = std::array<unsigned char, SHA256_DIGEST_LENGTH>;
Sha256Hash calcSha256Hash(std::istream& in)
{
SHA256_CTX sContext;
SHA256_Init(&sContext);
std::array<char, 1024> data;
while (in.good()) {
in.read(data.data(), data.size());
SHA256_Update(&sContext, data.data(), in.gcount());
}
Sha256Hash hash;
SHA256_Final(hash.data(), &sContext);
return hash;
}
https://godbolt.org/z/Y3nY13jao
Side note: I'm surprised that std::istream::read sets failure bit when end of file is reached and some data has bee read. This is a inconsistent with behavior of streaming operator.

Answer 2

You are returning a pointer to local variable pSum, that is undefined behaviour(UB).
An explanation for differen outputs might be that since the array is located on stack, calls to future functions like sprintf or MessageBoxA likely overwrite the array with their own variables. But anything can happen after UB.
Use and return std::vector<unsigned char> instead.
Also do not use new, instead std::array or another std::vector is much safer.
Lastly, I would strongly advise to turn on compiler warnings for your compiler, it should warn about the above issue.

Answer 3

Your code's output depends on what value getsum returns. The value getsum returns is an address on the stack. So your code's output depends on where the stack is located.
Somewhere in your code, you should save the result of the SHA256 operation in a buffer of some kind and you should output the contents of that buffer. Your code does neither of those two things.
If you think you save the output of the SHA256 operation in some kind of buffer, where do you think that buffer is allocated. The only buffer you use is pBuffer, and that's allocated on the stack in getsum and so doesn't exist when getsum returns.
If you think you actually look in the buffer somewhere, where do you think that code is? Your sprintf call never looks in the buffer, it just outputs a pointer in hex.

Answer 4

There are several problems with your code:
your function is returning a pointer to a local variable. When your function exits, the variable is destroyed, leaving the returned pointer dangling.
you are printing the value of the returned pointer itself (ie, the memory address it holds), not the SHA data that it points at. That is why you are seeing the same value being displayed in the message box. You are seeing the memory address of the local variable, and multiple invocations of your function may reuse the same memory.
your reading loop is not making sure that read() is successful before calling SHA256_Update().
Try something more like this instead:
using Sha256Digest = std::array<unsigned char, SHA256_DIGEST_LENGTH>;
Sha256Digest getsum( const char* filename ) {
std::ifstream file( filename, std::ios::binary );
SHA256_CTX sContext;
char buffer[ 1024*16 ];
Sha256Digest sum;
SHA256_Init( &sContext );
while( file.read( buffer, sizeof(buffer) ) ) {
SHA256_Update( &sContext, buffer, pFile.gcount() );
}
SHA256_Final( sum.data(), &sContext );
return sum;
}
...
Sha256Digest sum = getsum("library.dll");
char cl_sum[256] = {}, *ptr = cl_sum;
for (auto ch : sum) {
ptr += sprintf( ptr, "%02x", ch );
}
MessageBoxA( NULL, cl_sum, NULL, NULL );
/* Alternatively:
std::ostringstream cl_sum;
for (auto ch : sum) {
cl_sum << std::hex << std::setw(2) << std::setfill('0') << ch;
}
MessageBoxA( NULL, cl_sum.str().c_str(), NULL, NULL );
*/