OpenSSL error "assertion failed" - c++

Question

I have a problem, when i use http://slproweb.com/products/Win32OpenSSL.html , i try to Init Encryption, but always catch error like
"OpenSSL assertion failed, evp_enc.c(282)"
Can somebody help me with this matter?
My code:
bool do_encrypt(const char *in, unsigned char *out, int *outlen, unsigned char *key, unsigned char *iv)
{
int buflen, tmplen;
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
EVP_EncryptInit_ex(&ctx, EVP_rc4(), NULL, key, iv);
if(!EVP_EncryptUpdate(&ctx, out, &buflen, (unsigned char*)in, strlen(in))) // error here
{
return false;
}
if(!EVP_EncryptFinal_ex(&ctx, out + buflen, &tmplen))
{
return false;
}
buflen += tmplen;
*outlen = buflen;
EVP_CIPHER_CTX_cleanup(&ctx);
return true;
}
I use key for test {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}, iv is NULL.
Code that uses this function above:
int WINAPI OwnSend(SOCKET s, const char FAR *buff, int len, int flags )
{
if(s == ServerSocket)
{
if(len > 0)
{
int outlen;
unsigned char EncryptBuffer[5500];
do_encrypt(buff, EncryptBuffer, &outlen, KeyTest, NULL);
buff = (const char *) EncryptBuffer;
return pTrampolineSend(s, buff, outlen, flags);
}
}
return pTrampolineSend(s, buff, len, flags);
}
Thanks!

Answer 1

The only assertion in EVP_EncryptUpdate is one that asserts that the buffer length is less than or equal to the block size of the encryption algorithm.
Instead of calling strlen(in) as the size of your input, try looping EVP_EncryptUpdate and each time you go through, make sure to limit the input size with:
int in_size_limit = EVP_CIPHER_CTX_block_size(&ctx);
While looping, be sure to increment the offset of the 2nd and 4th args by the number of bytes you've already encrypted:
if ( EVP_EncryptUpdate(&ctx,
out+encrypted_bytes,
&bytes_encrypted_this_call,
in+encrypted_bytes,
in_size_limit) != 1)
{
/* error */
}
else
{
encrypted_bytes += bytes_encrypted_this_call;
}