Unclear pointer defect in program - c++

Question

Im trying to build modified example of usage of DES CBC EDE on openssl. Im using gcc version 4.4.6 (Buildroot 2012.02) compiling for arm (not my choise of compiler) on ubuntu 10.04. Here is a problem: when pointer declaration is uncommented - everything is normal. Test message deciphers. But when pointer declaration gets commented - test message shows only 2 first letters after deciphering. I just can not understand what could do a declaration of an unused pointer. Here is code:
#include <openssl/des.h>
#include <cstring>
#define BUFSIZE 512
using namespace std;
int main(int argc, char *argv[]) {
unsigned char in[BUFSIZE] = {};
unsigned char out[BUFSIZE] = {};
unsigned char back[BUFSIZE] = {};
unsigned char *strangePointer = &out[0]; // what is wrong with it?
int len;
DES_cblock key1, key2, key3;
DES_cblock ivsetup = {0xE1, 0xE2, 0xE3, 0xD4, 0xD5, 0xC6, 0xC7, 0xA8};
DES_cblock ivecLocal;
DES_key_schedule ks1, ks2, ks3;
const char* key = "0A0A0B0B0C0C0A0A0B0B0C0C";
memcpy(&key1,key,8);
memcpy(&key2,key + 8,8);
memcpy(&key2,key + 16,8);
DES_set_odd_parity(&key1);
DES_set_odd_parity(&key2);
DES_set_odd_parity(&key3);
DES_set_key((C_Block *)key1, &ks1);
DES_set_key((C_Block *)key2, &ks2);
DES_set_key((C_Block *)key3, &ks3);
const char* message = "Now is the time for all men to stand up and be counted";
/* 64 bytes of plaintext */
len = strlen(message);
memcpy(in,message,len);
printf("Plaintext: [%s]\n", in);
memcpy(ivecLocal, ivsetup, sizeof(ivsetup));
DES_ede3_cbc_encrypt(in, out, len, &ks1, &ks2, &ks3, &ivecLocal, DES_ENCRYPT);
int lenout = 0;
while(out[lenout] != '\0') ++lenout;
memcpy(ivecLocal, ivsetup, sizeof(ivsetup));
DES_ede3_cbc_encrypt(out, back, lenout, &ks1, &ks2, &ks3, &ivecLocal, DES_DECRYPT);
printf("Decrypted Text: [%s]\n", back);
return 0;
}

Answer 1

You have several things incorrect in your code. I'll start with the simple answer first:
Your third key isn't even populated with key data. It is random data on the local activation stack.
memcpy(&key1,key,8);
memcpy(&key2,key + 8,8);
memcpy(&key2,key + 16,8); // <<=== NOTE still key2
Oh, copy-paste, ye are a cruel and heartless wench. Anyway, by removing the variable you're commenting out, this key is moving further up (or down, depending on your implementation) the stack in memory, and the result is a different value . But the bottom line, you're using indeterminate data for your third key.
But that isn't the entire problem. That change in key exposes another issue, you're output length calculation, which is also flat-wrong. This:
int lenout = 0;
while(out[lenout] != '\0') ++lenout;
assumes the length of the output cipher can be found by searching for a 0-byte. The DES algorithm can easily emit such a byte anywhere in an encryption block. This is plain wrong. The size of the output of a DES_ede3_cbc_encrypt encryption operation is always a multiple of the block size, which for DES is 8 bytes (not coincidentally the size of a DES_cblock), The correct calculation of your output buffer size is:
int lenout = ((len + sizeof(DES_cblock) - 1)/sizeof(DES_cblock))*sizeof(DES_cblock);
which may seem like a helluva lot, but in the end all this does is round up to the nearest multiple of the block length. Further, you don't even need the input buffer. The API will do the calculation for you if you simply pass your input data message and its true length. But I leave that for you to finish (hint: just remove input and pass message).
the result is:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <openssl/des.h>
#define BUFSIZE 512
int main(int argc, char *argv[])
{
unsigned char in[BUFSIZE] = {};
unsigned char out[BUFSIZE] = {};
unsigned char back[BUFSIZE] = {};
int len;
DES_cblock key1, key2, key3;
DES_cblock ivsetup = {0xE1, 0xE2, 0xE3, 0xD4, 0xD5, 0xC6, 0xC7, 0xA8};
DES_cblock ivecLocal;
DES_key_schedule ks1, ks2, ks3;
const char* key = "0A0A0B0B0C0C0A0A0B0B0C0C";
memcpy(&key1,key,8);
memcpy(&key2,key + 8,8);
memcpy(&key3,key + 16,8);
DES_set_odd_parity(&key1);
DES_set_odd_parity(&key2);
DES_set_odd_parity(&key3);
DES_set_key(&key1, &ks1);
DES_set_key(&key2, &ks2);
DES_set_key(&key3, &ks3);
const char* message = "Now is the time for all men to stand up and be counted";
len = strlen(message);
memcpy(in,message,len);
printf("Plaintext: [%s]\n", in);
memcpy(ivecLocal, ivsetup, sizeof(ivsetup));
DES_ede3_cbc_encrypt(in, out, len, &ks1, &ks2, &ks3, &ivecLocal, DES_ENCRYPT);
int lenout = ((len + sizeof(DES_cblock) - 1)/sizeof(DES_cblock))*sizeof(DES_cblock);
memcpy(ivecLocal, ivsetup, sizeof(ivsetup));
DES_ede3_cbc_encrypt(out, back, lenout, &ks1, &ks2, &ks3, &ivecLocal, DES_DECRYPT);
printf("Decrypted Text: [%s]\n", back);
return 0;
}
Output
Plaintext: [Now is the time for all men to stand up and be counted]
Decrypted Text: [Now is the time for all men to stand up and be counted]

Answer 2

You are most probably experiencing a buffer overflow. The declaration of len comes after back on the stack. If back overflows, then len will be overwritten. Adding another variable (strangePointer) in between adds a gap on the stack that stops len getting overwritten.
Probably increasing BUFSIZE should solve your issue.