Related
I have a method reads a json file and returns a const char* that can be any text, including emojis. I don't have access to the source of this method.
For example, I created a json file with the england flag, 🏴 ({message: "\uD83C\uDFF4\uDB40\uDC67\uDB40\uDC62\uDB40\uDC65\uDB40\uDC6E\uDB40\uDC67\uDB40\uDC7F"}).
When I call that method, it returns something like í ¼í¿´í€í±§í€í±¢í€í±¥í€í±®í€í±§í€í±¿, but in order to use it properly, I need to convert it to icu::UnicodeString because I use another method (closed source again) that expects it.
The only way I found to make it work was something like:
icu::UnicodeString unicode;
unicode.setTo((UChar*)convertMessage().data());
std::string messageAsString;
unicode.toUTF8String(messageAsString);
after doing that, messageAsString is usable and everything works.
convertMessage() is a method that uses std::wstring_convert<std::codecvt_utf8_utf16<char16_t>, char16_t>::from_bytes(str).
My question is, is there a way to create a icu::UnicodeString without using that extra convertMessage() call?
This is sample usage of ucnv_toUChars function. I took these function from postgresql source code and used it for my project.
UConverter *icu_converter;
static int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, int32_t nbytes)
{
UErrorCode status;
int32_t len_uchar;
status = U_ZERO_ERROR;
len_uchar = ucnv_toUChars(icu_converter, NULL, 0,buff, nbytes, &status);
if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR)
return -1;
*buff_uchar = (UChar *) malloc((len_uchar + 1) * sizeof(**buff_uchar));
status = U_ZERO_ERROR;
len_uchar = ucnv_toUChars(icu_converter, *buff_uchar, len_uchar + 1,buff, nbytes, &status);
if (U_FAILURE(status))
assert(0); //(errmsg("ucnv_toUChars failed: %s", u_errorName(status))));
return len_uchar;
}
static int32_t icu_from_uchar(char **result, const UChar *buff_uchar, int32_t len_uchar)
{
UErrorCode status;
int32_t len_result;
status = U_ZERO_ERROR;
len_result = ucnv_fromUChars(icu_converter, NULL, 0,
buff_uchar, len_uchar, &status);
if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR)
assert(0); // (errmsg("ucnv_fromUChars failed: %s", u_errorName(status))));
*result = (char *) malloc(len_result + 1);
status = U_ZERO_ERROR;
len_result = ucnv_fromUChars(icu_converter, *result, len_result + 1,
buff_uchar, len_uchar, &status);
if (U_FAILURE(status))
assert(0); // (errmsg("ucnv_fromUChars failed: %s", u_errorName(status))));
return len_result;
}
void main() {
const char *utf8String = "Hello";
int len = 5;
UErrorCode status = U_ZERO_ERROR;
icu_converter = ucnv_open("utf8", &status);
assert(status <= U_ZERO_ERROR);
UChar *buff_uchar;
int32_t len_uchar = icu_to_uchar(&buff_uchar, ut8String, len);
// use buff_uchar
free(buff_uchar);
}
Background
I am trying to verify signature of a given binary file using openssl. Actual signing of binary hash is done by a 3rd party. Both 3rd party and I have the same exact certificate - they sent me the certificate.
I have verified health of my certificate by running openssl x509 -noout -text -inform DER -in CERT_PATH. This displays contents of cert correctly.
Following is my code so far - I based it on openssl wiki example here:
static std::vector<char> ReadAllBytes(char const* filename){
std::ifstream ifs(filename, std::ios::binary|std::ios::ate);
std::ifstream::pos_type pos = ifs.tellg();
std::vector<char> result(pos);
ifs.seekg(0, std::ios::beg);
ifs.read(result.data(), pos);
return result;
}
int main(int ac, const char * av[]) {
OpenSSL_add_all_algorithms();
ERR_load_crypto_strings();
// most of error check omitted for brevity
auto foundBinBytes = ReadAllBytes("BINARY_PATH");
auto foundSgnBytes = ReadAllBytes("SIGNATURE_PATH");
auto foundCertBytes = ReadAllBytes("CERT_PATH");
ERR_clear_error();
BIO *b = NULL;
X509 *c;
b = BIO_new_mem_buf(reinterpret_cast<const unsigned char *>(foundCertBytes.data()), foundCertBytes.size());
c = d2i_X509_bio(b, NULL);
EVP_MD_CTX* ctx = NULL;
ctx = EVP_MD_CTX_create();
const EVP_MD* md = EVP_get_digestbyname("SHA256");
int rc = EVP_DigestInit_ex(ctx, md, NULL);
EVP_PKEY *k = NULL;
k = X509_get_pubkey(c);
rc = EVP_DigestVerifyInit(ctx, NULL, md, NULL, k);
rc = EVP_DigestVerifyUpdate(ctx, reinterpret_cast<const unsigned char *>(foundBinBytes.data()), foundBinBytes.size());
ERR_clear_error();
rc = EVP_DigestVerifyFinal(ctx, reinterpret_cast<const unsigned char *>(foundSgnBytes.data()), foundSgnBytes.size());
ERR_print_errors_fp( stdout );
// openssl free functions omitted
if(ctx) {
EVP_MD_CTX_destroy(ctx);
ctx = NULL;
}
return 0;
}
Issue
Running this code produces following errors:
4511950444:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:/.../crypto/asn1/asn1_lib.c:152:
4511950444:error:0D068066:asn1 encoding routines:ASN1_CHECK_TLEN:bad object header:/.../crypto/asn1/tasn_dec.c:1152:
4511950444:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:/.../crypto/asn1/tasn_dec.c:314:Type=X509_SIG
Question
What is wrong with my setup/code? Did I miss something along the way?
You never check the errors when reading the files. You might have errors there (does the file "CERT_PATH" exist? Do you have read permissions? ...).
If "CERT_PATH" cannot be read, then foundCertBytes.data() is an empty byte array, and this explains the subsequent errors.
If you get these errors in a mosquitto secured server's log, check the config file.
My mosquitto.conf was containing :
require_certificate true
my meross device doesn't send certificate. Turning this to "false" solved my problem after restart
Is it possible to detect encoding errors with ICU at conversion time, or is it necessary to pre or post check the conversion?
Given the initialization where a conversion from UTF8 to UTF32 is setup:
#include <stdio.h>
#include "unicode/ucnv.h" /* C Converter API */
static void eval(UConverter* from, UConverter* to);
int main(int argc, char** argv)
{
UConverter* from;
UConverter* to;
UErrorCode status;
/* Initialize converter from UTF8 to Unicode ___________________________*/
status = U_ZERO_ERROR;
from = ucnv_open("UTF-8", &status);
if( ! from || ! U_SUCCESS(status) ) return 1;
status = U_ZERO_ERROR;
to = ucnv_open("UTF32", &status);
if( ! to || ! U_SUCCESS(status) ) return 1;
/*______________________________________________________________________*/
eval(from, to);
return 0;
}
Then, applying the conversion using ucnv_convertEx via
static void eval(UConverter* from, UConverter* to)
{
UErrorCode status = U_ZERO_ERROR;
uint32_t drain[1024];
uint32_t* drain_p = &drain[0];
uint32_t* p = &drain[0];
/* UTF8 sequence with error in third byte ______________________________*/
const char source[] = { "\xED\x8A\x0A\x0A" };
const char* source_p = &source[0];
ucnv_convertEx(to, from, (char**)&drain_p, (char*)&drain[1024],
&source_p, &source[5],
NULL, NULL, NULL, NULL, /* reset = */TRUE, /* flush = */TRUE,
&status);
/* Print conversion result _____________________________________________*/
printf("source_p: source + %i;\n", (int)(source_p - &source[0]));
printf("status: %s;\n", u_errorName(status));
printf("drain: (n=%i)[", (int)(drain_p - &drain[0]));
for(p=&drain[0]; p != drain_p ; ++p) { printf("%06X ", (int)*p); }
printf("]\n");
}
where source contains an inadmissible UTF8 code unit sequence, the function should somehow report an error. Storing the above fragments in "test.c" and compiling the above code with
$ gcc test.c $(icu-config --ldflags) -o test
The output of ./test is (surprisingly):
source_p: source + 5;
status: U_ZERO_ERROR;
drain: (n=5)[00FEFF 00FFFD 00000A 00000A 000000 ]
So, no obvious sign of a detected error. Can error detection be done more elegantly than manually checking the content?
As #Eljay suggests in the comments, you can use an error callback. You don't even need to write your own, since the built-in UCNV_TO_U_CALLBACK_STOP will do what you want (ie, return a failure for any bad characters).
int TestIt()
{
UConverter* utf8conv{};
UConverter* utf32conv{};
UErrorCode status{ U_ZERO_ERROR };
utf8conv = ucnv_open("UTF8", &status);
if (!U_SUCCESS(status))
{
return 1;
}
utf32conv = ucnv_open("UTF32", &status);
if (!U_SUCCESS(status))
{
return 2;
}
const char source[] = { "\xED\x8A\x0A\x0A" };
uint32_t target[10]{ 0 };
ucnv_setToUCallBack(utf8conv, UCNV_TO_U_CALLBACK_STOP, nullptr,
nullptr, nullptr, &status);
if (!U_SUCCESS(status))
{
return 3;
}
auto sourcePtr = source;
auto sourceEnd = source + ARRAYSIZE(source);
auto targetPtr = target;
auto targetEnd = reinterpret_cast<const char*>(target + ARRAYSIZE(target));
ucnv_convertEx(utf32conv, utf8conv, reinterpret_cast<char**>(&targetPtr),
targetEnd, &sourcePtr, sourceEnd, nullptr, nullptr, nullptr, nullptr,
TRUE, TRUE, &status);
if (!U_SUCCESS(status))
{
return 4;
}
printf("Converted '%s' to '", source);
for (auto start = target; start != targetPtr; start++)
{
printf("\\x%x", *start);
}
printf("'\r\n");
return 0;
}
This should return 4 for invalid Unicode codepoints, and print out the UTF-32 values if it was successful. It's unlikely we'd get an error from ucnv_setToUCallBack, but we check just in case. In the example above, we pass nullptr for the previous action since we don't care what it was and don't need to reset it.
I am working on a task to encrypt large files with AES CCM mode (256-bit key length). Other parameters for encryption are:
tag size: 8 bytes
iv size: 12 bytes
Since we already use OpenSSL 1.0.1c I wanted to use it for this task as well.
The size of the files is not known in advance and they can be very large. That's why I wanted to read them by blocks and encrypt each blocks individually with EVP_EncryptUpdate up to the file size.
Unfortunately the encryption works for me only if the whole file is encrypted at once. I get errors from EVP_EncryptUpdate or strange crashes if I attempt to call it multiple times. I tested the encryption on Windows 7 and Ubuntu Linux with gcc 4.7.2.
I was not able to find and information on OpenSSL site that encrypting the data block by block is not possible (or possible).
Additional references:
http://www.fredriks.se/?p=23
http://incog-izick.blogspot.in/2011/08/using-openssl-aes-gcm.html
Please see the code below that demonstrates what I attempted to achieve. Unfortunately it is failing where indicated in the for loop.
#include <QByteArray>
#include <openssl/evp.h>
// Key in HEX representation
static const char keyHex[] = "d896d105b05aaec8305d5442166d5232e672f8d5c6dfef6f5bf67f056c4cf420";
static const char ivHex[] = "71d90ebb12037f90062d4fdb";
// Test patterns
static const char orig1[] = "Very secret message.";
const int c_tagBytes = 8;
const int c_keyBytes = 256 / 8;
const int c_ivBytes = 12;
bool Encrypt()
{
EVP_CIPHER_CTX *ctx;
ctx = EVP_CIPHER_CTX_new();
EVP_CIPHER_CTX_init(ctx);
QByteArray keyArr = QByteArray::fromHex(keyHex);
QByteArray ivArr = QByteArray::fromHex(ivHex);
auto key = reinterpret_cast<const unsigned char*>(keyArr.constData());
auto iv = reinterpret_cast<const unsigned char*>(ivArr.constData());
// Initialize the context with the alg only
bool success = EVP_EncryptInit(ctx, EVP_aes_256_ccm(), nullptr, nullptr);
if (!success) {
printf("EVP_EncryptInit failed.\n");
return success;
}
success = EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_CCM_SET_IVLEN, c_ivBytes, nullptr);
if (!success) {
printf("EVP_CIPHER_CTX_ctrl(EVP_CTRL_CCM_SET_IVLEN) failed.\n");
return success;
}
success = EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_CCM_SET_TAG, c_tagBytes, nullptr);
if (!success) {
printf("EVP_CIPHER_CTX_ctrl(EVP_CTRL_CCM_SET_TAG) failed.\n");
return success;
}
success = EVP_EncryptInit(ctx, nullptr, key, iv);
if (!success) {
printf("EVP_EncryptInit failed.\n");
return success;
}
const int bsize = 16;
const int loops = 5;
const int finsize = sizeof(orig1)-1; // Don't encrypt '\0'
// Tell the alg we will encrypt size bytes
// http://www.fredriks.se/?p=23
int outl = 0;
success = EVP_EncryptUpdate(ctx, nullptr, &outl, nullptr, loops*bsize + finsize);
if (!success) {
printf("EVP_EncryptUpdate for size failed.\n");
return success;
}
printf("Set input size. outl: %d\n", outl);
// Additional authentication data (AAD) is not used, but 0 must still be
// passed to the function call:
// http://incog-izick.blogspot.in/2011/08/using-openssl-aes-gcm.html
static const unsigned char aadDummy[] = "dummyaad";
success = EVP_EncryptUpdate(ctx, nullptr, &outl, aadDummy, 0);
if (!success) {
printf("EVP_EncryptUpdate for AAD failed.\n");
return success;
}
printf("Set dummy AAD. outl: %d\n", outl);
const unsigned char *in = reinterpret_cast<const unsigned char*>(orig1);
unsigned char out[1000];
int len;
// Simulate multiple input data blocks (for example reading from file)
for (int i = 0; i < loops; ++i) {
// ** This function fails ***
if (!EVP_EncryptUpdate(ctx, out+outl, &len, in, bsize)) {
printf("DHAesDevice: EVP_EncryptUpdate failed.\n");
return false;
}
outl += len;
}
if (!EVP_EncryptUpdate(ctx, out+outl, &len, in, finsize)) {
printf("DHAesDevice: EVP_EncryptUpdate failed.\n");
return false;
}
outl += len;
int finlen;
// Finish with encryption
if (!EVP_EncryptFinal(ctx, out + outl, &finlen)) {
printf("DHAesDevice: EVP_EncryptFinal failed.\n");
return false;
}
outl += finlen;
// Append the tag to the end of the encrypted output
if (!EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_CCM_GET_TAG, c_tagBytes, out + outl)) {
printf("DHAesDevice: EVP_CIPHER_CTX_ctrl failed.\n");
return false;
};
outl += c_tagBytes;
out[outl] = '\0';
EVP_CIPHER_CTX_cleanup(ctx);
EVP_CIPHER_CTX_free(ctx);
QByteArray enc(reinterpret_cast<const char*>(out));
printf("Plain text size: %d\n", loops*bsize + finsize);
printf("Encrypted data size: %d\n", outl);
printf("Encrypted data: %s\n", enc.toBase64().data());
return true;
}
EDIT (Wrong Solution)
The feedback that I received made me think in a different direction and I discovered that EVP_EncryptUpdate for size must be called for each block that it being encrypted, not for the total size of the file. I moved it just before the block is encrypted: like this:
for (int i = 0; i < loops; ++i) {
int buflen;
(void)EVP_EncryptUpdate(m_ctx, nullptr, &buflen, nullptr, bsize);
// Resize the output buffer to buflen here
// ...
// Encrypt into target buffer
(void)EVP_EncryptUpdate(m_ctx, out, &len, in, buflen);
outl += len;
}
AES CCM encryption block by block works this way, but not correctly, because each block is treated as independent message.
EDIT 2
OpenSSL's implementation works properly only if the complete message is encrypted at once.
http://marc.info/?t=136256200100001&r=1&w=1
I decided to use Crypto++ instead.
For AEAD-CCM mode you cannot encrypt data after associated data was feed to the context.
Encrypt all the data, and only after it pass the associated data.
I found some mis-conceptions here
first of all
EVP_EncryptUpdate(ctx, nullptr, &outl
calling this way is to know how much output buffer is needed so you can allocate buffer and second time give the second argument as valid big enough buffer to hold the data.
You are also passing wrong (over written by previous call) values when you actually add the encrypted output.
I am trying to write an application which amongst other things uses the openssl blowfish implementation (blowfish.h) to transport files over a simple server/client pair.
However, whilst some files are encrypted, transported, received and decrypted correctly, some end up being corrupted, after the final decryption stage. This leads me to think that the encryption routines are not being called correctly (since I have also tried with equivalent DES library calls, with the same 'intermittent corruption' results).
The relevant code is pasted below.
Basically, it starts with the function send_file (called by a connected client). This splits the file into chunks. Each 1024 byte chunk is encrypted separately and then sent. Each chunk is then received by the server in the receive_file function, decrypted and saved to disc.
Any idea what the problem could be? (Note if necessary, I will add the code for the whole application).
Cheers,
Ben.
void encryptHelper(const char*,int);
void decryptHelper(const char*,int);
inline void blowfish(unsigned char *data, int data_len, char* key, int enc)
{
// hash the key first!
unsigned char obuf[20];
bzero(obuf,20);
SHA1((const unsigned char*)key, strlen(key), obuf);
BF_KEY bfkey;
int keySize = strlen(key);
BF_set_key(&bfkey, 16, (const unsigned char*)obuf);
unsigned char ivec[8];
memset(ivec, 0, 8);
unsigned char out[1024];// = (unsigned char*) malloc(1024);
bzero(out,1024);
int num = 0;
BF_cfb64_encrypt(data, out, data_len, &bfkey, ivec, &num, enc);
data=out;
//memcpy(data, out, data_len);
//free(out);
}
void MyFrame::encryptHelper(char* orig, int inlength)
{
char *pb=(char*)(std::string((passInput->GetValue()).mb_str()).c_str());
blowfish((unsigned char*)orig, inlength, pb, DES_ENCRYPT);
}
void MyFrame::decryptHelper(char* orig, int inlength)
{
char *pb=(char*)(std::string((passInput->GetValue()).mb_str()).c_str());
blowfish((unsigned char*)orig, inlength, pb, DES_DECRYPT);
}
int MyFrame::send_file(int fd)
{
char rec[10];
struct stat stat_buf;
fstat (fd, &stat_buf);
int size=stat_buf.st_size;
int remSize=size;
int value=0;
while(size > 0)
{
char buffer[1030];
bzero(buffer,1030);
bzero(rec,10);
int n;
if(size>=1024)
{
value+=1024;
n=read(fd, buffer, 1024);
// encrypt is necessary
if(encButtonOn->GetValue()) encryptHelper(buffer,1024);
// Send a chunk of data
n=send(sockFile_, buffer, 1024, 0 );
// Wait for an acknowledgement
n = recv(sockFile_, rec, 10, 0 );
}
else // reamining file bytes
{
value+=size;
n=read(fd, buffer, size);
if(encButtonOn->GetValue()) encryptHelper(buffer,size);
buffer[size]='\0';
n=send(sockFile_,buffer, size, 0 );
n=recv(sockFile_, rec, 10, 0 );
}
MyFooEvent event( 0, 992 );
double firstBit = (double)value/remSize;
firstBit=firstBit*100.0;
event.adouble=firstBit;
wxPostEvent (this, event);
size -= 1024;
}
// Send a completion string
int n = send(sockFile_, "COMP",strlen("COMP"), 0 );
char buf[10];
bzero(buf,10);
// Receive an acknowledgemnt
n = recv(sockFile_, buf, 10, 0 );
return(0);
}
int MyFrame::receive_file()
{
// receive file size and send ack
char sizeBuffer[50];
bzero(sizeBuffer,50);
int n;
//read(This->sockpw,buffer,bufferSize);
n=read(sockFile_, sizeBuffer, 50);
n=send(sockFile_,"OK", strlen("OK"), 0 );
int size = atoi(sizeBuffer);
//std::cout<<size<<std::endl;
// receive file name and send ack
char saveName[256];
bzero(saveName,256);
n=read(sockFile_, saveName, 256);
n=send(sockFile_,"OK",strlen("OK"), 0 );
//std::cout<<saveName_<<std::endl;
// start file writing process to local disk
// decrypt first if necessary
std::cout<<arraySize(saveName)<<std::endl;
std::cout<<strlen(saveName)<<std::endl;
if(encButtonOn->GetValue()) decryptHelper(saveName,strlen(saveName));
ofstream outFile(saveName,ios::out|ios::binary|ios::app);
// vars for status gauge
int remSize=size;
int value=0;
while(size > 0)
{
// buffer for storing incoming data
char buf[1030];
bzero(buf,1030);
if(size>=1024)
{
value+=1024; // for status gauge
// receive chunk of data
n=recv(sockFile_, buf, 1024, 0 );
// decrypt if necessary
if(encButtonOn->GetValue()) decryptHelper(buf,1024);
// write chunk of data to disk
outFile.write(buf,1024);
// send acknowledgement
n = send(sockFile_, "OK", strlen("OK"), 0 );
}
else
{
value+=size;
n=recv(sockFile_, buf, size, 0 );
if(encButtonOn->GetValue()) decryptHelper(buf,size);
buf[size]='\0';
outFile.write(buf,size);
n = send(sockFile_, "OK", strlen("OK"), 0 );
}
// Update status gauge
MyFooEvent event( 0, 992 );
double firstBit = (double)value/remSize;
firstBit=firstBit*100.0;
event.adouble=firstBit;
wxPostEvent (this, event);
size -= 1024;
}
outFile.close();
// Receive 'COMP' and send acknowledgement
// ---------------------------------------
char buf[10];
bzero(buf,10);
n = recv(sockFile_, buf, 10, 0 );
n = send(sockFile_, "OK", strlen("OK"), 0 );
std::cout<<"File received..."<<std::endl;
// Display image event
MyFooEvent eventF( 0, 995 );
eventF.SetText(wxString(saveName, wxConvUTF8));
wxPostEvent (this, eventF);
return(0);
}
I'm assuming that:
char *pb=(char*)(std::string((passInput->GetValue()).mb_str()).c_str());
blowfish((unsigned char*)orig, inlength, pb, DES_DECRYPT);
decrypts into pb, which is actually the buffer of a temporary string. You simply cannot use std::string like this. The fact that you had to use so many casrs to do this shouldhave been a warning - good C and C++ code does not normally require casts at all. Basically, you need to rethink what you are doing.
not sure, could be a buffer overrun somewhere or memory corruption...
you could use valgrind to detect the issue or perhaps try simplifying the conversions/...
Having fixed a few bugs by asking a few other questions, I have gotten the file encryption process working, but only when the client and server are both on the same localhost machine. When they reside on different machines, the file still ends up being corrupted. I think it is due to the fact that send_file and receive file are called from threads as follows:
void
*MyFrame::send_fileT(void* tid)
{
accessHelper* ah = static_cast<accessHelper*>(tid);
MyFrame* This = ah->This;
This->send_file(fileSendID);
pthread_exit(NULL);
}
void
*MyFrame::receive_fileT(void* tid)
{
accessHelper* ah = static_cast<accessHelper*>(tid);
MyFrame* This = ah->This;
This->receive_file();
pthread_exit(NULL);
}
....and then the receive_file or send_file functions are calling the blowfish function to carry out the encryption. Now if a function is called within a pthread (i.e. send_file and receive_file), then if that function calls another function (i.e. encryptHelper -- blowfish), is it possible that the calling function will not 'properly' wait for the called function to finish correctly?
Fixed:
n=read(fd, buffer, 2048);
if(enc)encryptHelper(buffer,n);
n=send(sockFile_, buffer, n, 0 );
[called in a loop]
The problem was, was that it cannot be ensured that all n bytes of the encrypted buffer are transferred. Thus only some of the encrypted bytes are sent leading to inconsistent decryption on the receiving end.