How can I Construct key pair values using strings in CPP? - c++

I used poco json for key pair values, and it consuming more time(Sending Images consuming more time).
std::vector<unsigned char> buff, buff1;
cv::imencode(".png", image1, buff);
cv::imencode(".png", image2, buff1);
Poco::JSON::Object obj;
obj.set("photo1", buff);
obj.set("photo2", buff1);
std::stringstream ss;
obj.stringify(ss);
I want to construct same in terms of string in CPP.
cv::imencode(".png", image1, buff);
cv::imencode(".png", image2, buff1);
std::string data ="{\"photo1\":" +std::string(buff.begin(),buff.end());
data .append(",");
data.append("\"photo2"\":" + std::string(buff1.begin(),buff1.end()));
data.append("}");
If I Done step 2 How can i extract back the imencode data.
I tried converting json string.
a) imencode-> string -> stringstream.
I want to construct key pair value in terms of string.
Minimal Code
std::string stringifyTheBothImages(cv::Mat image, cv::Mat image1) {
std::ostringstream ostr;
std::vector<unsigned char> buffer, buffer1;
cv::imencode(".png", image, buffer);
std::string address_json = "{\"photo1\":" + data;
address_json.append(",");
address_json.append("\"photo2\":" + data);
address_json.append("}");
return address_json;
}
int main(){
cv::Mat image =
cv::imread("C:/Amar/dec_16_2022_slabs/train/class0/image0.png", 1);
data2 = stringifyTheBothImages(image, image);
Poco::JSON::Parser parser;
auto result = parser.parse(data2);
Poco::JSON::Object::Ptr ptr_object =
result.extract<Poco::JSON::Object::Ptr>();
std::cout << ptr_object->get("photo1").toString() << std::endl;
}

Since with JSON method you get an array of numbers for both the keys, you can do similar in C++ by converting each value in the buffer to an integer.
Also, instead of using string::append use stringstream.
#include <iostream>
#include <vector>
#include <sstream>
int main()
{
std::vector<uint8_t> buffer1 = {129, 0, 234, 231, 255, 234, 13, 134, 2};
std::vector<uint8_t> buffer2 = {109, 0, 204, 201, 205, 204, 10, 104, 4};
std::stringstream ss;
ss << "{\"photo1\":[";
for (size_t i = 0; i < buffer1.size(); i++)
{
ss << static_cast<int>(buffer1[i]);
if (i != buffer1.size() - 1)
ss << ',';
}
ss << "],\"photo2\":[";
for (size_t i = 0; i < buffer2.size(); i++)
{
ss << static_cast<int>(buffer2[i]);
if (i != buffer2.size() - 1)
ss << ',';
}
ss << "]}";
std::cout << ss.str();
}

Related

Reading file into buffer and avoiding splitting lines between reads

I was reading sehe's answer for fast text file reading in C++, which looks like this.
static uintmax_t wc(char const *fname)
{
static const auto BUFFER_SIZE = 16*1024;
int fd = open(fname, O_RDONLY);
if(fd == -1)
handle_error("open");
/* Advise the kernel of our access pattern. */
posix_fadvise(fd, 0, 0, 1); // FDADVICE_SEQUENTIAL
char buf[BUFFER_SIZE + 1];
uintmax_t lines = 0;
while(size_t bytes_read = read(fd, buf, BUFFER_SIZE))
{
if(bytes_read == (size_t)-1)
handle_error("read failed");
if (!bytes_read)
break;
for(char *p = buf; (p = (char*) memchr(p, '\n', (buf + bytes_read) - p)); ++p)
++lines;
}
return lines;
}
This is cool, but I was wondering if a similar approach can be taken when we aren't dealing with a character operation like counting newlines, but want to operate on each line of data. Say for instance I had a file of doubles, and already some function parse_line_to_double to use on each line.
12.44243
4242.910
...
That is, how can I read BUFFER_SIZE bytes into my buffer but avoid splitting the last line read? Effectively, can I ask "Give me BUFFER_SIZE or less bytes while ensuring that the last byte read is a newline character (or EOF)"?
Knowing extremely little about low level IO like this, ideas that came to mind were
Can I "back up" fd to the most recent newline between iterations?
Do I have to keep a second buffer holding a copy of the current line being read all the time?
Here is a comparison test. First, lets try the easy way. Just read the file with standard C++ functions:
#include <iostream>
#include <string>
#include <fstream> //std::ifstream
#include <sstream> //std::stringstream
uintmax_t test1(char const *fname)
{
std::ifstream fin(fname);
if(!fin) return 0;
uintmax_t lines = 0;
std::string str;
double value;
while(fin >> value)
{
//std::cout << value << "\n";
lines++;
}
return lines;
}
Next, with std::stringstream this is about 2.5 times faster:
uintmax_t test2(char const *fname)
{
std::ifstream fin(fname);
if(!fin) return 0;
uintmax_t lines = 0;
std::string str;
double value;
std::stringstream ss;
ss << fin.rdbuf();
while(ss >> value)
lines++;
return lines;
}
Next, lets read the whole file in to memory. This will be fine as long as the file is less than 1 GiB or so. Assuming there is a double value on each line, lets extract that value. test3 is more complicated and less flexible, and it's not any faster than test2:
uintmax_t test3(char const *fname)
{
std::ifstream fin(fname, std::ios::binary);
if(!fin) return 0;
fin.seekg(0, std::ios::end);
size_t filesize = (size_t)fin.tellg();
fin.seekg(0);
std::string str(filesize, 0);
fin.read(&str[0], filesize);
double value;
uintmax_t lines = 0;
size_t beg = 0;
size_t i;
size_t len = str.size();
for(i = 0; i < len; i++)
{
if(str[i] == '\n' || i == len - 1)
{
try
{
value = std::stod(str.substr(beg, i - beg));
//std::cout << value << "\n";
beg = i + 1;
lines++;
}
catch(...)
{
}
}
}
return lines;
}
For comparison to the wc function in the question, let's read the whole file in to memory and only count the number of lines. This runs a little faster than wc (as expected), suggesting that there is no need for additional optimizations
uintmax_t test_countlines(char const *fname)
{
std::ifstream fin(fname, std::ios::binary);
if(!fin) return 0;
fin.seekg(0, std::ios::end);
size_t filesize = (size_t)fin.tellg();
fin.seekg(0);
std::string str(filesize, 0);
fin.read(&str[0], filesize);
uintmax_t lines = 0;
for(auto &c : str)
if(c == '\n')
lines++;
return lines;
}

OpenSSL HMAC_SHA1 as b64 hash (not base64 encoded) [duplicate]

I've been trying to figure out the openssl documentation for base64 decoding and encoding. I found some code snippets below
#include <openssl/sha.h>
#include <openssl/hmac.h>
#include <openssl/evp.h>
#include <openssl/bio.h>
#include <openssl/buffer.h>
char *base64(const unsigned char *input, int length)
{
BIO *bmem, *b64;
BUF_MEM *bptr;
b64 = BIO_new(BIO_f_base64());
bmem = BIO_new(BIO_s_mem());
b64 = BIO_push(b64, bmem);
BIO_write(b64, input, length);
BIO_flush(b64);
BIO_get_mem_ptr(b64, &bptr);
char *buff = (char *)malloc(bptr->length);
memcpy(buff, bptr->data, bptr->length-1);
buff[bptr->length-1] = 0;
BIO_free_all(b64);
return buff;
}
char *decode64(unsigned char *input, int length)
{
BIO *b64, *bmem;
char *buffer = (char *)malloc(length);
memset(buffer, 0, length);
b64 = BIO_new(BIO_f_base64());
bmem = BIO_new_mem_buf(input, length);
bmem = BIO_push(b64, bmem);
BIO_read(bmem, buffer, length);
BIO_free_all(bmem);
return buffer;
}
This only seems to work for single line strings such as "Start", the moment I introduce complex strings with newlines and spaces etc it fails horribly.
It doesn't even have to be openssl, a simple class or set of functions that do the same thing would be fine, theres a very complicated build process for the solution and I am trying to avoid having to go in there and make multiple changes. The only reason I went for openssl is because the solution is already compiled with the libraries.
Personally, I find the OpenSSL API to be so incredibly painful to use, I avoid it unless the cost of avoiding it is extremely high. I find it quite upsetting that it has become the standard API in the crypto world.
I was feeling bored, and I wrote you one in C++. This one should even handle the edge cases that can cause security problems, like, for example, encoding a string that results in integer overflow because it's too large.
I have done some unit testing on it, so it should work.
#include <string>
#include <cassert>
#include <limits>
#include <stdexcept>
#include <cctype>
static const char b64_table[65] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
static const char reverse_table[128] = {
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 62, 64, 64, 64, 63,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 64, 64, 64, 64, 64, 64,
64, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 64, 64, 64, 64, 64,
64, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 64, 64, 64, 64, 64
};
::std::string base64_encode(const ::std::string &bindata)
{
using ::std::string;
using ::std::numeric_limits;
if (bindata.size() > (numeric_limits<string::size_type>::max() / 4u) * 3u) {
throw ::std::length_error("Converting too large a string to base64.");
}
const ::std::size_t binlen = bindata.size();
// Use = signs so the end is properly padded.
string retval((((binlen + 2) / 3) * 4), '=');
::std::size_t outpos = 0;
int bits_collected = 0;
unsigned int accumulator = 0;
const string::const_iterator binend = bindata.end();
for (string::const_iterator i = bindata.begin(); i != binend; ++i) {
accumulator = (accumulator << 8) | (*i & 0xffu);
bits_collected += 8;
while (bits_collected >= 6) {
bits_collected -= 6;
retval[outpos++] = b64_table[(accumulator >> bits_collected) & 0x3fu];
}
}
if (bits_collected > 0) { // Any trailing bits that are missing.
assert(bits_collected < 6);
accumulator <<= 6 - bits_collected;
retval[outpos++] = b64_table[accumulator & 0x3fu];
}
assert(outpos >= (retval.size() - 2));
assert(outpos <= retval.size());
return retval;
}
::std::string base64_decode(const ::std::string &ascdata)
{
using ::std::string;
string retval;
const string::const_iterator last = ascdata.end();
int bits_collected = 0;
unsigned int accumulator = 0;
for (string::const_iterator i = ascdata.begin(); i != last; ++i) {
const int c = *i;
if (::std::isspace(c) || c == '=') {
// Skip whitespace and padding. Be liberal in what you accept.
continue;
}
if ((c > 127) || (c < 0) || (reverse_table[c] > 63)) {
throw ::std::invalid_argument("This contains characters not legal in a base64 encoded string.");
}
accumulator = (accumulator << 6) | reverse_table[c];
bits_collected += 6;
if (bits_collected >= 8) {
bits_collected -= 8;
retval += static_cast<char>((accumulator >> bits_collected) & 0xffu);
}
}
return retval;
}
Rather than using the BIO_ interface it's much easier to use the EVP_ interface. For instance:
#include <iostream>
#include <stdlib.h>
#include <openssl/evp.h>
char *base64(const unsigned char *input, int length) {
const auto pl = 4*((length+2)/3);
auto output = reinterpret_cast<char *>(calloc(pl+1, 1)); //+1 for the terminating null that EVP_EncodeBlock adds on
const auto ol = EVP_EncodeBlock(reinterpret_cast<unsigned char *>(output), input, length);
if (pl != ol) { std::cerr << "Whoops, encode predicted " << pl << " but we got " << ol << "\n"; }
return output;
}
unsigned char *decode64(const char *input, int length) {
const auto pl = 3*length/4;
auto output = reinterpret_cast<unsigned char *>(calloc(pl+1, 1));
const auto ol = EVP_DecodeBlock(output, reinterpret_cast<const unsigned char *>(input), length);
if (pl != ol) { std::cerr << "Whoops, decode predicted " << pl << " but we got " << ol << "\n"; }
return output;
}
The EVP functions include a streaming interface too, see the man page.
Here is an example of OpenSSL base64 encode/decode I wrote:
Notice, I have some macros/classes in the code that I wrote, but none of them is important for the example. It is simply some C++ wrappers I wrote:
buffer base64::encode( const buffer& data )
{
// bio is simply a class that wraps BIO* and it free the BIO in the destructor.
bio b64(BIO_f_base64()); // create BIO to perform base64
BIO_set_flags(b64,BIO_FLAGS_BASE64_NO_NL);
bio mem(BIO_s_mem()); // create BIO that holds the result
// chain base64 with mem, so writing to b64 will encode base64 and write to mem.
BIO_push(b64, mem);
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, data.data, (int)data.size);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
BIO_flush(b64);
// get a pointer to mem's data
char* dt;
long len = BIO_get_mem_data(mem, &dt);
// assign data to output
std::string s(dt, len);
return buffer(s.length()+sizeof(char), (byte*)s.c_str());
}
This works for me, and verified no memory leaks with valgrind.
#include <openssl/bio.h>
#include <openssl/evp.h>
#include <cstring>
#include <memory>
#include <string>
#include <vector>
#include <iostream>
namespace {
struct BIOFreeAll { void operator()(BIO* p) { BIO_free_all(p); } };
}
std::string Base64Encode(const std::vector<unsigned char>& binary)
{
std::unique_ptr<BIO,BIOFreeAll> b64(BIO_new(BIO_f_base64()));
BIO_set_flags(b64.get(), BIO_FLAGS_BASE64_NO_NL);
BIO* sink = BIO_new(BIO_s_mem());
BIO_push(b64.get(), sink);
BIO_write(b64.get(), binary.data(), binary.size());
BIO_flush(b64.get());
const char* encoded;
const long len = BIO_get_mem_data(sink, &encoded);
return std::string(encoded, len);
}
// Assumes no newlines or extra characters in encoded string
std::vector<unsigned char> Base64Decode(const char* encoded)
{
std::unique_ptr<BIO,BIOFreeAll> b64(BIO_new(BIO_f_base64()));
BIO_set_flags(b64.get(), BIO_FLAGS_BASE64_NO_NL);
BIO* source = BIO_new_mem_buf(encoded, -1); // read-only source
BIO_push(b64.get(), source);
const int maxlen = strlen(encoded) / 4 * 3 + 1;
std::vector<unsigned char> decoded(maxlen);
const int len = BIO_read(b64.get(), decoded.data(), maxlen);
decoded.resize(len);
return decoded;
}
int main()
{
const char* msg = "hello";
const std::vector<unsigned char> binary(msg, msg+strlen(msg));
const std::string encoded = Base64Encode(binary);
std::cout << "encoded = " << encoded << std::endl;
const std::vector<unsigned char> decoded = Base64Decode(encoded.c_str());
std::cout << "decoded = ";
for (unsigned char c : decoded) std::cout << c;
std::cout << std::endl;
return 0;
}
Compile:
g++ -lcrypto main.cc
Output:
encoded = aGVsbG8=
decoded = hello
So many horrible C code examples with buffers and malloc(), what about using std::string properly on this C++ tagged question?
#include <openssl/bio.h>
#include <openssl/evp.h>
#include <openssl/buffer.h>
#include <string>
std::string base64_encode(const std::string& input)
{
const auto base64_memory = BIO_new(BIO_s_mem());
auto base64 = BIO_new(BIO_f_base64());
base64 = BIO_push(base64, base64_memory);
BIO_write(base64, input.c_str(), static_cast<int>(input.length()));
BIO_flush(base64);
BUF_MEM* buffer_memory{};
BIO_get_mem_ptr(base64, &buffer_memory);
auto base64_encoded = std::string(buffer_memory->data, buffer_memory->length - 1);
BIO_free_all(base64);
return base64_encoded;
}
I like mtrw's use of EVP.
Below is my "modern C++" take on his answer without manual memory allocation (calloc). It will take a std::string but it can easily be overloaded to use raw bytes for example.
#include <openssl/evp.h>
#include <memory>
#include <stdexcept>
#include <vector>
auto EncodeBase64(const std::string& to_encode) -> std::string {
/// #sa https://www.openssl.org/docs/manmaster/man3/EVP_EncodeBlock.html
const auto predicted_len = 4 * ((to_encode.length() + 2) / 3); // predict output size
const auto output_buffer{std::make_unique<char[]>(predicted_len + 1)};
const std::vector<unsigned char> vec_chars{to_encode.begin(), to_encode.end()}; // convert to_encode into uchar container
const auto output_len = EVP_EncodeBlock(reinterpret_cast<unsigned char*>(output_buffer.get()), vec_chars.data(), static_cast<int>(vec_chars.size()));
if (predicted_len != static_cast<unsigned long>(output_len)) {
throw std::runtime_error("EncodeBase64 error");
}
return output_buffer.get();
}
auto DecodeBase64(const std::string& to_decode) -> std::string {
/// #sa https://www.openssl.org/docs/manmaster/man3/EVP_DecodeBlock.html
const auto predicted_len = 3 * to_decode.length() / 4; // predict output size
const auto output_buffer{std::make_unique<char[]>(predicted_len + 1)};
const std::vector<unsigned char> vec_chars{to_decode.begin(), to_decode.end()}; // convert to_decode into uchar container
const auto output_len = EVP_DecodeBlock(reinterpret_cast<unsigned char*>(output_buffer.get()), vec_chars.data(), static_cast<int>(vec_chars.size()));
if (predicted_len != static_cast<unsigned long>(output_len)) {
throw std::runtime_error("DecodeBase64 error");
}
return output_buffer.get();
}
There's probably a cleaner/better way of doing this (I'd like to get rid of reinterpret_cast). You'll also definitely want a try/catch block to deal with the potential exception.
Improved TCS answer to remove macros/datastructures
unsigned char *encodeb64mem( unsigned char *data, int len, int *lenoutput )
{
// bio is simply a class that wraps BIO* and it free the BIO in the destructor.
BIO *b64 = BIO_new(BIO_f_base64()); // create BIO to perform base64
BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL);
BIO *mem = BIO_new(BIO_s_mem()); // create BIO that holds the result
// chain base64 with mem, so writing to b64 will encode base64 and write to mem.
BIO_push(b64, mem);
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, data, len);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
BIO_flush(b64);
// get a pointer to mem's data
unsigned char* output;
*lenoutput = BIO_get_mem_data(mem, &output);
// assign data to output
//std::string s(dt, len2);
return output;
}
To write to file
int encodeb64(unsigned char* input, const char* filenm, int leni)
{
BIO *b64 = BIO_new(BIO_f_base64());
BIO_set_flags(b64,BIO_FLAGS_BASE64_NO_NL);
BIO *file = BIO_new_file(filenm, "w");
BIO *mem = BIO_new(BIO_f_buffer());
BIO_push(b64, mem);
BIO_push(mem, file);
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, input, leni);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
BIO_flush(b64);
BIO_pop(b64);
BIO_free_all(b64);
return 0;
}
Base64 encoding from file to file. Many times due to file constraint we have read in chunks of data and do encoding. Below is the code.
int encodeb64FromFile(const char* input, const char* outputfilename)
{
BIO *b64 = BIO_new(BIO_f_base64());
BIO_set_flags(b64,BIO_FLAGS_BASE64_NO_NL);
int leni = 3*64;
unsigned char *data[3*64];
BIO *file = BIO_new_file(outputfilename, "w");
BIO *mem = BIO_new(BIO_f_buffer());
BIO_push(b64, mem);
BIO_push(mem, file);
FILE *fp = fopen(input, "rb");
while ((leni = fread(data, 1, sizeof data, fp)) > 0) {
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, data, leni);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
}
BIO_flush(b64);
BIO_pop(b64);
BIO_free_all(b64);
fclose(fp);
return 0;
}
Base64 is really pretty simple; you shouldn't have trouble finding any number of implementations via a quick Google. For example here is a reference implementation in C from the Internet Software Consortium, with detailed comments explaining the process.
The openssl implementation layers a lot of complexity with the "BIO" stuff that's not (IMHO) very useful if all you're doing is decoding/encoding.
#include <openssl/bio.h>
typedef unsigned char byte;
namespace base64 {
static void Encode(const byte* in, size_t in_len,
char** out, size_t* out_len) {
BIO *buff, *b64f;
BUF_MEM *ptr;
b64f = BIO_new(BIO_f_base64());
buff = BIO_new(BIO_s_mem());
buff = BIO_push(b64f, buff);
BIO_set_flags(buff, BIO_FLAGS_BASE64_NO_NL);
BIO_set_close(buff, BIO_CLOSE);
BIO_write(buff, in, in_len);
BIO_flush(buff);
BIO_get_mem_ptr(buff, &ptr);
(*out_len) = ptr->length;
(*out) = (char *) malloc(((*out_len) + 1) * sizeof(char));
memcpy(*out, ptr->data, (*out_len));
(*out)[(*out_len)] = '\0';
BIO_free_all(buff);
}
static void Decode(const char* in, size_t in_len,
byte** out, size_t* out_len) {
BIO *buff, *b64f;
b64f = BIO_new(BIO_f_base64());
buff = BIO_new_mem_buf((void *)in, in_len);
buff = BIO_push(b64f, buff);
(*out) = (byte *) malloc(in_len * sizeof(char));
BIO_set_flags(buff, BIO_FLAGS_BASE64_NO_NL);
BIO_set_close(buff, BIO_CLOSE);
(*out_len) = BIO_read(buff, (*out), in_len);
(*out) = (byte *) realloc((void *)(*out), ((*out_len) + 1) * sizeof(byte));
(*out)[(*out_len)] = '\0';
BIO_free_all(buff);
}
}
Late to the party, but I came across this problem recently myself, but was unhappy with both the BIO solution, which is unnecessarily convoluted, but did not like 'EncodeBlock' either, because it introduces newline characters I do not want in my Base64 encoded string.
After a little sniffing, I came across the header file openssl/include/crypto/evp.h which is not part of the default installation (which only exports the include/openssl folder for me), but exports the solution to the problem.
void evp_encode_ctx_set_flags(EVP_ENCODE_CTX *ctx, unsigned int flags);
/* EVP_ENCODE_CTX flags */
/* Don't generate new lines when encoding */
#define EVP_ENCODE_CTX_NO_NEWLINES 1
/* Use the SRP base64 alphabet instead of the standard one */
#define EVP_ENCODE_CTX_USE_SRP_ALPHABET 2
Using this function, the 'no newline' becomes possible using the EVP interface.
Example:
if (EVP_ENCODE_CTX *context = EVP_ENCODE_CTX_new())
{
EVP_EncodeInit(context);
evp_encode_ctx_set_flags(context, EVP_ENCODE_CTX_NO_NEWLINES);
while (hasData())
{
uint8_t *data;
int32_t length = fetchData(&data);
int32_t size = (((EVP_ENCODE_CTX_num(context) + length)/48) * 65) + 1;
uint8_t buffer[size];
EVP_EncodeUpdate(context, buffer, &size, pData, length);
//process encoded data.
}
uint8_t buffer[65];
int32_t writtenBytes;
EVP_EncodeFinal(context, buffer, &writtenBytes);
//Do something with the final remainder of the encoded string.
EVP_ENCODE_CTX_free(context);
}
This piece of code will encode the buffer to Base64 without the newlines.
Please note the use of EVP_ENCODE_CTX_num to obtain the 'leftover bytes' still stored in the context object to calculate the correct buffer size.
It is only necessary, if you need to call EVP_EncodeUpdate multiple times, because your data is exceedingly large or not available at once.

Hex manipulations

There are very nice Hex.Decode(string hexString) and Hex.ToHexString(byte[] hexArray) methods in BouncyCastle crypto library (C#, Java).
How to make the same conversions with CString variable, that stores hex-string for example "af010cdb" to unsigned char* and vice-versa in C++?
These are real easy to implement:
CString ToHexString(const CByteArray& Array)
{
CString sHexString;
for (int i=0; i<Array.GetSize(); i++)
sHexString.AppendFormat(_T("%02X"), Array[i]);
return sHexString;
}
void DecodeHexString(const CString& sHexString, CByteArray &Array)
{
if (sHexString.IsEmpty())
return;
int nLen = sHexString.GetLength();
if ((nLen % 2) != 0)
{
ASSERT(FALSE);
return;
}
Array.SetSize(nLen/2);
int nByte = 0;
for(int i=0; i<nLen; i += 2)
{
CString sByte = sHexString.Mid(i, 2);
BYTE byte = (BYTE)_tcstol(sByte, NULL, 16);
Array[nByte] = byte;
nByte++;
}
}
You can also use std::string and std::vector<unsigned char>
Edit: removed sscanf and sprintf
std::vector<unsigned char> str_to_byte(std::string str)
{
std::vector<unsigned char> bytes;
std::stringstream ss(str);
std::string part;
while (ss >> std::setw(2) >> part)
{
try {
int i = std::stoi(part, 0, 16);
bytes.push_back(static_cast<unsigned char>(i));
}
catch (...) {//error msg
}
}
return bytes;
}
std::string byte_to_str(std::vector<unsigned char> bytes)
{
std::stringstream ss;
ss << std::hex << std::setfill('0') << std::uppercase;
for (auto c:bytes)
ss << std::setw(2) << static_cast<unsigned int>(c);
return ss.str();
}
int main()
{
std::vector<unsigned char> bytes = str_to_byte("EF010203A0A1FFz");
std::string str = byte_to_str(bytes);
cout << str << endl; //should print "EF010203A0A1FF"
return 0;
}

Is there a better way to search a file for a string?

I need to search a (non-text) file for the byte sequence "9µ}Æ" (or "\x39\xb5\x7d\xc6").
After 5 hours of searching online this is the best I could do. It works but I wanted to know if there is a better way:
char buffer;
int pos=in.tellg();
// search file for string
while(!in.eof()){
in.read(&buffer, 1);
pos=in.tellg();
if(buffer=='9'){
in.read(&buffer, 1);
pos=in.tellg();
if(buffer=='µ'){
in.read(&buffer, 1);
pos=in.tellg();
if(buffer=='}'){
in.read(&buffer, 1);
pos=in.tellg();
if(buffer=='Æ'){
cout << "found";
}
}
}
}
in.seekg((streampos) pos);
Note:
I can't use getline(). It's not a text file so there are probably not many line breaks.
Before I tried using a multi-character buffer and then copying the buffer to a C++ string, and then using string::find(). This didn't work because there are many '\0' characters throughout the file, so the sequence in the buffer would be cut very short when it was copied to the string.
Similar to what bames53 posted; I used a vector as a buffer:
std::ifstream ifs("file.bin");
ifs.seekg(0, std::ios::end);
std::streamsize f_size = ifs.tellg();
ifs.seekg(0, std::ios::beg);
std::vector<unsigned char> buffer(f_size);
ifs.read(buffer.data(), f_size);
std::vector<unsigned char> seq = {0x39, 0xb5, 0x7d, 0xc6};
bool found = std::search(buffer.begin(), buffer.end(), seq.begin(), seq.end()) != buffer.end();
If you don't mind loading the entire file into an in-memory array (or using mmap() to make it look like the file is in memory), you could then search for your character sequence in-memory, which is a bit easier to do:
// Works much like strstr(), except it looks for a binary sub-sequence rather than a string sub-sequence
const char * MemMem(const char * lookIn, int numLookInBytes, const char * lookFor, int numLookForBytes)
{
if (numLookForBytes == 0) return lookIn; // hmm, existential questions here
else if (numLookForBytes == numLookInBytes) return (memcmp(lookIn, lookFor, numLookInBytes) == 0) ? lookIn : NULL;
else if (numLookForBytes < numLookInBytes)
{
const char * startedAt = lookIn;
int matchCount = 0;
for (int i=0; i<numLookInBytes; i++)
{
if (lookIn[i] == lookFor[matchCount])
{
if (matchCount == 0) startedAt = &lookIn[i];
if (++matchCount == numLookForBytes) return startedAt;
}
else matchCount = 0;
}
}
return NULL;
}
.... then you can just call the above function on the in-memory data array:
char * ret = MemMem(theInMemoryArrayContainingFilesBytes, numBytesInFile, myShortSequence, 4);
if (ret != NULL) printf("Found it at offset %i\n", ret-theInMemoryArrayContainingFilesBytes);
else printf("It's not there.\n");
This program loads the entire file into memory and then uses std::search on it.
int main() {
std::string filedata;
{
std::ifstream fin("file.dat");
std::stringstream ss;
ss << fin.rdbuf();
filedata = ss.str();
}
std::string key = "\x39\xb5\x7d\xc6";
auto result = std::search(std::begin(filedata), std::end(filedata),
std::begin(key), std::end(key));
if (std::end(filedata) != result) {
std::cout << "found\n";
// result is an iterator pointing at '\x39'
}
}
const char delims[] = { 0x39, 0xb5, 0x7d, 0xc6 };
char buffer[4];
const size_t delim_size = 4;
const size_t last_index = delim_size - 1;
for ( size_t i = 0; i < last_index; ++i )
{
if ( ! ( is.get( buffer[i] ) ) )
return false; // stream to short
}
while ( is.get(buffer[last_index]) )
{
if ( memcmp( buffer, delims, delim_size ) == 0 )
break; // you are arrived
memmove( buffer, buffer + 1, last_index );
}
You are looking for 4 bytes:
unsigned int delim = 0xc67db539;
unsigned int uibuffer;
char * buffer = reinterpret_cast<char *>(&uibuffer);
for ( size_t i = 0; i < 3; ++i )
{
if ( ! ( is.get( buffer[i] ) ) )
return false; // stream to short
}
while ( is.get(buffer[3]) )
{
if ( uibuffer == delim )
break; // you are arrived
uibuffer >>= 8;
}
Because you said you cannot search the entire file because of null terminator characters in the string, here's an alternative for you, which reads the entire file in and uses recursion to find the first occurrence of a string inside of the whole file.
#include <iostream>
#include <fstream>
#include <string>
using namespace std;
string readFile (char *fileName) {
ifstream fi (fileName);
if (!fi)
cerr << "ERROR: Cannot open file" << endl;
else {
string str ((istreambuf_iterator<char>(fi)), istreambuf_iterator<char>());
return str;
}
return NULL;
}
bool findFirstOccurrenceOf_r (string haystack, char *needle, int haystack_pos, int needle_pos, int needle_len) {
if (needle_pos == needle_len)
return true;
if (haystack[haystack_pos] == needle[needle_pos])
return findFirstOccurrenceOf_r (haystack, needle, haystack_pos+1, needle_pos+1, needle_len);
return false;
}
int findFirstOccurrenceOf (string haystack, char *needle, int length) {
int pos = -1;
for (int i = 0; i < haystack.length() - length; i++) {
if (findFirstOccurrenceOf_r (haystack, needle, i, 0, length))
return i;
}
return pos;
}
int main () {
char str_to_find[4] = {0x39, 0xB5, 0x7D, 0xC6};
string contents = readFile ("input");
int pos = findFirstOccurrenceOf (contents, str_to_find, 4);
cout << pos << endl;
}
If the file is not too large, your best solution would be to load the whole file into memory, so you don't need to keep reading from the drive. If the file is too large to load in at once, you would want to load in chunks of the file at a time. But if you do load in chucks, make sure you check to edges of the chunks. It's possible that your chunk happens to split right in the middle of the string you're searching for.

Base64 encoding and decoding with OpenSSL

I've been trying to figure out the openssl documentation for base64 decoding and encoding. I found some code snippets below
#include <openssl/sha.h>
#include <openssl/hmac.h>
#include <openssl/evp.h>
#include <openssl/bio.h>
#include <openssl/buffer.h>
char *base64(const unsigned char *input, int length)
{
BIO *bmem, *b64;
BUF_MEM *bptr;
b64 = BIO_new(BIO_f_base64());
bmem = BIO_new(BIO_s_mem());
b64 = BIO_push(b64, bmem);
BIO_write(b64, input, length);
BIO_flush(b64);
BIO_get_mem_ptr(b64, &bptr);
char *buff = (char *)malloc(bptr->length);
memcpy(buff, bptr->data, bptr->length-1);
buff[bptr->length-1] = 0;
BIO_free_all(b64);
return buff;
}
char *decode64(unsigned char *input, int length)
{
BIO *b64, *bmem;
char *buffer = (char *)malloc(length);
memset(buffer, 0, length);
b64 = BIO_new(BIO_f_base64());
bmem = BIO_new_mem_buf(input, length);
bmem = BIO_push(b64, bmem);
BIO_read(bmem, buffer, length);
BIO_free_all(bmem);
return buffer;
}
This only seems to work for single line strings such as "Start", the moment I introduce complex strings with newlines and spaces etc it fails horribly.
It doesn't even have to be openssl, a simple class or set of functions that do the same thing would be fine, theres a very complicated build process for the solution and I am trying to avoid having to go in there and make multiple changes. The only reason I went for openssl is because the solution is already compiled with the libraries.
Personally, I find the OpenSSL API to be so incredibly painful to use, I avoid it unless the cost of avoiding it is extremely high. I find it quite upsetting that it has become the standard API in the crypto world.
I was feeling bored, and I wrote you one in C++. This one should even handle the edge cases that can cause security problems, like, for example, encoding a string that results in integer overflow because it's too large.
I have done some unit testing on it, so it should work.
#include <string>
#include <cassert>
#include <limits>
#include <stdexcept>
#include <cctype>
static const char b64_table[65] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
static const char reverse_table[128] = {
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64,
64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 62, 64, 64, 64, 63,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 64, 64, 64, 64, 64, 64,
64, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 64, 64, 64, 64, 64,
64, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 64, 64, 64, 64, 64
};
::std::string base64_encode(const ::std::string &bindata)
{
using ::std::string;
using ::std::numeric_limits;
if (bindata.size() > (numeric_limits<string::size_type>::max() / 4u) * 3u) {
throw ::std::length_error("Converting too large a string to base64.");
}
const ::std::size_t binlen = bindata.size();
// Use = signs so the end is properly padded.
string retval((((binlen + 2) / 3) * 4), '=');
::std::size_t outpos = 0;
int bits_collected = 0;
unsigned int accumulator = 0;
const string::const_iterator binend = bindata.end();
for (string::const_iterator i = bindata.begin(); i != binend; ++i) {
accumulator = (accumulator << 8) | (*i & 0xffu);
bits_collected += 8;
while (bits_collected >= 6) {
bits_collected -= 6;
retval[outpos++] = b64_table[(accumulator >> bits_collected) & 0x3fu];
}
}
if (bits_collected > 0) { // Any trailing bits that are missing.
assert(bits_collected < 6);
accumulator <<= 6 - bits_collected;
retval[outpos++] = b64_table[accumulator & 0x3fu];
}
assert(outpos >= (retval.size() - 2));
assert(outpos <= retval.size());
return retval;
}
::std::string base64_decode(const ::std::string &ascdata)
{
using ::std::string;
string retval;
const string::const_iterator last = ascdata.end();
int bits_collected = 0;
unsigned int accumulator = 0;
for (string::const_iterator i = ascdata.begin(); i != last; ++i) {
const int c = *i;
if (::std::isspace(c) || c == '=') {
// Skip whitespace and padding. Be liberal in what you accept.
continue;
}
if ((c > 127) || (c < 0) || (reverse_table[c] > 63)) {
throw ::std::invalid_argument("This contains characters not legal in a base64 encoded string.");
}
accumulator = (accumulator << 6) | reverse_table[c];
bits_collected += 6;
if (bits_collected >= 8) {
bits_collected -= 8;
retval += static_cast<char>((accumulator >> bits_collected) & 0xffu);
}
}
return retval;
}
Rather than using the BIO_ interface it's much easier to use the EVP_ interface. For instance:
#include <iostream>
#include <stdlib.h>
#include <openssl/evp.h>
char *base64(const unsigned char *input, int length) {
const auto pl = 4*((length+2)/3);
auto output = reinterpret_cast<char *>(calloc(pl+1, 1)); //+1 for the terminating null that EVP_EncodeBlock adds on
const auto ol = EVP_EncodeBlock(reinterpret_cast<unsigned char *>(output), input, length);
if (pl != ol) { std::cerr << "Whoops, encode predicted " << pl << " but we got " << ol << "\n"; }
return output;
}
unsigned char *decode64(const char *input, int length) {
const auto pl = 3*length/4;
auto output = reinterpret_cast<unsigned char *>(calloc(pl+1, 1));
const auto ol = EVP_DecodeBlock(output, reinterpret_cast<const unsigned char *>(input), length);
if (pl != ol) { std::cerr << "Whoops, decode predicted " << pl << " but we got " << ol << "\n"; }
return output;
}
The EVP functions include a streaming interface too, see the man page.
Here is an example of OpenSSL base64 encode/decode I wrote:
Notice, I have some macros/classes in the code that I wrote, but none of them is important for the example. It is simply some C++ wrappers I wrote:
buffer base64::encode( const buffer& data )
{
// bio is simply a class that wraps BIO* and it free the BIO in the destructor.
bio b64(BIO_f_base64()); // create BIO to perform base64
BIO_set_flags(b64,BIO_FLAGS_BASE64_NO_NL);
bio mem(BIO_s_mem()); // create BIO that holds the result
// chain base64 with mem, so writing to b64 will encode base64 and write to mem.
BIO_push(b64, mem);
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, data.data, (int)data.size);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
BIO_flush(b64);
// get a pointer to mem's data
char* dt;
long len = BIO_get_mem_data(mem, &dt);
// assign data to output
std::string s(dt, len);
return buffer(s.length()+sizeof(char), (byte*)s.c_str());
}
This works for me, and verified no memory leaks with valgrind.
#include <openssl/bio.h>
#include <openssl/evp.h>
#include <cstring>
#include <memory>
#include <string>
#include <vector>
#include <iostream>
namespace {
struct BIOFreeAll { void operator()(BIO* p) { BIO_free_all(p); } };
}
std::string Base64Encode(const std::vector<unsigned char>& binary)
{
std::unique_ptr<BIO,BIOFreeAll> b64(BIO_new(BIO_f_base64()));
BIO_set_flags(b64.get(), BIO_FLAGS_BASE64_NO_NL);
BIO* sink = BIO_new(BIO_s_mem());
BIO_push(b64.get(), sink);
BIO_write(b64.get(), binary.data(), binary.size());
BIO_flush(b64.get());
const char* encoded;
const long len = BIO_get_mem_data(sink, &encoded);
return std::string(encoded, len);
}
// Assumes no newlines or extra characters in encoded string
std::vector<unsigned char> Base64Decode(const char* encoded)
{
std::unique_ptr<BIO,BIOFreeAll> b64(BIO_new(BIO_f_base64()));
BIO_set_flags(b64.get(), BIO_FLAGS_BASE64_NO_NL);
BIO* source = BIO_new_mem_buf(encoded, -1); // read-only source
BIO_push(b64.get(), source);
const int maxlen = strlen(encoded) / 4 * 3 + 1;
std::vector<unsigned char> decoded(maxlen);
const int len = BIO_read(b64.get(), decoded.data(), maxlen);
decoded.resize(len);
return decoded;
}
int main()
{
const char* msg = "hello";
const std::vector<unsigned char> binary(msg, msg+strlen(msg));
const std::string encoded = Base64Encode(binary);
std::cout << "encoded = " << encoded << std::endl;
const std::vector<unsigned char> decoded = Base64Decode(encoded.c_str());
std::cout << "decoded = ";
for (unsigned char c : decoded) std::cout << c;
std::cout << std::endl;
return 0;
}
Compile:
g++ -lcrypto main.cc
Output:
encoded = aGVsbG8=
decoded = hello
So many horrible C code examples with buffers and malloc(), what about using std::string properly on this C++ tagged question?
#include <openssl/bio.h>
#include <openssl/evp.h>
#include <openssl/buffer.h>
#include <string>
std::string base64_encode(const std::string& input)
{
const auto base64_memory = BIO_new(BIO_s_mem());
auto base64 = BIO_new(BIO_f_base64());
base64 = BIO_push(base64, base64_memory);
BIO_write(base64, input.c_str(), static_cast<int>(input.length()));
BIO_flush(base64);
BUF_MEM* buffer_memory{};
BIO_get_mem_ptr(base64, &buffer_memory);
auto base64_encoded = std::string(buffer_memory->data, buffer_memory->length - 1);
BIO_free_all(base64);
return base64_encoded;
}
I like mtrw's use of EVP.
Below is my "modern C++" take on his answer without manual memory allocation (calloc). It will take a std::string but it can easily be overloaded to use raw bytes for example.
#include <openssl/evp.h>
#include <memory>
#include <stdexcept>
#include <vector>
auto EncodeBase64(const std::string& to_encode) -> std::string {
/// #sa https://www.openssl.org/docs/manmaster/man3/EVP_EncodeBlock.html
const auto predicted_len = 4 * ((to_encode.length() + 2) / 3); // predict output size
const auto output_buffer{std::make_unique<char[]>(predicted_len + 1)};
const std::vector<unsigned char> vec_chars{to_encode.begin(), to_encode.end()}; // convert to_encode into uchar container
const auto output_len = EVP_EncodeBlock(reinterpret_cast<unsigned char*>(output_buffer.get()), vec_chars.data(), static_cast<int>(vec_chars.size()));
if (predicted_len != static_cast<unsigned long>(output_len)) {
throw std::runtime_error("EncodeBase64 error");
}
return output_buffer.get();
}
auto DecodeBase64(const std::string& to_decode) -> std::string {
/// #sa https://www.openssl.org/docs/manmaster/man3/EVP_DecodeBlock.html
const auto predicted_len = 3 * to_decode.length() / 4; // predict output size
const auto output_buffer{std::make_unique<char[]>(predicted_len + 1)};
const std::vector<unsigned char> vec_chars{to_decode.begin(), to_decode.end()}; // convert to_decode into uchar container
const auto output_len = EVP_DecodeBlock(reinterpret_cast<unsigned char*>(output_buffer.get()), vec_chars.data(), static_cast<int>(vec_chars.size()));
if (predicted_len != static_cast<unsigned long>(output_len)) {
throw std::runtime_error("DecodeBase64 error");
}
return output_buffer.get();
}
There's probably a cleaner/better way of doing this (I'd like to get rid of reinterpret_cast). You'll also definitely want a try/catch block to deal with the potential exception.
Improved TCS answer to remove macros/datastructures
unsigned char *encodeb64mem( unsigned char *data, int len, int *lenoutput )
{
// bio is simply a class that wraps BIO* and it free the BIO in the destructor.
BIO *b64 = BIO_new(BIO_f_base64()); // create BIO to perform base64
BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL);
BIO *mem = BIO_new(BIO_s_mem()); // create BIO that holds the result
// chain base64 with mem, so writing to b64 will encode base64 and write to mem.
BIO_push(b64, mem);
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, data, len);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
BIO_flush(b64);
// get a pointer to mem's data
unsigned char* output;
*lenoutput = BIO_get_mem_data(mem, &output);
// assign data to output
//std::string s(dt, len2);
return output;
}
To write to file
int encodeb64(unsigned char* input, const char* filenm, int leni)
{
BIO *b64 = BIO_new(BIO_f_base64());
BIO_set_flags(b64,BIO_FLAGS_BASE64_NO_NL);
BIO *file = BIO_new_file(filenm, "w");
BIO *mem = BIO_new(BIO_f_buffer());
BIO_push(b64, mem);
BIO_push(mem, file);
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, input, leni);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
BIO_flush(b64);
BIO_pop(b64);
BIO_free_all(b64);
return 0;
}
Base64 encoding from file to file. Many times due to file constraint we have read in chunks of data and do encoding. Below is the code.
int encodeb64FromFile(const char* input, const char* outputfilename)
{
BIO *b64 = BIO_new(BIO_f_base64());
BIO_set_flags(b64,BIO_FLAGS_BASE64_NO_NL);
int leni = 3*64;
unsigned char *data[3*64];
BIO *file = BIO_new_file(outputfilename, "w");
BIO *mem = BIO_new(BIO_f_buffer());
BIO_push(b64, mem);
BIO_push(mem, file);
FILE *fp = fopen(input, "rb");
while ((leni = fread(data, 1, sizeof data, fp)) > 0) {
// write data
bool done = false;
int res = 0;
while(!done)
{
res = BIO_write(b64, data, leni);
if(res <= 0) // if failed
{
if(BIO_should_retry(b64)){
continue;
}
else // encoding failed
{
/* Handle Error!!! */
}
}
else // success!
done = true;
}
}
BIO_flush(b64);
BIO_pop(b64);
BIO_free_all(b64);
fclose(fp);
return 0;
}
Base64 is really pretty simple; you shouldn't have trouble finding any number of implementations via a quick Google. For example here is a reference implementation in C from the Internet Software Consortium, with detailed comments explaining the process.
The openssl implementation layers a lot of complexity with the "BIO" stuff that's not (IMHO) very useful if all you're doing is decoding/encoding.
#include <openssl/bio.h>
typedef unsigned char byte;
namespace base64 {
static void Encode(const byte* in, size_t in_len,
char** out, size_t* out_len) {
BIO *buff, *b64f;
BUF_MEM *ptr;
b64f = BIO_new(BIO_f_base64());
buff = BIO_new(BIO_s_mem());
buff = BIO_push(b64f, buff);
BIO_set_flags(buff, BIO_FLAGS_BASE64_NO_NL);
BIO_set_close(buff, BIO_CLOSE);
BIO_write(buff, in, in_len);
BIO_flush(buff);
BIO_get_mem_ptr(buff, &ptr);
(*out_len) = ptr->length;
(*out) = (char *) malloc(((*out_len) + 1) * sizeof(char));
memcpy(*out, ptr->data, (*out_len));
(*out)[(*out_len)] = '\0';
BIO_free_all(buff);
}
static void Decode(const char* in, size_t in_len,
byte** out, size_t* out_len) {
BIO *buff, *b64f;
b64f = BIO_new(BIO_f_base64());
buff = BIO_new_mem_buf((void *)in, in_len);
buff = BIO_push(b64f, buff);
(*out) = (byte *) malloc(in_len * sizeof(char));
BIO_set_flags(buff, BIO_FLAGS_BASE64_NO_NL);
BIO_set_close(buff, BIO_CLOSE);
(*out_len) = BIO_read(buff, (*out), in_len);
(*out) = (byte *) realloc((void *)(*out), ((*out_len) + 1) * sizeof(byte));
(*out)[(*out_len)] = '\0';
BIO_free_all(buff);
}
}
Late to the party, but I came across this problem recently myself, but was unhappy with both the BIO solution, which is unnecessarily convoluted, but did not like 'EncodeBlock' either, because it introduces newline characters I do not want in my Base64 encoded string.
After a little sniffing, I came across the header file openssl/include/crypto/evp.h which is not part of the default installation (which only exports the include/openssl folder for me), but exports the solution to the problem.
void evp_encode_ctx_set_flags(EVP_ENCODE_CTX *ctx, unsigned int flags);
/* EVP_ENCODE_CTX flags */
/* Don't generate new lines when encoding */
#define EVP_ENCODE_CTX_NO_NEWLINES 1
/* Use the SRP base64 alphabet instead of the standard one */
#define EVP_ENCODE_CTX_USE_SRP_ALPHABET 2
Using this function, the 'no newline' becomes possible using the EVP interface.
Example:
if (EVP_ENCODE_CTX *context = EVP_ENCODE_CTX_new())
{
EVP_EncodeInit(context);
evp_encode_ctx_set_flags(context, EVP_ENCODE_CTX_NO_NEWLINES);
while (hasData())
{
uint8_t *data;
int32_t length = fetchData(&data);
int32_t size = (((EVP_ENCODE_CTX_num(context) + length)/48) * 65) + 1;
uint8_t buffer[size];
EVP_EncodeUpdate(context, buffer, &size, pData, length);
//process encoded data.
}
uint8_t buffer[65];
int32_t writtenBytes;
EVP_EncodeFinal(context, buffer, &writtenBytes);
//Do something with the final remainder of the encoded string.
EVP_ENCODE_CTX_free(context);
}
This piece of code will encode the buffer to Base64 without the newlines.
Please note the use of EVP_ENCODE_CTX_num to obtain the 'leftover bytes' still stored in the context object to calculate the correct buffer size.
It is only necessary, if you need to call EVP_EncodeUpdate multiple times, because your data is exceedingly large or not available at once.