I'm new to flatbuffer and I want to know if it's possible to get full (not const*) access to data in flatbuffers::Vector. Looking at the example below, I want to steal the ownership of img2::mem::data to store it in an Img-struct and process it in any way I want. Is this somehow possible, without memcopy-ing?
struct Img
{
int iXCount;
int iYCount;
int iXOffset;
unsigned char *mem;
};
int _tmain(int argc, _TCHAR* argv[])
{
Img img;
//init img;
flatbuffers::FlatBufferBuilder fbb;
auto mem = fbb.CreateVector(img.mem, img.iXOffset * img.iYCount);
auto mloc = CreateImage(fbb, img.iXCount, img.iYCount, img.iXOffset, mem);
fbb.Finish(mloc);
//unpack
auto img2 = flatbuffers::GetRoot<Image>(fbb.GetBufferPointer());
const std::uint8_t*pMem = img2->mem()->data(); //unfortunately only const*
return 0;
}
Your pMem points to data that sits somewhere in the middle of the FlatBuffer you're using. So this means you can access it, but only as long as you can keep the parent buffer around.
Since these are bytes, you could const-cast them, and modify them in place without copying. Note that if you ever try this with things that are not bytes, you have to be aware that data in a FlatBuffer is always little-endian.
Alternatively to const-cast, you can compile your schema with --gen-mutable, which will give you additional accessors to modify data in-place starting from GetMutableRoot, and data() will also be non-const from that.
Related
I am trying to use the PEM_read_bio function to get data from a file.
The version of SSLeay we are using is from 1997, so documentation is a bit thin on the ground. Thankfully in this case it seems there is a matching function documented here: https://www.openssl.org/docs/man1.1.0/crypto/PEM_read_bio.html
I originally tried this:
char ** names;
char ** headers;
unsigned char ** data;
long len;
BIO *in = BIO_new_file("C:\\filename.txt", "r");
if (!in)
{
// error
}
else
{
int result = PEM_read_bio(in, names, headers, data, &len);
}
BIO_free(in);
OPENSSL_free(names);
OPENSSL_free(headers);
OPENSSL_free(data);
However this results in a run-time check failure: The variable 'names' is being used without being initialized.
The documentation mentions OPENSSL_malloc( num ) is used to initialize memory, but it fails to mention whether it does this behind the scenes, or the user does it.
OPENSSL_malloc is similar in usage to C's malloc, but how are we supposed to know how much memory to allocate in advance, before reading the file?
I have tried the following at the beginning:
char ** names = reinterpret_cast<char **>(OPENSSL_malloc(2));
char ** headers = reinterpret_cast<char **>(OPENSSL_malloc(2));
unsigned char ** data = reinterpret_cast<unsigned char **>(OPENSSL_malloc(2));
long len;
This results in apparently random data.
The documentation you linked to says:
The name, header and data pointers are allocated via OPENSSL_malloc() and should be freed by the caller via OPENSSL_free() when no longer needed.
That means PEM_read_bio() calls OPENSSL_malloc() for you, and then you call OPENSSL_free() on the allocated memory it returns when you are doing with it.
You are passing uninitialized pointers to PEM_read_bio(), that is why it is failing. The name, header and data parameters are all output parameters. You need to pass in the addresses of your own pointer variables to receive the memory that PEM_read_bio() allocates for you, eg:
char *name;
char *headers;
unsigned char *data;
long len;
BIO *in = BIO_new_file("C:\\filename.txt", "r");
if (!in)
{
// error
}
else
{
int result = PEM_read_bio(in, &name, &headers, &data, &len);
if (!result)
{
// error
}
else
{
...
OPENSSL_free(name);
OPENSSL_free(headers);
OPENSSL_free(data);
}
BIO_free(in);
}
The initial problem is that I have some data to save and retrieve from plate and would like to use a helper struct in the following way:
// this is how I want to use my struct for saving
void safeData()
{
Data data;
data.name = getNameToSave(); // returns const char*
save(data);
}
// ... and in the same way for writing.
void readData()
{
Data data;
read(data);
use(data);
}
... whereas
struct Data
{
const char* name;
// potentially many more data members
}
I'm bound to (const) char* instead of std::string for now because this should be used with legacy code. Unfortunately this will bring up a constness problem:
void read(Data& data)
{
// initial legacy free
free(data.name); // warning: can't convert from const char* to void*
// fill with some data
data.name = getNameFromPlate();
}
So the thing is this: Since my data retrieval methods for reading like getNameFromSomewhere returns const char* I'm bound to making my struct members const, too. This however clashes with write-usage because in this case they must be non-const. What is the best and cleanest way to work around this?
The cleanest way is for the Data struct to own the name data. So when you assign to it you should allocate memory and do a memcpy. This is the only way to make sure that a) you don't leak memory b) the data is not freed before you are done with it and c) data is not changed by other processes in the mean time.
Since you do a copy anyway, you can probably make it more manageable by using std::string instead of const char*. When you assign for const char*, std::string will do the copy for you and you can get a const char* back with c_str(), so that you can still work with the legacy architecture. You don't have to do the switch but will make it easier to manage in the long run.
You have not only a problem of constness, but also a problem of ownership. Except if the API states that the return value of getNameToSave() has to be freed by caller after use, you should not free it. While when you read it from a file, you must free it.
So the simpler way is to always use a local copy that you consistently free after use. As you are required to use old const char * instead of std::string you'd better keep using the good old C library function. Here the best function is strdup that automagically allocate memory and copy old string into it.
An alternative way would be to store the status of the member Data::name in a boolean member :
struct Data
{
const char* name;
bool dyn; // must be freed if true
// potentially many more data members
}
You can then safely use:
if (data.dyn && (data.name != nullptr)) {
free(const_cast<char *>(data.name);
}
I have a program that employs an entity-component-system framework. Essentially this means that I have a collection of entities that have various components attached to them. Entities are actually just integer ID numbers, and components are attached to them by mapping the component to the specified ID number of the entity.
Now, I need to store collections of entities and the associated components to a file that can be modified later on, so basically I need a saving and loading functionality. However, being somewhat a newcomer to C++, I have hard time figuring out how to exactly do this.
Coming from Java and C#, my first choice would be to serialize the objects into, say, JSON, and then deserialize them when the JSON is loaded. However, C++ does not have any reflection features. So, the question is: how do I save and load C++ objects? I don't mean the actual file operations, I mean the way the objects and structs should be handled in order to preserve them between program launches.
One way of doing is to create Persistent Objects in C++, and store the your data.
check out the following links:
C++ object persistence library similar to eternity
http://sourceforge.net/projects/litesql/
http://en.wikipedia.org/wiki/ODB_(C%2B%2B)
http://drdobbs.com/cpp/184408893
http://tools.devshed.com/c/a/Web-Development/C-Programming-Persistence/
C++ doesn't support persistence directly (there are proposals for adding persistence and reflection to C++ in the future). Persistence support is not as trivial as it may seem at first. The size and memory layout of the same object may vary from one platform to another. Different byte ordering, or endian-ness, complicate matters even further. To make an object persistent, we have to reserve its state in a non-volatile storage device. ie: Write a persistent object to retain its state outside the scope of the program in which it was created.
Other Way, is to store the objects into an array, then push the array buffer to a file.
The advantage are that the disk platters don't have waste time ramping up and also the writing can be performed contiguously.
You can increase the performance by using threads. Dump the objects to a buffer, once done trigger a thread to handle the output.
Example:
The following code has not been compiled and is for illustrative purposes only.
#include <fstream>
#include <algorithm>
using std::ofstream;
using std::fill;
#define MAX_DATA_LEN 1024 // Assuming max size of data be 1024
class stream_interface
{
virtual void load_from_buffer(const unsigned char *& buf_ptr) = 0;
virtual size_t size_on_stream(void) const = 0;
virtual void store_to_buffer(unsigned char *& buf_ptr) const = 0;
};
struct Component
: public stream_interface,
data_length(MAX_DATA_LEN)
{
unsigned int entity;
std::string data;
const unsigned int data_length;
void load_from_buffer(const unsigned char *& buf_ptr)
{
entity = *((unsigned int *) buf_ptr);
buf_ptr += sizeof(unsigned int);
data = std::string((char *) buf_ptr);
buf_ptr += data_length;
return;
}
size_t size_on_stream(void) const
{
return sizeof(unsigned int) + data_length;
}
void store_to_buffer(unsigned char *& buf_ptr) const
{
*((unsigned int *) buf_ptr) = entity;
buf_ptr += sizeof(unsigned int);
std::fill(buf_ptr, 0, data_length);
strncpy((char *) buf_ptr, data.c_str(), data_length);
buf_ptr += data_length;
return;
}
};
int main(void)
{
Component c1;
c1.data = "Some Data";
c1.entity = 5;
ofstream data_file("ComponentList.bin", std::ios::binary);
// Determine size of buffer
size_t buffer_size = c1.size_on_stream();
// Allocate the buffer
unsigned char * buffer = new unsigned char [buffer_size];
unsigned char * buf_ptr = buffer;
// Write / store the object into the buffer.
c1.store_to_buffer(buf_ptr);
// Write the buffer to the file / stream.
data_file.write((char *) buffer, buffer_size);
data_file.close();
delete [] buffer;
return 0;
}
Consider a typical function that fills in a buffer:
const char* fillMyBuffer( const char* buf, int size );
Suppose this function fills the buffer with some useful data, that I want to use almost immediately after the call, and then I want to get rid of the buffer.
An efficient way of doing this is to allocate on the stack:
doStuff();
{
char myBuf[BUF_LEN];
const char* pBuf = fillMyBuffer( myBuf, BUF_LEN );
processBuffer( pBuf );
}
doOtherStuff();
So this is great for my library because the buffer is allocated on the stack - being essentially no cost to allocate, use and discard. It lasts the entire scope of the containing braces.
But I have a library where I do this pattern all the time. I'd like to automate this a little. Ideally I'd like code that looks like this:
doStuff();
{
// tricky - the returned buffer lasts the entire scope of the braces.
const char* pBuf = fillMyBufferLocal();
processBuffer( pBuf );
}
doOtherStuff();
But how to achieve this?
I did the following, which seems to work, but I know is counter to the standard:
class localBuf
{
public:
operator char* () { return &mBuf[0]; }
char mBuf[BUF_LEN];
};
#define fillMyBufferLocal() fillMyBuffer( localBuf(), BUF_LEN );
As a practical matter, the buffer is lasting on the stack during the entire lifetime of the containing braces. But the standard says that the object only has to last until the function returns. E.g. technically its just as unsafe as if I'd allocated the buffer on the stack inside the function.
Is there a safe way to achieve this?
I would generally recommend your original solution. It separates the allocation of the buffer from filling it. However, if you want to implement this fillMyBufferLocal alternative, it will have to dynamically allocate the buffer and return a pointer to it. Of course, if you return a raw pointer to dynamically allocated memory, it's very unclear that the memory should later be destroyed. Instead, return a smart pointer that encapsulates the appropriate ownership:
std::unique_ptr<char[]> fillMyBufferLocal()
{
std::unique_ptr<char[]> buffer(new char[BUF_LEN]);
// Fill it
return buffer;
}
Then you can use it like so:
auto buffer = fillMyBufferLocal();
processBuffer(buffer.get());
I do not think you should want to do this. It just makes the code harder to understand.
Automatic storage duration means that when an object goes out of scope, it is destroyed. Here you want trick the system into something that behaves like creating an object with automatic storage duration (i.e. allocates on the stack), but without respecting the corresponding rules (i.e. without being destroyed when returning from fillMyBuffer()).
The closest, meaningful thing you can do in my opinion is to use a global buffer that fillMyBuffer() can reuse, or let that buffer be a static variable inside fillMyBuffer(). For instance:
template<int BUF_LEN = 255>
const char* fill_my_buffer()
{
static char myBuf[BUF_LEN];
// Fill...
return myBuf;
}
However, I strongly suggest reconsidering your requirements, and either:
Keep using the solution you are currently adopting (i.e. transparently allocate on the stack); or
Allocate the buffer dynamically inside fillMyBuffer() and return a RAII wrapper (like a unique_ptr) to this dynamically allocated buffer.
UPDATE:
As a last, desperate attempt, you could define a macro that does the allocation and the invocation of fill_my_buffer() for you:
#define PREPARE_BUFFER(B, S) \
char buffer[S]; \
const char* B = fill_my_buffer(buffer, S);
You would then use it this way:
PREPARE_BUFFER(pBuf, 256);
processBuffer(pBuf);
You could write a class that contains a stack-based buffer and converts to char const *, e.g.
void processBuffer(char const * buffer);
char const * fillMyBuffer(char const * buffer, int size);
int const BUF_LEN = 123;
class Wrapper
{
public:
Wrapper(char const * (*fill)(char const *, int))
{
fill(&m_buffer[0], m_buffer.size());
}
operator char const * () const { return &m_buffer[0]; }
private:
std::array<char, BUF_LEN> m_buffer;
};
void foo()
{
Wrapper wrapper(fillMyBuffer);
processBuffer(wrapper);
}
I'm semi new to c++. I have tried everything but couldn't solve my problem. Just for some background: I'm using the Botan library for encryption. I don't think this problem has anything to do with the library, but more with pointers and objects.
When I use the following code, there are no problems. The comments after each line explains what happens. (The code is for explanation purposes only)
int main(int argc, char** argv)
{
DH_PrivateKey *Apriv = 0; // Apriv points to 0x00
MemoryVector<unsigned char> *Apub = 0; // Irrelevant for now
AutoSeeded_RNG rng; // Irrelevant object
DL_Group domain("modp/ietf/3072"); // Irrelevant object
Apriv = new DH_PrivateKey(rng,domain); // Apriv points to 0x8079098
The main observation here is that an object is instantiated and Apriv points to that object. This is what I want to happen. The problem comes in when I try to do this in another function which I pass the Apriv pointer to.
My main code changes to the following:
int main(int argc, char** argv)
{
DH_PrivateKey *Apriv = 0;
MemoryVector<unsigned char> *Apub = 0;
AutoSeeded_RNG rng;
DL_Group domain("modp/ietf/3072");
encryption::dh_generatekeys("modp/ietf/3072",Apriv,Apub); //Passing pointers
And the function I pass these pointers to looks like this:
void encryptionalgorithm::dh_generatekeys(string shareddomain,DH_PrivateKey *privatekey,MemoryVector<unsigned char> *publickey)
{
AutoSeeded_RNG rng; // Irrelevant
DL_Group shared_domain(shareddomain); // Irrelevant
privatekey = new DH_PrivateKey(rng, shared_domain); //[PROBLEM 1]
MemoryVector<unsigned char> pubkey = privatekey->public_value();
publickey = &pubkey;[PROBLEM 2]
I will now describe the problems:
Problem 1: The privatekey pointer still points to 0x00, it does not point to the newly instantiated object, and this later causes segmentation errors.
Problem 2: The same with publickey, when I create "pubkey" on the stack and then point publickey pointer to its address, publickey just stays 0x00.
Any help would be appreciated. I hope it is something simple!
Thanks guys.
Your parameters are passed by value.
This means, if you modify the value of the pointer privatekey inside a function, you only modify a variable local to that function.
You might want to change your function signature to accept parameters as references instead:
void generatekeys(Something *& Apub, Something *& Apriv) { ... }
This reads right-to-left: Apub is a reference & to a pointer * to Something. Now any changes to Apub and Apriv inside generatekeys will propagate to the outside as you want, but note that you'll need to call that function passing proper variables (lvalues), not just some expressions (rvalues).
If in doubt, ask further or have a google for reference parameters.
Your problem is that dh_generatekeys takes both privatekey and publickey by value (that is the value of the pointer). Therefore what you modify are local copies of the pointers, instead of the pointers instead. The solution is simply to either pass a pointer or a reference to the pointer to your object, so:
void encryptionalgorithm::dh_generatekeys(string shareddomain,DH_PrivateKey** privatekey, MemoryVector<unsigned char>** publickey)
or
void encryptionalgorithm::dh_generatekeys(string shareddomain,DH_PrivateKey*& privatekey, MemoryVector<unsigned char>*& publickey)
If I understand correctly, yo ushould change dh_generatekeys to:
void encryptionalgorithm::dh_generatekeys(string shareddomain,DH_PrivateKey* &privatekey,MemoryVector<unsigned char>* &publickey) { ... }
That is to say use references to pointers. This way you can change them.
You need to pass the address of the pointers (or a reference) to be able to modify the value in the caller.
Address of pointer (pointer to pointer) example:
int main(int argc, char** argv)
{
DH_PrivateKey *Apriv = 0;
MemoryVector<unsigned char> *Apub = 0;
AutoSeeded_RNG rng;
DL_Group domain("modp/ietf/3072");
encryption::dh_generatekeys("modp/ietf/3072",&Apriv,&Apub); //Passing address of pointers
//...
}
void encryptionalgorithm::dh_generatekeys(string shareddomain,DH_PrivateKey **privatekey,MemoryVector<unsigned char> **publickey)
{
AutoSeeded_RNG rng; // Irrelevant
DL_Group shared_domain(shareddomain); // Irrelevant
*privatekey = new DH_PrivateKey(rng, shared_domain); //dereference to assign
MemoryVector<unsigned char> pubkey = privatekey->public_value();
*publickey = &pubkey; //dereference to assign
//...
}