Hokay so I have an application where I need some IPC... I'm thinking named pipes are the way to go because they are so easy to use.
Anyways, I have a question about how to handle dynamic memory using named pipes.
Say I have a class such as this:
class MyTestClass {
public:
MyTestClass() { _data = new int(4); }
int GetData() { return *_data; }
int GetData2() { return _data2; }
private:
int* _data;
int _data2;
};
Now when I create a buffer full of MyTestClass objects then send them over the pipe, I'm obviously losing _data in the destination process and getting garbage. Is there some strategy to this that I should use? I can use value types for simple cases, but for many complex classes I need to use some sort of dynamic memory and I like pointers.
Or, should I just look at using shared memory instead? Thanks
Both named pipes and shared memory have similar issues: You need to serialize the contents of the structure into the on the sending side and deserialize the structure from the on the receiving side.
The serialization process is essentially identical whether you're using named pipes or shared memory. For embedded pointers (like _data and _data2) you need to serialize the contents of the pointer in a consistent fashion.
There are lots of serialization strategies that you could use, depending on how your structures are laid out in memory and how efficient your IPC has to be. Or you could use DCE RPC and let the RPC marshaling code handle the complexities for you.
To send the data over a named pipe, you must serialize (or marshal) the data on the sending end, and deserialize (or unmarshal) it on the receiving end.
It sounds suspiciously as if you are simply writing a copy of the bytes in the data structure. This is no good whatsoever. You aren't copying the allocated data (it isn't stored between the first and last bytes of the data structure, but somewhere else altogether) and you are copying a pointer (_data) from one machine (or process) to another, and the memory address in the local process has no guaranteed meaning in the other.
Define yourself a wire protocol (if desparate, look at ASN.1 - no, on second thoughts, don't get that desparate) that defines the layout of the data for transmission over the wire. Then implement sender and receiver (or seralizer and deserializer) functions. Or find someone else's code that does this already.
Also remember to deal with endian-ness - you must define which sequence the bytes are sent through the named pipe.
For example, you could define that the message sent consists of a 4-byte unsigned integer in network byte order defining how many structures follow, and each structure could be a sequence of 4 signed 4-byte integers for the array followed by a single signed 4-byte integer for _data2 (also sent in network byte order).
Note that the choice of named pipe as the IPC mechanism is largely immaterial; unless you are using shared memory (inherently on the same machine), then endian-ness has to be dealt with, and even with shared memory, you need to deal with serialization.
Related
This is something the professor showed us in his scripts. I have not used this method in any code I have written.
Basically, we take a class, or struct, and reinterpret_cast it and save off the entire struct like so:
struct Account
{
Account()
{ }
Account(std::string one, std::string two)
: login_(one), pass_(two)
{ }
private:
std::string login_;
std::string pass_;
};
int main()
{
Account *acc = new Account("Christian", "abc123");
std::ofstream out("File.txt", std::ios::binary);
out.write(reinterpret_cast<char*>(acc), sizeof(Account));
out.close();
This produces the output (in the file)
ÍÍÍÍChristian ÍÍÍÍÍÍ ÍÍÍÍabc123 ÍÍÍÍÍÍÍÍÍ
I'm confused. Does this method actually work, or does it cause UB because magical things happen within classes and structs that are at the whims of individual compilers?
It doesn't actually work, but it also does not cause undefined behavior.
In C++ it is legal to reinterpret any object as an array of char, so there is no undefined behavior here.
The results, however, are usually only usable if the class is POD (effectively, if the class is a simple C-style struct) and self-contained (that is, the struct doesn't have pointer data members).
Here, Account is not POD because it has std::string members. The internals of std::string are implementation-defined, but it is not POD and it usually has pointers that refer to some heap-allocated block where the actual string is stored (in your specific example, the implementation is using a small-string optimization where the value of the string is stored in the std::string object itself).
There are a few issues:
You aren't always going to get the results you expect. If you had a longer string, the std::string would use a buffer allocated on the heap to store the string and so you will end up just serializing the pointer, not the pointed-to string.
You can't actually use the data you've serialized here. You can't just reinterpret the data as an Account and expect it to work, because the std::string constructors would not get called.
In short, you cannot use this approach for serializing complex data structures.
It's not undefined. Rather, it's platform dependent or implementation defined behavior. This is, in general bad code, because differing versions of the same compiler, or even different switches on the same compiler, can break your save file format.
This could work depending on the contents of the struct, and the platform on which the data is read back. This is a risky, non-portable hack which your teacher should not be propagating.
Do you have pointers or ints in the struct? Pointers will be invalid in the new process when read back, and int format is not the same on all machines (to name but two show-stopping problems with this approach). Anything that's pointed to as part of an object graph will not be handled. Structure packing could be different on the target machine (32-bit vs 64-bit) or even due to compiler options changing on the same hardware, making sizeof(Account) unreliable as a read back data size.
For a better solution, look at a serialization library which handles those issues for you. Boost.Serialization is a good example.
Here, we use the term "serialization"
to mean the reversible deconstruction
of an arbitrary set of C++ data
structures to a sequence of bytes.
Such a system can be used to
reconstitute an equivalent structure
in another program context. Depending
on the context, this might used
implement object persistence, remote
parameter passing or other facility.
Google Protocol Buffers also works well for simple object hierarchies.
It's no substitute for proper serialization. Consider the case of any complex type that contains pointers - if you save the pointers to a file, when you load them up later, they're not going to point to anything meaningful.
Additionally, it's likely to break if the code changes, or even if it's recompiled with different compiler options.
So it's really only useful for short-term storage of simple types - and in doing so, it takes up way more space than necessary for that task.
This method, if it works at all, is far from robust. It is much better to decide on some "serialized" form, whether it is binary, text, XML, etc., and write that out.
The key here: You need a function/code to reliably convert your class or struct to/from a series of bytes. reinterpret_cast does not do this, as the exact bytes in memory used to represent the class or struct can change for things like padding, order of members, etc.
No.
In order for it to work, the structure must be a POD (plain old data: only simple data members and POD data members, no virtual functions... probably some other restrictions which I can't remember).
So if you wanted to do that, you'd need a struct like this:
struct Account {
char login[20];
char password[20];
};
Note that std::string's not a POD, so you'd need plain arrays.
Still, not a good approach for you. Keyword: "serialization" :).
Some version of string don;t actually use dynamic memory for the string when the string is small. Thus store the string internally in the string object.
Think of this:
struct SimpleString
{
char* begin; // beginning of string
char* end; // end of string
char* allocEnd; // end of allocated buffer end <= allocEnd
int* shareCount; // String are usually copy on write
// as a result you need to track the number of people
// using this buffer
};
Now on a 64 bit system. Each pointer is 8 bytes. Thus a string of less than 32 bytes could fit into the same structure without allocating a buffer.
struct CompressedString
{
char buffer[sizeof(SimpleString)];
};
stuct OptString
{
int type; // Normal /Compressed
union
{
SimpleString simple;
CompressedString compressed;
}
};
So this is what I believe is happening above.
A very efficient string implementation is being used thus allowing you to dump the object to file without worrying about pointers (as the std::string are not using pointers).
Obviously this is not portable as it depends on the implementation details of std::string.
So interesting trick, but not portable (and liable to break easily without some compile time checks).
I have a situation where I need to send an object (which has pointers to dynamically allocated objects) from one process to anther process via IPC.
Blueprint of the class is:
class Example
{
std::string str;
char *cstring;
int value;
};
As you can see it has dynamically allocated data members that points to heap area mapped into that process.
If it was static object members then they are mapped in continuous memory allocation, and we can mem copy that object, right ? May be I am wrong or missing something.
Can we copy this entire object and send it over IPC, using any of the IPC mechanisms ?
Hope anyone of you can share some details how it will work or can be made to work by pointing me in right direction.
Thanks
Generally you need to serialise it into a sequential byte buffer, similar to how you might write or read say a file or over a network/internet. There are various libraries to help you with this, either with binary formats, or structured text like JSON or XML. Some may generate the boilerplate code for you to convert to and from structs or classes.
If you are using shared memory at the same virtual address, you could possibly take advantage of custom allocators to make sure STL types and other things store everything also in shared memory. I am not sure if the standard says anything much on if such use of shared memory for standard defined objects is technically compliant. I would expect string, vector, etc. to work, while things like mutex, fstream, etc. won't.
Let's say I have a process written in C++ that has many instances of class A.
class A {
std::vector<std::vector<short>> tab;
/* other random data */
}
Once created, these objects should be accessed (read-only) by other processes of a different application, also in C++.
I would like to avoid at all cost making copies or moving the objects as it would significantly increase the memory consumption and probably take more time.
A 'simple' portable solution would be to serialize the objects in a shared memory and then when asked for the data, the process would just give the location of the various instances of class A in memory, and the second process would deserialize them before being able to read the data.
That means that we would create a copy each time a process wants to read the data. This is what I would like to avoid.
Given that both processes are written in C++ and both know the definition of the class A, is it possible to avoid the serialization and therefore copy/displacement of the data ? Of course it would not be portable anymore but it does not need to be.
Can a simple static_cast through the shared memory allow the second process to read the data in it as its own without making any processing of any kind therefore costing no time and no additional memory at all ?
If not, is there a simpler form of serialization adding just an overhead that would allow the second process to understand and read the data without having to make a copy ?
I want to use pointer magikry to save a C++ class using the following method that writes byte data into a file:
result Osp::Io::File::Write (const void *buffer, int length);
Parameters:
buffer — A pointer to the user-supplied buffer that contains byte data to be written
length — The buffer length in bytes
Exceptions:
E_SUCCESS — The method is successful.
E_INVALID_STATE — The file has not been opened as yet.
E_ILLEGAL_ACCESS — The file is not opened for write operation, or access is denied due to insufficient permission.
E_INVALID_ARG — Either of the following conditions has occurred:
The specified buffer contains a null pointer.
The specified buffer length is equal or smaller than 0.
The file handle is invalid (either the file is closed by another method, or the memory is corrupted).
E_STORAGE_FULL — The disk space is full.
E_IO — An unexpected device failure has occurred as the media ejected suddenly or file corruption is detected.
I'd rather not assume that there will be any sort of buffering, although I am confident each byte won't occasion a whole block of flash to be rewritten but I was wondering if there is a niftier way to write all the data fields of a class (and nothing else, eg static fields) by, eg, a pointer to the object (*this)?
In C++ you don't write "raw" objects into files, but rather serialize them. There's no magic, you need to write your serialization code yourself (overloading operators << and >>, for convenience).
You can do it the old C-style by just dumping memory, but in addition to the problems this would generally cause with C (alignment, endian issues when transferring data between systems), you also get the problems introduced by C++ (internal class representation, possible "hidden" data members such as a v-table, etc).
If you want to ensure you read and write reliable data that can be transferred between different systems and/or different pieces of software - you better implement the serialization, and don't look for shortcuts.
You can use libraries like Boost.Serialization for that.
This is something the professor showed us in his scripts. I have not used this method in any code I have written.
Basically, we take a class, or struct, and reinterpret_cast it and save off the entire struct like so:
struct Account
{
Account()
{ }
Account(std::string one, std::string two)
: login_(one), pass_(two)
{ }
private:
std::string login_;
std::string pass_;
};
int main()
{
Account *acc = new Account("Christian", "abc123");
std::ofstream out("File.txt", std::ios::binary);
out.write(reinterpret_cast<char*>(acc), sizeof(Account));
out.close();
This produces the output (in the file)
ÍÍÍÍChristian ÍÍÍÍÍÍ ÍÍÍÍabc123 ÍÍÍÍÍÍÍÍÍ
I'm confused. Does this method actually work, or does it cause UB because magical things happen within classes and structs that are at the whims of individual compilers?
It doesn't actually work, but it also does not cause undefined behavior.
In C++ it is legal to reinterpret any object as an array of char, so there is no undefined behavior here.
The results, however, are usually only usable if the class is POD (effectively, if the class is a simple C-style struct) and self-contained (that is, the struct doesn't have pointer data members).
Here, Account is not POD because it has std::string members. The internals of std::string are implementation-defined, but it is not POD and it usually has pointers that refer to some heap-allocated block where the actual string is stored (in your specific example, the implementation is using a small-string optimization where the value of the string is stored in the std::string object itself).
There are a few issues:
You aren't always going to get the results you expect. If you had a longer string, the std::string would use a buffer allocated on the heap to store the string and so you will end up just serializing the pointer, not the pointed-to string.
You can't actually use the data you've serialized here. You can't just reinterpret the data as an Account and expect it to work, because the std::string constructors would not get called.
In short, you cannot use this approach for serializing complex data structures.
It's not undefined. Rather, it's platform dependent or implementation defined behavior. This is, in general bad code, because differing versions of the same compiler, or even different switches on the same compiler, can break your save file format.
This could work depending on the contents of the struct, and the platform on which the data is read back. This is a risky, non-portable hack which your teacher should not be propagating.
Do you have pointers or ints in the struct? Pointers will be invalid in the new process when read back, and int format is not the same on all machines (to name but two show-stopping problems with this approach). Anything that's pointed to as part of an object graph will not be handled. Structure packing could be different on the target machine (32-bit vs 64-bit) or even due to compiler options changing on the same hardware, making sizeof(Account) unreliable as a read back data size.
For a better solution, look at a serialization library which handles those issues for you. Boost.Serialization is a good example.
Here, we use the term "serialization"
to mean the reversible deconstruction
of an arbitrary set of C++ data
structures to a sequence of bytes.
Such a system can be used to
reconstitute an equivalent structure
in another program context. Depending
on the context, this might used
implement object persistence, remote
parameter passing or other facility.
Google Protocol Buffers also works well for simple object hierarchies.
It's no substitute for proper serialization. Consider the case of any complex type that contains pointers - if you save the pointers to a file, when you load them up later, they're not going to point to anything meaningful.
Additionally, it's likely to break if the code changes, or even if it's recompiled with different compiler options.
So it's really only useful for short-term storage of simple types - and in doing so, it takes up way more space than necessary for that task.
This method, if it works at all, is far from robust. It is much better to decide on some "serialized" form, whether it is binary, text, XML, etc., and write that out.
The key here: You need a function/code to reliably convert your class or struct to/from a series of bytes. reinterpret_cast does not do this, as the exact bytes in memory used to represent the class or struct can change for things like padding, order of members, etc.
No.
In order for it to work, the structure must be a POD (plain old data: only simple data members and POD data members, no virtual functions... probably some other restrictions which I can't remember).
So if you wanted to do that, you'd need a struct like this:
struct Account {
char login[20];
char password[20];
};
Note that std::string's not a POD, so you'd need plain arrays.
Still, not a good approach for you. Keyword: "serialization" :).
Some version of string don;t actually use dynamic memory for the string when the string is small. Thus store the string internally in the string object.
Think of this:
struct SimpleString
{
char* begin; // beginning of string
char* end; // end of string
char* allocEnd; // end of allocated buffer end <= allocEnd
int* shareCount; // String are usually copy on write
// as a result you need to track the number of people
// using this buffer
};
Now on a 64 bit system. Each pointer is 8 bytes. Thus a string of less than 32 bytes could fit into the same structure without allocating a buffer.
struct CompressedString
{
char buffer[sizeof(SimpleString)];
};
stuct OptString
{
int type; // Normal /Compressed
union
{
SimpleString simple;
CompressedString compressed;
}
};
So this is what I believe is happening above.
A very efficient string implementation is being used thus allowing you to dump the object to file without worrying about pointers (as the std::string are not using pointers).
Obviously this is not portable as it depends on the implementation details of std::string.
So interesting trick, but not portable (and liable to break easily without some compile time checks).