Best way to "mangle" (represent) the memory - c++

I would like to know what would be the best way to map/represent the memory. I mean, how to describe, for example, a structure with all its field to be serialized.
I am creating a RPC library that will create the client and server using the dwarf debug data, so i need to create a function wrapper to serialize and deserialize the functions´s parameters.
Now, i am using the gcc mangling types to identify all the fields, but the compiler sometimes creates holes to optimize the memory access time;
Any idea ?

I use the "cereal" library for serialization (http://uscilab.github.io/cereal/)
Alternatives include Google's Protocol Buffers, although I found it too difficult to integrate for my comparably simple serialization tasks.
For communication between processes, and languages, I've had a good experience with ZeroC's ICE library (https://zeroc.com/products/ice). You specific the structure as an external compilation step similar to Google's Protocol Buffers. The nice part is that the network connect was also taken care off.

Related

How to best expose ocaml library to other languages?

There are various exchange languages - json, ect - that provide an ability to quickly and reliably export and parse data to a common format. This is a boon between languages, and for it there is Piqi, which basically generates parsable exchange formats for any type that you define; it automates the process of writing boiler code (writing functions that read in some exchange info and build up a instance of some arbitrary type). Basically, the best option to date is protocol buffers, and I absolutely want, if I go down the route of ocaml-rpc, to use protocol buffers.
It would be nice if there were some declarative pattern to manage function exposure, so that the ocaml library can be reached over some medium (like RPC, or map a function to a url with encoding for arguments).
Imagine offering a library as a service; where you don't want to or can't make actual bindings between every single pair of languages. But servers and the data parsing has already been written... so wouldn't there be some way to integrate the two, and just specify what functions should be exposed and where/how?
Lastly, it appears to me that protocol buffers is a mechanism by which you can encode/decode data quickly, but not a transport mechanism... is there some kind of ocaml-RPC spec or some ocaml RPC library? Aren't there various RPC protocols (and ergo, if I try to point two languages using diff protocols at one another, achieve failure)? Additionally, the server mechanism that waits and receives RPC calls is (possibly) another module(?)
How do I achieve this?
To update this, the latest efforts under the piqi project are aimed at producing a working OCaml RPC service. From this, it would be, in vision, easy to specify what functions to expose at the RPC service end, and target function selection on the client side should allow for some mechanized facility to allow those exposed functions to be selected.
At the current time, this RPC system for ocaml facilitates inter-language exchange of data that can be reconstructed by parsers through the use of proto-buffers; it is under development and still being discussed here
I think that ocaml-rpc library suits your requirements. It can infer serialization functions and, also, can generate client and server code. The interesting part, is that they use OCaml as a IDL language. For example, this is a definition of the rpc function:
external rpc2 : ?opt:string -> variant -> unit = ""
From which there will be inferred server and client functorized code, that will take care on transporting marshaling and demarshaling the data, so that you need to work only with pure OCaml data types.
The problem with this library is that it is barely documented, so you may find it hard to use.
Also, as now I know, that you're tackling with BAP, I would like to bring your attention to a new BAP 1.x, that will be ready soon, and it will have bindings, that will allow to call it from any language, although currently we're mostly targeting python.

Serialize/ Deserialize C++ classes

I'm looking for a way to send C++ class between 2 clients aptication.
I was looking for a way doing so and all i can find is that I need to create for each Class Serialize/ Deserialize (to JSON for example) functions and send it over TCP/IP.
The main problem I'm faceing is that I have ~600 classes (some are classes including instances of others) that I need to pass which mean I need to spent the next writing Serialize/ Deserialize functions.
Is there any generic way writing Serialize/Deserialize functions ?
Is there any other way sending C++ classes ?
Thanks,
Guy Ergas.
Are you using a Framework at all? Qt and MFC for example have built in Serialization that would make your task easier. Otherwise I would guess that you'd need to spend at least some effort on each of the 600 classes.
As recommended above Boost Serialization is probably a good way to go, you can send the serialized class over Tcp using Boost Asio too:
http://www.boost.org/doc/libs/1_54_0/doc/html/boost_asio.html
Alternatively, there is a C++ API for Google Protocol Buffers (protobuf):
https://developers.google.com/protocol-buffers/docs/reference/cpp/
Boost Serialization
Although I haven't used it my self, it is very popular around my peers at work.
More info about it can be found in "Boost (1.54.00) Serialization"
Thrift
Thrift have a very limited serialize functionality which I don't think fits your requirements. But it can help you "move" the data from one client to anther even if they are using different languages.
More info about it can be found in "Thrift: The Missing Guide"
try s11n or nosjob
s11n (an abbreviation for serialization) is an Open Source project
focused on the generic serialization of objects (i.e., object
persistence) in the C++ programming language.
nosjob, a C++ library for generating and consuming JSON data.
You may be interested in ASN.1. It's not necessarily the easiest to use and tools/libraries are a little hard to come by (Objective Systems at http://www.obj-sys.com/index.php is worth a look, though not free).
However the big advantage is that it is very heavily standardised (so no trouble with library version incompatibilities) and most languages are supported one way or another. Handy if you need support across multiple platforms. It also does binary encodings, so its way less bloaty than XML (which it also supports). I chose it for these reasons and didn't regret it.
If you are at linux platform , You can directly use json.h library for serialization.
Here is sample code i have come across :)
Json Serializer

Working with protobuf and POCOs in C++

I would like to use protobuf with a C++ project I'm working on.
However, I don't like to work with the auto-generated classes protoc creates and prefer to stick with the POCOs I already have. This is because the POCOs are already in use in other parts of the code and I want to be able to switch the serialization mechanism with ease later on. But manually writing converters between POCOs and protobuf message classes seems tedious and wrong.
I want to know if there's a way to use protobuf to create a serializer - an auto-generated class that will be able to serialize and deserialize my POCOs, without bugging me with internals.
Thanks.
First, you may like Cap'n Proto better, it was created by one of Google's former Google Protocol Buffer maintainers. Worth looking into, anyway.
But otherwise, you really need to consider why you're using Google Protocol Buffers.
If you want to achieve the forward and backward compatibility, and to be able to open, then edit, then save an object that possibly a different person created, with a different version of your protocol buffer declaration, and then sent along to yet another person with an even different version of the declaration... then you need to just bite the bullet and use the generated C++ from the Google Protocol Buffer Compiler.
It's really not just a serialization format. It's specifically designed to make it easy living with different versions of your serialization, over time.
If you don't need that flexibility, and you don't like the generated code, you may want to consider a different serialization tool.

Thrift or Protocol buffer as cross-language serialization solution?

I've already chosen to use thrift as RPC framework in a project. This project has a lot of serialization / deserialization operations (e.g., store the data to disks). And the serialized format should be accessible for at least C++/Java/Python. It seems that thrift's serialization solution is more complicated than Protobuf (e.g., it needs to create a protocol before serializing an object).
So my question is: is it worth to use Protobuf for the serialization / deserialization part even if thrift is capable of this task?
I would agree that Thrift is a better choice for cross language RPC than Protobuf RPC ( see http://pjklauser.wordpress.com/2013/02/27/why-googles-protobuf-rpc-will-not-reach-widespread-adoption/ ). If you're using thrift already it's difficult to justify using a different "library" for serialization to file/storage. You'll need to write endless mapping code. Both libraries will have different maintenance cycles which you need to maintain independently which will give extra future effort. The cost of writing a line or two more code, or save one or two bytes of space, or save a microsecond of CPU time will be nothing compared to your additional efforts.

What's a lightweight shared memory-based IPC mechanism in C/C++ for Windows?

I've been working on a few C++ projects now which involve doing some simple IPC using window messages. In a number of cases, some extra data is passed with the window messages by putting the data into a shared memory segment and then passing the pointer into the shared memory with the SendMessage call. Re-doing this all the time is annoying, but before resolving this dull repetition by inventing yet another IPC system I'd like to ask:
Is there an existing framework which satisfies the criteria?:
Written in C or C++ (we're using MSVC here)
As few dependencies as possible; in the best case, it's just a few source files which use plain C++ and Windows standard libraries and which can be compiled directly into the application/library.
Works on Windows XP and newer
Is built on window messages plus a shared memory segment
Proper error reporting would be highly desireable (remote process is gone, remote process doesn't understand given message, argument mismatch, etc.)
For what it's worth, COM is not really an option for us since it's so painful to work with it (unless you start using all kinds of wrappers on top of it which we'd like to avoid). I don't really need interfaces and all that stuff; just a convenient way to send messages with (in the best case arbitrary) arguments back and forth with a bit of error handling. Also, I discarded DBUS for doing so much more than I need.
I've had success using a memory mapped file for interprocess communication. I like it mainly because it's simple, fast, and available on any version of windows you're likely to come across (well, i doubt it will work on Win9x, but....)
There's a basic article at http://msdn.microsoft.com/en-us/library/ms810613.aspx (writtin in 1993!) that shows how to use them.
Although does not meet all of your criteria, ZeroMQ (http://www.zeromq.org/) might be worth looking at. It is small, simple and fast. Also it gives you message passing semantics which may help depending on the type of applications you are using
This question is almost 3 years old at the time of this answer, but I can't believe nobody has formally suggested Boost.Interprocess.
Light-weight, IPC, and C++ wrappings for WINAPI mechanisms are available.