I am trying to create C-style structs in Clojure, so I can call a poorly documented C++ API from Clojure.
The API is designed to send and receive serialized protobuf messages (the good) preceded by a C Header struct (the bad). The initial handshake is an RPCHandshakeHeader struct and the process is roughly described in the code below:
struct RPCHandshakeHeader {
char magic[8];
int version;
static const char REQUEST_MAGIC[9];
static const char RESPONSE_MAGIC[9];
};
[...snip...]
const char RPCHandshakeHeader::REQUEST_MAGIC[9] = "FooBar?\n";
[...snip...]
RPCHandshakeHeader header;
memcpy(header.magic, RPCHandshakeHeader::REQUEST_MAGIC, sizeof(header.magic));
header.version = 1;
socket = new CActiveSocket();
socket->Initialize();
socket->Open((const uint8 *)"localhost", 5000);
socket->Send((uint8*)&header, sizeof(header));
[...code to read response...]
How can I do this in clojure? Do I need to use JNA/JNI?
Is there a way to create a C struct, turn it into binary and send it over a socket? (I think this is what I need to do)
Sounds like a job for gloss! I don't know the details of this part of the API, but you want to look particularly at compile-frame, and repeated for the character strings.
Related
I would like to serialize/deserialize some structured data in order to send it over the network via a char* buffer.
More precisely, suppose I have a message of type struct Message.
struct Message {
Header header;
Address address;
size_t size; // size of data part
char* data;
} message
In C, I would use something such as:
size = sizeof(Header) + sizeof(Address) + sizeof(size_t) + message.size;
memcpy(buffer, (char *) message, size);
to serialize, and
Message m = (Message) buffer;
to deserialize.
What would be the "right" way to do it in C++. Is it better to define a class rather than a struct. Should I overload some operators? are there alignment issues to consider?
EDIT: thanks for pointing the "char *" problem. The provided C version is incorrect. The data section pointed to by the data field should be copied separately.
Actually there are many flavors:
You can boost let it do for you: http://www.boost.org/doc/libs/1_52_0/libs/serialization/doc/tutorial.html
Overloading the stream operators << for serialization and >> for deserialization works well with file and string streams
You could specify a constructor Message (const char*) for constructing from a char*.
I am a fan of static methods for deserialization like:
Message {
...
static bool desirialize (Message& dest, char* source);
}
since you could catch errors directly when deserializing.
And the version you proposed is ok, when applying the modifications in the comments are respected.
Why not insert a virtual 'NetworkSerializable' Class into your inheritance tree? A 'void NetSend(fd socket)' method would send stuff, (without exposing any private data), and 'int(bufferClass buffer)' could return -1 if no complete, valid message was deserilalized, or, if a valid message has been assembled, the number of unused chars in 'buffer'.
That encapsulates all the assembly/disassembly protocol state vars and other gunge inside the class, where it belongs. It also allows message/s to be assembled from multiple stream input buffers.
I'm not a fan of static methods. Protocol state data associated with deserialization should be per-instance, (thread-safety).
I am trying to pass data from an x64 app to a x86 app using named pipes and overlapped I/O like what is defined here:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365603(v=vs.85).aspx
My server application's call to WriteFileEx succeeds and the structure I am sending through the pipe seems ok, however when I call ReadFile on the client side the data structure I retrieve is corrupted or different to the data that I sent, but it also successfully reads.
My client application has a unicode character set and the server's character set is 'not set', which I assume defaults to multibyte. I'm not in a position to change the server's character set to unicode.
Would this data corruption just be because I need to convert from multibyte to wide char on the client after I retrieve / read the data structure? If so is there built in helper functions that I can call on do to that?
Data structure being sent (defined identically on the server and client):
typedef struct
{
int id;
float vertices[VERTICES_COUNT][VERTICES_COMPONENTS];
unsigned short indices[INDICES_COUNT];
float texCoords[TEXTURE_COORD_COUNT][TEXTURE_COORD_COMPONENT];
unsigned char texData[TEXTURE_SIZE];
} MESHINST, *LPMESHINST;
typedef struct
{
OVERLAPPED oOverlap;
HANDLE pipeInst;
int addedCount;
MESHINST meshes[MESH_GROUP_BUFFER];
int removedCount;
int removed[MESH_REMOVE_BUFFER];
} MESHGROUPINST, *LPMESHGROUPINST;
WriteFileEx call on the server:
LPMESHGROUPINST meshes = (LPMESHGROUPINST)lpOverLap;
fWrite = WriteFileEx(
meshes->pipeInst,
(wchar_t*)meshes,
sizeof(MESHGROUPINST),
(LPOVERLAPPED)meshes,
(LPOVERLAPPED_COMPLETION_ROUTINE)CompletedWriteRoutine);
ReadFile call on the client:
(in header)
MESHGROUPINST _meshes;
(in cpp)
do
{
_success = ReadFile(
_pipe,
(wchar_t*)&_meshes,
sizeof(MESHGROUPINST),
&_numOfBytesRead,
NULL);
} while (!_success);
What is the type of _meshes in the ReadFile call? If it's a pointer, you'll be reading into the pointer, not the data being pointed to:
&_meshes
Should be:
_meshes
Also, it looks like you're writing process-specific HANDLE and OVERLAPPED info. Did you mean to write those?
You'll need to add more code for better help.
You need to ensure the structure is sent and received with 1-byte packing. Use #pragma pack(1) around the struct you wish to send/receive:
#pragma pack(1)
typedef struct
{
int id;
float vertices[VERTICES_COUNT][VERTICES_COMPONENTS];
unsigned short indices[INDICES_COUNT];
float texCoords[TEXTURE_COORD_COUNT][TEXTURE_COORD_COMPONENT];
unsigned char texData[TEXTURE_SIZE];
} MESHINST, *LPMESHINST;
#pragma pack()
I am developing a plug-in for two 3rd party programs: A and B in C++ on Windows (7) and need a robust, relatively simple (and fast) way to communicate between the two programs.
The communication is one way: based on user interaction in program A I want my plug-in inside program A to send a signal that ends up calling a function inside my plug-in in program B.
The protocol is simple. This is the signature of the receiving function inside my plug-in in B:
struct XYZ {
double x, y, z;
}
void polyLineSelected(long id, std::vector<XYZ> & points);
How would you recommend to do this?
By far the easiest way to implement one-way communication on Windows is to send a WM_COPYDATA message. It takes a COPYDATASTRUCT parameter to move arbitrary data from one application to another.
For your specific example an implementation of the sender would look like this:
// Declare symbolic constants to identify data
enum DataType {
DataType_Points
};
// Declare struct to hold compound data
struct IPCData {
long id;
XYZ pts[];
};
// Allocate buffer
const size_t bufferSize = offsetof(IPCData, pts[points.size()]);
vector<char> buffer(bufferSize);
IPCData* pData = reinterpret_cast<IPCData*>(&buffer[0]);
// Fill the buffer
pData->id = 42;
copy(points.begin(), points.end(), &pData->pts[0]);
// Prepare COPYDDATASTRUCT
COPYDATASTRUCT cds = { 0 };
cds.dwData = DataType_Points; // Can be used by the receiver to identify data
cds.cbData = bufferSize;
cds.lpData = pData;
// Send the data
SendMessage(hWndRecv, WM_COPYDATA,
(WPARAM)hWndSender,
(LPARAM)(LPVOID)&cds);
How the implementation of the struct in the D language to ubyte [] or ubyte [] to the struct, please brothers help answer this question, thank you!
If a struct contains the string or char [] what to do?
For example, such a structure:
struct UserLogin
{
align(1):
ushort ClientId;
int AccectId;
string LoginUid;
string LoginPwd;
}
Attention to my application in the socket!
I don't think there's anything in the standard library to automatically serialize and deserialize structures to byte streams. std.stream does that for a variety of basic types, but not entire structs. Apache Thrift support is on the way. Among 3rd-party solutions, you can have a look at the Orange serialization library.
To convert the raw data, the suggested idiom is like this:
struct_type* s = new struct_type;
ubyte[] ub = cast(ubyte[]) s[0..1];
struct_type* s2 = cast(struct_type*) ub.ptr;
This will not handle serialization of strings and pointers, though. You will need to do that manually or with a library.
#Dev Wolf : You have to write serialization/deserialization yourself. Apart from the Orange mentioned by CyberShadow you also have the Thrift protocol implementation : http://klickverbot.at/code/gsoc/thrift/ . I remember some guys worked on Google Protocol Buffer implementation as well.
struct UserLogin
{
align(1):
ushort ClientId;
int AccectId;
char[10] LoginUid;
char[10] LoginPwd;
}
UserLogin* readByteToStruct = cast(UserLogin*)ne.data;
Will be able to properly take the data...
I'm currently working on a small project: there's a protocol for sending some strings via UDP implemented with standard C interface.
Although it works pretty fine, I'd like to rewrite it with some more sophisticated C++ (consider it exercise).
Currently it's something like that: A client wants that string so it sends the following struct:
struct request {
uint8_t msg_type;// == 1
uint64_t key; // generated randomly to identify each request
}
In new implementation, I want to use boost::asio so in server I have a following piece of code:
boost::asio::io_service io_service;
boost::asio::ip::udp::endpoint client_endpoint;
boost::asio::ip::udp::socket socket(io_service,
boost::asio::ip::udp::endpoint(boost::asio::ip::udp::v4(),
m_serverPort));
boost::asio::streambuf sb;
boost::asio::streambuf::mutable_buffers_type mutableBuf =
sb.prepare(sizeof(request));
size_t received_bytes = socket.receive_from(mutableBuf, client_endpoint);
sb.commit(received_bytes);
request r;
std::istream is(&sb);
is >> msg_type;
is >> key;
key = __bswap64(key); // I'm using network byteorder for numbers sent with this protocol
// and there's no ntohll function on Snow Leopard (at least I can't
// find one)
sb.consume(received_bytes);
And here's my problem: the "key" value which I try to receive this way is wrong - I mean I get something that I did not send.
Here are my suspicions:
__bswap64 does not convert network to host (little-endian) byteorder
I misunderstood how to use boost::asio::streambuf with streams
There's some incompatibility between old C interface and boost (but I don't think so
cause I've found out that boost functions are just wrappers for it)
EDIT:
hmm they say "don't praise a ford till you get over". Now I have a very similar issue in another place of my code. I have a following struct which is sent as a reply for request metioned above:
struct __attribute__ ((packed)) CITE_MSG_T
{
uint8_t msg_id;
uint64_t key; // must be the same as in request
uint16_t index; // part number
uint16_t parts; // number of all parts
CITE_PART_T text; // message being sent
};
//where CITE_PART_T is:
struct __attribute__ ((packed)) CITE_PART_T
{
uint16_t data_length;
char* data;
};
and following piece of code: http://pastebin.com/eTzq6AWQ.
Unfortunately there's another bug in it and again I read something I haven't sent - replyMsg.parts and replyMsg.index is always 0 although old implementation says they're for example 3 and 10. What's wrong this time? As you can see I take care of padding and I use read instead of operator>>. If you wonder why I read that struct field by field here's an answer: A server sends two different structures, both beginning with msg_id, one if it succeceeds and another if it fails. Right now, I simply have no idea how to do it other way.
You're using formatted input, as though the data being sent were textual -- you need unformatted input. Read about the std::istream::read member function, as it's what you should be using rather than operator>>.
Note that this would have been immediately obvious if you had been checking the stream state after each extraction, as one always should in non-throw-away code.
You forgot about padding. Your request structure probably has at least three bytes inserted by the compiler between the first and the second member, as in:
struct request {
uint8_t msg_type;
char __pad__[3]; // or 7 on 64-bit machine.
uint64_t key;
};
You can fix that, say in GCC, with attributes (see the GCC manual):
struct __attribute__ ((__packed__)) request { ...
And yes, I did miss the fact that you are trying to read text instead of binary. Fix that first, get bitten by alignment/padding later :)