I try to send and receive data via TCP. My problem is that I want to send 2 structs in one TCP message, is there any way to link the struct together.
Something like:
send(connected, struct1 + struct2, sizeof(struct1 + struct2), 0);
recv_data = recv(connected, struct1 + struct2,sizeof( struct1 + struct2),0);
If not would it be possible to add a signal byte at the beginning of the message something like:
send(connected, "0x01" + struct1, sizeof(struct1 + 1), 0);
recv_data = recv(connected, struct1,sizeof(struct1),0);
You could look into the writev and readv functions, which allow sending/receiving data with multiple non-contiguous memory locations.
struct iovec data[2];
data[0].iov_base = vector1.data();
data[0].iov_len = vector1.length() * sizeof(vector1[0]);
data[1].iov_base = vector2.data();
data[1].iov_len = vector2.length() * sizeof(vector2[0]);
writev(socket, data, 2);
However at the receiving side you need some way to know the number of incoming elements so you can reserve enough space to read into.
This will also only work for vectors of trivially-copyable objects, not something like std::vector<std::string>.
You really need to define some kind of properly-designed serialization format, not just copy raw bytes from vectors to sockets.
If you don't know how to do this properly (and your pseudo-code suggests you don't) then look at something like Protocol Buffers instead.
Edit: ok, you're not talking about vectors, despite saying "vectors" repeatedly. For simple structs of trivial types you can do it:
struct iovec data[2];
data[0].iov_base = &struct1;
data[0].iov_len = sizeof(struct1);
data[1].iov_base = &struct2;
data[1].iov_len = sizeof(struct2);
writev(socket, data, 2);
But you still shouldn't just be copying raw bytes to/from sockets. This will only work if the sending and receiving machines agree on the sizes, alignments and endianness of the data types. It is still safer and more correct to use a properly-designed serialization system like Protocol Buffers. That has been designed by people who understand how to do these things correctly and safely.
Related
I need to decode a message from a std::string object.
I can see how to do this in Java by using a ByteBuffer, but I failed to find a simple C++ equivalent.
More precisely, how to implement this?
const std::string serialized_data = ...; // Source
SomeDataType data = ???; // How to convert from serialized_data to data?
You can write code like this:
std::string serialized_data = ...;
kj::ArrayPtr<const capnp::word> words(
reinterpret_cast<const capnp::word*>(serialized_data.begin()),
serialized_data.size() / sizeof(capnp::word));
capnp::FlatAraryMessageReader reader(words);
Note that this assumes that the backing buffer of an std::string is always word-aligned. I think this is true, but I am not sure. If it is not aligned, you will get exceptions saying so. In this case it may be necessary to copy the data into an aligned buffer:
auto words = kj::heapArray<capnp::word>(
serialized_data.size() / sizeof(capnp::word));
memcpy(words.begin(), serialized_data.begin(), words.asBytes().size());
capnp::FlatAraryMessageReader reader(words);
Note that while std::string is technically able to store arbitrary bytes, it is not a particularly good data structure for this purpose, since it is optimized for text. For example, it may re-allocate and copy your buffer in order to add a NUL terminator to support .c_str(). I would recommend using a more data-oriented type for byte arrays.
My own answer is very similar to Kenton's:
std::string serialized_data = ...;
kj::ArrayPtr<const kj::byte> arr(
reinterpret_cast<const kj::byte*>(serialized_data.data()),
serialized_data.size());
kj::ArrayInputStream stream(arr);
capnp::InputStreamMessageReader reader(stream);
auto data = reader.getRoot<SomeDataType>();
This is isomorphic to the Java version, and it has no alignment requirements.
But Kenton's solution is more straightforward than this. Therefore I will simplify my program further.
in a SoC solution, the fpga is saving a lot of integer values directly in the RAM.
This Data(the integers) can be seen by the processor on the other side who should send this data over the network without modifying it using the asio library.
Until now this Data was not too big and I copied it to a vector and I send it over the Ethernet without problem (see the code please).
On a current project the amount of the data has been increased (about 200MB) and I would like to send it directly from the ram without copying it a vector before. Of course I will split this to parts.
Is there a way to send this raw data directly from a RAM pointer of type void (void *ptr) or there is a better way to do that ?
Thanks in advance
std::vector<int> int_vect;
for( uint32_t i=from_index ; i<=to_index ; i++ )
{
int_vect.push_back(my_memory_ptr->get_value_axis(....));
}
asio::write(z_sock_ptr->socket_, asio::buffer(int_vect));
Yes. One of the overloads of asio::buffer provides exactly this functionality:
mutable_buffer buffer(
void * data,
std::size_t size_in_bytes);
» more...
If the data is contiguous, we can use it like this:
void* data = /* get data */;
size_t size = /* get size */;
asio::write(z_sock_ptr->socket_, asio::buffer(data, size));
It is possible to create asio buffer from raw data, it is essentially just a non-owning array view:
asio::write(z_sock_ptr->socket_, asio::buffer{p_data, bytes_count});
I'm using IOKit framework to communicate with my driver using IOConnectCallMethod from the user-space client and IOExternalMethodDispatch on the driver side.
So far I was able to send fixed length commands, and now I wish to send a varied size array of chars (i.e. fullpath).
However, it seems that the driver and the client sides command lengths are coupled, which means that checkStructureInputSize from IOExternalMethodDispatch in driver must be equal to inputStructCnt from
IOConnectCallMethod in client side.
Here are the struct contents on both sides :
DRIVER :
struct IOExternalMethodDispatch
{
IOExternalMethodAction function;
uint32_t checkScalarInputCount;
uint32_t checkStructureInputSize;
uint32_t checkScalarOutputCount;
uint32_t checkStructureOutputSize;
};
CLIENT:
kern_return_t IOConnectCallMethod(
mach_port_t connection, // In
uint32_t selector, // In
const uint64_t *input, // In
uint32_t inputCnt, // In
const void *inputStruct, // In
size_t inputStructCnt, // In
uint64_t *output, // Out
uint32_t *outputCnt, // In/Out
void *outputStruct, // Out
size_t *outputStructCnt) // In/Out
Here's my failed attempt to use a varied size command :
std::vector<char> rawData; //vector of chars
// filling the vector with filePath ...
kr = IOConnectCallMethod(_connection, kCommandIndex , 0, 0, rawData.data(), rawData.size(), 0, 0, 0, 0);
And from the driver command handler side, I'm calling IOUserClient::ExternalMethod with IOExternalMethodArguments *arguments and IOExternalMethodDispatch *dispatch but this requires the exact length of data I'm passing from the client which is dynamic.
this doesn't work unless I set the dispatch function with the exact length of data it should expect.
Any idea how to resolve this or perhaps there's a different API I should use in this case ?
As you have already discovered, the answer for accepting variable-length "struct" inputs and outputs is to specify the special kIOUCVariableStructureSize value for input or output struct size in the IOExternalMethodDispatch.
This will allow the method dispatch to succeed and call out to your method implementation. A nasty pitfall however is that structure inputs and outputs are not necessarily provided via the structureInput and structureOutput pointer fields in the IOExternalMethodArguments structure. In the struct definition (IOKit/IOUserClient.h), notice:
struct IOExternalMethodArguments
{
…
const void * structureInput;
uint32_t structureInputSize;
IOMemoryDescriptor * structureInputDescriptor;
…
void * structureOutput;
uint32_t structureOutputSize;
IOMemoryDescriptor * structureOutputDescriptor;
…
};
Depending on the actual size, the memory region might be referenced by structureInput or structureInputDescriptor (and structureOutput or structureOutputDescriptor) - the crossover point has typically been 8192 bytes, or 2 memory pages. Anything smaller will come in as a pointer, anything larger will be referenced by a memory descriptor. Don't count on a specific crossover point though, that's an implementation detail and could in principle change.
How you handle this situation depends on what you need to do with the input or output data. Usually though, you'll want to read it directly in your kext - so if it comes in as a memory descriptor, you need to map it into the kernel task's address space first. Something like this:
static IOReturn my_external_method_impl(OSObject* target, void* reference, IOExternalMethodArguments* arguments)
{
IOMemoryMap* map = nullptr;
const void* input;
size_t input_size;
if (arguments->structureInputDescriptor != nullptr)
{
map = arguments->structureInputDescriptor->createMappingInTask(kernel_task, 0, kIOMapAnywhere | kIOMapReadOnly);
if (map == nullptr)
{
// insert error handling here
return …;
}
input = reinterpret_cast<const void*>(map->getAddress());
input_size = map->getLength();
}
else
{
input = arguments->structureInput;
input_size = arguments->structureInputSize;
}
// …
// do stuff with input here
// …
OSSafeReleaseNULL(map); // make sure we unmap on all function return paths!
return …;
}
The output descriptor can be treated similarly, except without the kIOMapReadOnly option of course!
CAUTION: SUBTLE SECURITY RISK:
Interpreting user data in the kernel is generally a security-sensitive task. Until recently, the structure input mechanism was particularly vulnerable - because the input struct is memory-mapped from user space to kernel space, another userspace thread can still modify that memory while the kernel is reading it. You need to craft your kernel code very carefully to avoid introducing a vulnerability to malicious user clients. For example, bounds-checking a userspace-supplied value in mapped memory and then re-reading it under the assumption that it's still within the valid range is wrong.
The most straightforward way to avoid this is to make a copy of the memory once and then only use the copied version of the data. To take this approach, you don't even need to memory-map the descriptor: you can use the readBytes() member function. For large amounts of data, you might not want to do this for efficiency though.
Recently (during the 10.12.x cycle) Apple changed the structureInputDescriptor so it's created with the kIOMemoryMapCopyOnWrite option. (Which as far as I can tell was created specifically for this purpose.) The upshot of this is that if userspace modifies the memory range, it doesn't modify the kernel mapping but transparently creates copies of the pages it writes to. Relying on this assumes your user's system is fully patched up though. Even on a fully patched system, the structureOutputDescriptor suffers from the same issue, so treat it as write-only from the kernel's point of view. Never read back any data you wrote there. (Copy-on-write mapping makes no sense for the output struct.)
After going through the relevant manual again, I've found the relevant paragraph :
The checkScalarInputCount, checkStructureInputSize, checkScalarOutputCount, and checkStructureOutputSize fields allow for sanity-checking of the argument list before passing it along to the target object. The scalar counts should be set to the number of scalar (64-bit) values the target's method expects to read or write. The structure sizes should be set to the size of any structures the target's method expects to read or write. For either of the struct size fields, if the size of the struct can't be determined at compile time, specify kIOUCVariableStructureSize instead of the actual size.
So all I had to do in order to avoid the size verification, is to set the field checkStructureInputSize to value kIOUCVariableStructureSize in IoExternalMethodDispatch and the command passed to the driver properly.
I have a program that deals char[] buffers to send/receive messages. Until now, this is how it has been handled:
#pramga pack(1)
struct messageType
{
uint8_t data0:4;
uint8_t data1:4;
uint8_t data2;
//etc...
};
#pragma pack()
void MyClass::processMessage(char* buf)
{
// I already know buf is big enough to hold a messageType
messageType* msg = reinterpret_cast<messageType*>(buf);
//populate class member variables
m_data0 = msg->data0;
m_data1 = msg->data1;
m_data2 = msg->data2;
//etc
}
Now from what I've gathered from reading around is that this is technically undefined behavior due to strict aliasing, and that memcpy should be used instead? What I don't quite understand is, what potential issues does copying buf byte for byte to messageType msgNotPtr, then reading from that msgNotPtr, actually avoid?
Regarding sending, instead of doing this:
void MyClass::sendMessage()
{
char buf[max_tx_size];
messageType* msg = reinterpret_cast<messageType*>(buf);
msg->data0 = m_data0;
//etc...
send(buf);
}
I've read that I should be using placement new isntead, ala:
messageType* msg = new(buf) messageType;
If I do it this way, do I need to add additional cleanup, given that the struct messageType only contains POD types (such as manually firing the destructor)?
Edit: Now that I think about it, is sendMessage still undefined? Do I need to also swap out the last command with something like send(reinterpret_cast<char*>(msg)) to make sure the compiler does not optimize out the call?
In my opinion, the proper method to load a class instance from a uint8_t buffer is to load each member separately.
The class should know the positions of its members within the buffer and where and the size of any padding or reserved areas.
One of the issues with mapping structures to buffers is that the compiler can add space between members. If you pack the structure to eliminate the padding, you slow down your program every time you access a member.
So, bite the performance on input and output by placing the members where you want them in the buffer (or extracting the members according to a specification). The rest of the program can access the members the way the compiler aligned them.
This includes bit fields.
Also, by having the members loaded individually, the Endianess of the fields in the buffer can be account for much easier.
I have a Struct to send over a socket to a client. Both the Client and the Server is on the same architecture so there is no endian problem. I receive the int values properly. But not able to receive the char[] values properly.
This is the structure.
struct Packet {
int id;
int number;
char data[256];
};
In the Server side I serialize the data and write to the client.
struct Packet *s = new Packet();
s->id= htonl(1000);
s->number= htonl(7788);
memcpy(s->data, "MESSAGE", 7);
n = write(NewSockFD , s ,sizeof(s) );
In the Client side I deserialize the data.
n = read(SockFD , Buffer , sizeof(Buffer));
struct Packet *s = (struct Packet*)Buffer;
char b[256];
int i = ntohl(s->id);
int j = ntohl(s->number);
memcpy(b, s->data, sizeof(s));
I receive the id and number values correctly. Problem is with the data value. What I'm doing wrong here??..
In your code, you use sizeof(s). This will be the size of a Packet*, not a Packet. Replace it with sizeof(*s) to get the correct size.
Additionally, since the values of data are not all initialised, you cause undefined behaviour by reading from it. You need to initialise all the elements one way or the other (the shortest way would be to do char data[256] { }; in the struct definition).
Also since this is C++, you don't need to say struct Packet, you can say just Packet, unless you also have a function named Packet. But it appears in your code only half the time which means you don't, so you can just drop it.
And as Chris G mentioned, you have another problem after you fix that, which is that you're copying an entire Packet into a char[] only big enough to hold a Packet's data. Change
memcpy(b, s->data, sizeof(s))
to
memcpy(b, s->data, sizeof(s->data))
And realise that this data may not be nul-terminated if the sender didn't do it for you (which you may want to take precautions against).