in a SoC solution, the fpga is saving a lot of integer values directly in the RAM.
This Data(the integers) can be seen by the processor on the other side who should send this data over the network without modifying it using the asio library.
Until now this Data was not too big and I copied it to a vector and I send it over the Ethernet without problem (see the code please).
On a current project the amount of the data has been increased (about 200MB) and I would like to send it directly from the ram without copying it a vector before. Of course I will split this to parts.
Is there a way to send this raw data directly from a RAM pointer of type void (void *ptr) or there is a better way to do that ?
Thanks in advance
std::vector<int> int_vect;
for( uint32_t i=from_index ; i<=to_index ; i++ )
{
int_vect.push_back(my_memory_ptr->get_value_axis(....));
}
asio::write(z_sock_ptr->socket_, asio::buffer(int_vect));
Yes. One of the overloads of asio::buffer provides exactly this functionality:
mutable_buffer buffer(
void * data,
std::size_t size_in_bytes);
» more...
If the data is contiguous, we can use it like this:
void* data = /* get data */;
size_t size = /* get size */;
asio::write(z_sock_ptr->socket_, asio::buffer(data, size));
It is possible to create asio buffer from raw data, it is essentially just a non-owning array view:
asio::write(z_sock_ptr->socket_, asio::buffer{p_data, bytes_count});
Related
I'm using IOKit framework to communicate with my driver using IOConnectCallMethod from the user-space client and IOExternalMethodDispatch on the driver side.
So far I was able to send fixed length commands, and now I wish to send a varied size array of chars (i.e. fullpath).
However, it seems that the driver and the client sides command lengths are coupled, which means that checkStructureInputSize from IOExternalMethodDispatch in driver must be equal to inputStructCnt from
IOConnectCallMethod in client side.
Here are the struct contents on both sides :
DRIVER :
struct IOExternalMethodDispatch
{
IOExternalMethodAction function;
uint32_t checkScalarInputCount;
uint32_t checkStructureInputSize;
uint32_t checkScalarOutputCount;
uint32_t checkStructureOutputSize;
};
CLIENT:
kern_return_t IOConnectCallMethod(
mach_port_t connection, // In
uint32_t selector, // In
const uint64_t *input, // In
uint32_t inputCnt, // In
const void *inputStruct, // In
size_t inputStructCnt, // In
uint64_t *output, // Out
uint32_t *outputCnt, // In/Out
void *outputStruct, // Out
size_t *outputStructCnt) // In/Out
Here's my failed attempt to use a varied size command :
std::vector<char> rawData; //vector of chars
// filling the vector with filePath ...
kr = IOConnectCallMethod(_connection, kCommandIndex , 0, 0, rawData.data(), rawData.size(), 0, 0, 0, 0);
And from the driver command handler side, I'm calling IOUserClient::ExternalMethod with IOExternalMethodArguments *arguments and IOExternalMethodDispatch *dispatch but this requires the exact length of data I'm passing from the client which is dynamic.
this doesn't work unless I set the dispatch function with the exact length of data it should expect.
Any idea how to resolve this or perhaps there's a different API I should use in this case ?
As you have already discovered, the answer for accepting variable-length "struct" inputs and outputs is to specify the special kIOUCVariableStructureSize value for input or output struct size in the IOExternalMethodDispatch.
This will allow the method dispatch to succeed and call out to your method implementation. A nasty pitfall however is that structure inputs and outputs are not necessarily provided via the structureInput and structureOutput pointer fields in the IOExternalMethodArguments structure. In the struct definition (IOKit/IOUserClient.h), notice:
struct IOExternalMethodArguments
{
…
const void * structureInput;
uint32_t structureInputSize;
IOMemoryDescriptor * structureInputDescriptor;
…
void * structureOutput;
uint32_t structureOutputSize;
IOMemoryDescriptor * structureOutputDescriptor;
…
};
Depending on the actual size, the memory region might be referenced by structureInput or structureInputDescriptor (and structureOutput or structureOutputDescriptor) - the crossover point has typically been 8192 bytes, or 2 memory pages. Anything smaller will come in as a pointer, anything larger will be referenced by a memory descriptor. Don't count on a specific crossover point though, that's an implementation detail and could in principle change.
How you handle this situation depends on what you need to do with the input or output data. Usually though, you'll want to read it directly in your kext - so if it comes in as a memory descriptor, you need to map it into the kernel task's address space first. Something like this:
static IOReturn my_external_method_impl(OSObject* target, void* reference, IOExternalMethodArguments* arguments)
{
IOMemoryMap* map = nullptr;
const void* input;
size_t input_size;
if (arguments->structureInputDescriptor != nullptr)
{
map = arguments->structureInputDescriptor->createMappingInTask(kernel_task, 0, kIOMapAnywhere | kIOMapReadOnly);
if (map == nullptr)
{
// insert error handling here
return …;
}
input = reinterpret_cast<const void*>(map->getAddress());
input_size = map->getLength();
}
else
{
input = arguments->structureInput;
input_size = arguments->structureInputSize;
}
// …
// do stuff with input here
// …
OSSafeReleaseNULL(map); // make sure we unmap on all function return paths!
return …;
}
The output descriptor can be treated similarly, except without the kIOMapReadOnly option of course!
CAUTION: SUBTLE SECURITY RISK:
Interpreting user data in the kernel is generally a security-sensitive task. Until recently, the structure input mechanism was particularly vulnerable - because the input struct is memory-mapped from user space to kernel space, another userspace thread can still modify that memory while the kernel is reading it. You need to craft your kernel code very carefully to avoid introducing a vulnerability to malicious user clients. For example, bounds-checking a userspace-supplied value in mapped memory and then re-reading it under the assumption that it's still within the valid range is wrong.
The most straightforward way to avoid this is to make a copy of the memory once and then only use the copied version of the data. To take this approach, you don't even need to memory-map the descriptor: you can use the readBytes() member function. For large amounts of data, you might not want to do this for efficiency though.
Recently (during the 10.12.x cycle) Apple changed the structureInputDescriptor so it's created with the kIOMemoryMapCopyOnWrite option. (Which as far as I can tell was created specifically for this purpose.) The upshot of this is that if userspace modifies the memory range, it doesn't modify the kernel mapping but transparently creates copies of the pages it writes to. Relying on this assumes your user's system is fully patched up though. Even on a fully patched system, the structureOutputDescriptor suffers from the same issue, so treat it as write-only from the kernel's point of view. Never read back any data you wrote there. (Copy-on-write mapping makes no sense for the output struct.)
After going through the relevant manual again, I've found the relevant paragraph :
The checkScalarInputCount, checkStructureInputSize, checkScalarOutputCount, and checkStructureOutputSize fields allow for sanity-checking of the argument list before passing it along to the target object. The scalar counts should be set to the number of scalar (64-bit) values the target's method expects to read or write. The structure sizes should be set to the size of any structures the target's method expects to read or write. For either of the struct size fields, if the size of the struct can't be determined at compile time, specify kIOUCVariableStructureSize instead of the actual size.
So all I had to do in order to avoid the size verification, is to set the field checkStructureInputSize to value kIOUCVariableStructureSize in IoExternalMethodDispatch and the command passed to the driver properly.
I try to send and receive data via TCP. My problem is that I want to send 2 structs in one TCP message, is there any way to link the struct together.
Something like:
send(connected, struct1 + struct2, sizeof(struct1 + struct2), 0);
recv_data = recv(connected, struct1 + struct2,sizeof( struct1 + struct2),0);
If not would it be possible to add a signal byte at the beginning of the message something like:
send(connected, "0x01" + struct1, sizeof(struct1 + 1), 0);
recv_data = recv(connected, struct1,sizeof(struct1),0);
You could look into the writev and readv functions, which allow sending/receiving data with multiple non-contiguous memory locations.
struct iovec data[2];
data[0].iov_base = vector1.data();
data[0].iov_len = vector1.length() * sizeof(vector1[0]);
data[1].iov_base = vector2.data();
data[1].iov_len = vector2.length() * sizeof(vector2[0]);
writev(socket, data, 2);
However at the receiving side you need some way to know the number of incoming elements so you can reserve enough space to read into.
This will also only work for vectors of trivially-copyable objects, not something like std::vector<std::string>.
You really need to define some kind of properly-designed serialization format, not just copy raw bytes from vectors to sockets.
If you don't know how to do this properly (and your pseudo-code suggests you don't) then look at something like Protocol Buffers instead.
Edit: ok, you're not talking about vectors, despite saying "vectors" repeatedly. For simple structs of trivial types you can do it:
struct iovec data[2];
data[0].iov_base = &struct1;
data[0].iov_len = sizeof(struct1);
data[1].iov_base = &struct2;
data[1].iov_len = sizeof(struct2);
writev(socket, data, 2);
But you still shouldn't just be copying raw bytes to/from sockets. This will only work if the sending and receiving machines agree on the sizes, alignments and endianness of the data types. It is still safer and more correct to use a properly-designed serialization system like Protocol Buffers. That has been designed by people who understand how to do these things correctly and safely.
I am using an older network transmission function for a legacy product, which takes a char array and transmits it over the network. This char array is just data, no need for it make sense (or be null terminated). As such in the past the following occurred:
struct robot_info {
int robot_number;
int robot_type;
...
} // A robot info data structure for sending info.
char str[1024], *currentStrPos = str;
robot_info r_info;
... // str has some header data added to it.
... // robot info structure is filled out
memcpy(currentStrPos, (char *)&r_info, sizeof robot_info); // Add the robot info
scanSocket.writeTo(str, currentStrPos - str); // Write to the socket.
We have just added a bunch of stuff to robot_info but i am not happy with the single length method of the above code, i would prefer a dynamiclly allocated raii type in order to be expandable, especially since there can be multiple robot_info structures. I propose the following:
std::vector<char> str;
... // str has some header information added to it.
... // r_info is filled out.
str.insert(str.end(), (char *)&r_info, (char *)&r_info + sizeof r_info);
scanSocket.writeTo(str.data(), str.size());
Live example.
Using the std::vector insert function (with a pointer to the start of r_info as the iterator) and relying on the fact that a struct here would be aligned to at least a char and can be operated on like this. The struct has no dynamic memory elements, and no inheritance.
Will this have well defined behavior? Is there a better way to perform the same action?
While this works, it is ultimately solving a compile time problem with a run time solution. Since robot_info is a defined type, a better solution would be this:
std::array<char, sizeof robot_info> str;
memcpy(str.data(), static_cast<char *>(&r_info), sizeof robot_info);
scanSocket.writeTo(str.data(), str.size());
This has the advantages:
Can never be over size, or undersized
Automatic Storage duration and stack allocation means this is potentially faster
Problem statement : User provides some data which I have to store inside a structure. This data which I receive come in a data structure which allows user to dynamically add data to it.
Requirement: I need a way to store this data 'inside' the structure, contiguously.
eg. Suppose user can pass me strings which I have to store. So I wrote something like this :
void pushData( string userData )
{
struct
{
string junk;
} data;
data.junk = userData;
}
Problem : When I do this kind of storage, actual data is not really stored 'inside' the structure because string is not POD. Similar problem comes when I receive vector or list.
Then I could do something like this :
void pushData( string userData )
{
struct
{
char junk[100];
} data;
// Copy userdata into array junk
}
This store the data 'inside' the structure, but then, I can't put an upper limit on the size of string user can provide.
Can someone suggest some approach ?
P.S. : I read something about serializability, but couldnt really make out clearly if it could be helpful in my case. If it is the way to go forward, can someone give idea how to proceed with it ?
Edit :
No this is not homework.
I have written an implementation which can pass this kind of structure over message queues. It works fine with PODs, but I need to extend it to pass on dynamic data as well.
This is how message queue takes data:
i. Give it a pointer and tell the size till which it should read and transfer data.
ii. For plain old data types, data is store inside the structure, I can easily pass on the pointer of this structure to message queue to other processes.
iii. But in case of vector/string/list etc, actual data is not inside the structure and thus if I pass on the pointer of this structure, message queue will not really pass on the actual data, but rather the pointers which would be stored inside this structure.
You can see this and this. I am trying to achieve something similar.
void pushData( string userData )
{
struct Data
{
char junk[1];
};
struct Data* data = malloc(userData.size() + 1);
memcpy(data->junk, userData.data(), userData.size());
data->junk[userData.size()] = '\0'; // assuming you want null termination
}
Here we use an array of length 1, but we allocate the struct using malloc so it can actually have any size we want.
You ostensibly have some rather artificial constraints, but to answer the question: for a single struct to contain a variable amount of data is not possible... the closest you can come is to have the final member be say char [1], put such a struct at the start of a variably-sized heap region, and use the fact that array indexing is not checked to access memory beyond that character. To learn about this technique, see http://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html (or the answer John Zwinck just posted)
Another approach is e.g. template <size_t N> struct X { char data_[size]; };, but each instantiation will be a separate struct type, and you can't pre-instantiate every size you might want at run-time (given you've said you don't want an upper bound). Even if you could, writing code that handles different instantiations as the data grows would be nightmarish, as would the code bloat caused.
Having a structure in one place with a string member with data in another place is almost always preferable to the hackery above.
Taking a hopefully-not-so-wild guess, I assume your interest is in serialising the object based on starting address and size, in some generic binary block read/write...? If so, that's still problematic even if your goal were satisfied, as you need to find out the current data size from somewhere. Writing struct-specific serialisation routines that incorporates the variable-length data on the heap is much more promising.
Simple solution:estimate max_size of data (ex 1000), to prevent memory leak(if free memory & malloc new size memory -> fragment memory) when pushData multiple called.
#define MAX_SIZE 1000
void pushData( string userData )
{
struct Data
{
char junk[MAX_SIZE];
};
memcpy(data->junk, userData.data(), userData.size());
data->junk[userData.size()] = '\0'; // assuming you want null termination
}
As mentioned by John Zwinck....you can use dynamic memory allocation to solve your problem.
void pushData( string userData )
{
struct Data
{
char *junk;
};
struct Data *d = calloc(sizeof(struct data), 1);
d->junk = malloc(strlen(userData)+1);
strcpy(d->junk, userdata);
}
My C++ project has a buffer which could be any size and is filled by Bluetooth. The format of the incoming messages is like 0x43 0x0B 0x00 0x06 0xA2 0x03 0x03 0x00 0x01 0x01 0x0A 0x0B 0x0B 0xE6 0x0D in which starts with 0x43 and ends with 0x0D. So, it means that each time when buffer is filled, it can have different order of contents according to the above message format.
static const int BufferSize = 1024;
byte buffer[BufferSize];
What is the best way to parse the incoming messages in this buffer?
Since I have come from Java and .NET, What is the best way to make each extracted message as an object? Class could be solution?
I have created a separate class for parsing the buffer like bellow, am I in a right direction?
#include<parsingClass.h>
class A
{
parsingClass ps;
public:
parsingClass.parse(buffer, BufferSize);
}
class ReturnMessage{
char *message;
public:
char *getMessage(unsigned char *buffer,int count){
message = new char[count];
for(int i = 1; i < count-2; i++){
message[i-1] = buffer[i];
}
message[count-2] = '\0';
return message;
}
};
class ParserToMessage{
static int BufferSize = 1024;
unsigned char buffer[BufferSize];
unsigned int counter;
public:
static char *parse_buffer()
{
ReturnMessage rm;
unsigned char buffByte;
buffByte = blueToothGetByte(); //fictional getchar() kind of function for bluetooth
if(buffByte == 0x43){
buffer[counter++] = buffByte;
//continue until you find 0x0D
while((buffByte = blueToothGetByte()) != 0x0D){
buffer[counter++] = buffByte;
}
}
return rm.getMessage(buffer,counter);
}
};
Can you have the parser as a method of a 'ProtocolUnit' class? The method could take a buffer pointer/length as a parameter and return an int that indicates how many bytes it consumed from the buffer before it correctly assembled a complete protocol unit, or -1 if it needs more bytes from the next buffer.
Once you have a complete ProtocolUnit, you can do what you wish with it, (eg. queue it off to some processing thread), and create a new one for the remaining bytes/next buffer.
My C++ project has a buffer which could be any size
The first thing I notice is that you have hard-coded the buffer size. You are in danger of buffer overflow if an attempt is made to read data bigger than the size you have specified into the buffer.
If possible keep the buffer size dynamic and create the byte array according to the size of the data to be received into the buffer. Try and inform the object where your byte array lives of the incoming buffer size, before you create the byte array.
int nBufferSize = GetBufferSize();
UCHAR* szByteArray = new UCHAR[nBufferSize];
What is the best way to parse the incoming messages in this buffer?
You are on the right lines, in that you have created and are using a parser class. I would suggest using memcpy to copy the individual data items one at a time, from the buffer to a variable of your choice. Not knowing the wider context of your intention at this point, I cannot add much to that.
Since I have come from Java and .NET, What is the best way to make
each extracted message as an object? Class could be solution?
Depending on the complexity of the data you are reading from the buffer and what your plans are, you could use a class or a struct. If you do not need to create an object with this data, which provides services to other objects, you could use a struct. Structs are great when your need isn't so complex, whereby a full class might be overkill.
I have created a separate class for parsing the buffer like bellow, am
I in a right direction?
I think so.
I hope that helps for starters!
The question "how should I parse this" depends largely on how you want to parse the data. Two things are missing from your question:
Exactly how do you receive the data? You mention Bluetooth but what is the programming medium? Are you reading from a socket? Do you have some other kind of API? Do you receive it byte at a time or in blocks?
What are the rules for dealing with the data you are receiving? Most data is delimited in some way or of fixed field length. In your case, you mention that it can be of any length but unless you explain how you want to parse it, I can't help.
One suggestion I would make is to change the type of your buffer to use std::vector :
std::vector<unsigned char> buffer(normalSize)
You should choose normalSize to be something around the most frequently observed size of your incoming message. A vector will grow as you push items onto it so, unlike the array you created, you won't need to worry about buffer overrun if you get a large message. However, if you do go above normalSize under the covers the vector will reallocate enough memory to cope with your extended requirements. This can be expensive so you don't want to do it too often.
You use a vector in pretty much the same way as your array. One key difference is that you can simply push elements onto the end of the vector, rather than having to keep a running pointer. SO imagine you received a single int at a time from the Bluetooth source, your code might look something like this:
// Clear out the previous contents of the buffer.
buffer.clear();
int elem(0);
// Find the start of your message. Throw away elements
// that we don't need.
while ( 0x43 != ( elem = getNextBluetoothInt() ) );
// Push elements of the message into the buffer until
// we hit the end.
while ( 0x0D != elem )
{
buffer.push_back( elem );
}
buffer.push_back( elem ); // Remember to add on the last one.
The key benefit is that array will automatically resize the vector without you having to do it no matter whether the amount of characters pushed on is 10 or 10,000.