I am reading data from a serial port (in an Arduino) and framing it (syncing on a few bytes). To do that, I am reading the data into a big buffer.
Once I got the frame, I extract data and I want to send it to a different serial port using serial.write (Serial.write(buf, len)) which accepts a byte array and its size.
Since the data size can be random, I need something like a dynamic array (which is not recommended in Arduino). Any ideas?
Since the data size can be random, I need something like a dynamic array
In C you rarely need a dynamic array, because arrays passed to functions do not carry their size with them. That is why all functions that take an array also take length.
Let's say you have your data inside bigBuffer at position startPos, and you wish to send length bytes. All you need to do is
Serial.write(&bigBuffer[startPos], length);
or with pointer arithmetic syntax
Serial.write(bigBuffer+startPos, length);
Related
I have a controller and I need to store in flash memory an array of structs, which can have 2000+ items.
What is the best way to encode the array and save it in the flash memory?
Is this good enough?
char b[sizeof(array)];
memcpy(b, &array, sizeof(array));
Thanks
char b[sizeof(array)];
memcpy(b, &array, sizeof(array));
This will give you a bit-by-bit identical copy of the array. If you then write that to flash using whatever interface you have to the flash then you might as well just cast the original array to std::byte and write it directly.
The problem comes when the objects in the array are not trivially copyable. Anything with pointers (like std::string for example) will not work. You would have to serialize before writing.
You might actually want to do that even with just fundamental types in a struct because the struct might contain padding. The size of the struct might be larger than the size of the data it contains. With 2000+ objects in the array even saving 1 byte would save you 2000 byte of flash every time your write. You could also use variable length encoding if you for example have 4 bytes ints that are mostly small numbers.
Please look up this code for context.
auto normalized_log = CreateNormalizedLog(builder, pairs);
builder.Finish(normalized_log);
auto buffPtr = builder.GetBufferPointer();
SendUdpPacket(&ipInfo, reinterpret_cast<SOCKET>(ipInfo.hdle), buffPtr, builder.GetSize());
I need to pack the size of the created buffPtr (with fixed two bytes). Is there any preferred way of appending/offsetting without copying the entire buffer?
I think I cannot add size to the schema, because after receiving I should know the size without calling getRootAsNormalizedLog.
Is there any way to add extra bytes to the resulting buffer?
There's no built-in facility to make a buffer length-prefixed. You shouldn't need to either: UDP packets (and most transfer mechanisms) know the size of their payload, so prefixing it yourself would just be duplicate information.
That said, if you insist to do this without copying, you could do something like this:
auto size = static_cast<uint16_t>(builder.GetSize());
builder.PushElement(size);
This will prefix the buffer with a 16bit size. The problem with this approach is that the buffer will already have been aligned for their largest elements, so the buffer is now possibly unaligned at the destination. Hence, you're better off using a 32bit (or 64bit) length, depending on what are the largest scalars in your buffer.
I don't quite understand the advantage of using streambuf over the regular array.
Let me explain my problem. I have a network connection which is encrypted using Rijndael 128 ECB + some easy cipher to encrypt remaining data that is shorter than 16 bytes. The packets are structured as length_of_whole_packet+operationcode+data. I have to actually copy all the data from the streambuf so I can apply decryption algorithm? Why making another copy of data I already have?
Same problem I have with the sending data. When in securedmode the packet structure is length_of_whole_packet+crc+data, where crc and data is encrypted. I could make some monstrosity as MakePacket(HEADER, FORMAT, ...) that would allocate array, format the packet, add crc and encrypt it, but I would like to avoid vararg function. I can't use structs as the packets has dynamic length because there can be arrays in it, or strings. If I used MakePacket(unsigned char opcode, &streambuf sb) then there would be problem with the crc again -> have to make a copy to encrypt it.
Should I use the vararg monstrosity for sending using regular array as buffer in combination with unsigned char pbyRecvBuffer[BUFFERMAXLEN] for recv?
I'm not really sure how to design this communication with avoiding copies of data.
Thank you for you answers.
When using streambufs, copying of data can often be minimized by using algorithms that operate on iterators, such as std::istreambuf_iterator or boost::asio::buffers_iterator, rather than copying data from a streambuf into another data structure.
For stream-like application protocols, boost::asio::streambuf is often superior to boost::asio::buffer() compatible types, such as a raw array. For example, consider HTTP, where a delimiter is used to identify boundaries between variable length headers and bodies. The higher-level read_until() operations provide an elegant way to to read the protocol, as Boost.Asio will handle the memory allocation, detect the delimiter, and invoke the completion handler once the message boundary has been reached. If an application used a raw array, it would need to read chunks and copy each fragmented chunk into an aggregated memory buffer until the appropriate delimiter was identified.
If the application can determine the exact amount of bytes to read, then it may be worth considering using boost::array for fixed length portions and std::vector for the variable length portions. For example, an application protocol with a:
fixed length body could be read into a boost::array.
fixed length header that contains enough information to determine the length of the following variable length body could use a std::vector to read the fixed size header, resize the vector once the body length has been determined, then read the body.
In context of the question, if length_of_whole_packet is of a fixed length, the application could read it into std::vector, resize the vector based on the determined body length, then read the remaining data into the vector. The decryption algorithm could then operate directly on the vector and use an output iterator, such as std::back_insert_iterator, with an auxiliary output buffer if the algorithm cannot be done in-place. The same holds true for encrypting data to be written.
In the game I'm making, I nee to be able to send std::vectors of integer over a network.
A packet seems to be made up entirely of a string. Since enet, the network libray im using takes care of endian, my first idea on solving this is to send a messsage where the first byte is the message id, as usual, the next 4 bytes would be an integer indicating the length of the array, and all subsequent bytes would be the ints in the array. On the client side I can then push these back into a vector.
Is this how it is usually done or am I missing something critical? Is there a better way to do it?
Thanks
In general, there are two approaches to solving this problem which can be combined. One is to put the length of the array before the actual array. The other involves including some framing to mark the end (or beginning) of your message. There are advantages and disadvantages to each.
Putting the length before the array is simplest. However, if there should ever be a bug where the length does not match the number of integers, there is no way to detect this or recover from it.
Using a framing byte(s) to mark the end of the message has the advantage of more robustness and the ability to recover from an improperly formatted message at the cost of complexity. The complexity comes in the fact that if your framing bytes appear in your array of integers, you must escape the bytes (i.e. prepend and escape character). Also, your code to read messages from the network becomes a little more complicated.
Of course, this all assumes that you are dealing stream and not a datagram. If your messages are clearly packetized already, then the length should be enough.
There are two ways to send variable length data on a stream: by prefixing the data with the length, or by suffixing it with a delimiter.
A suffix can have a theoretically infinite size, but it means the delimiter must not appear in the data. This approach can be used for strings, with a NUL ('\0') character as the delimiter.
When dealing with binary data, then you don't have any choice but to prefix the length. The size of the data will be limited to the the size of the prefix, which is rarely a problem with a 4 byte prefix (because otherwise it means you're sending more than 4 gigabytes of data).
So, it all depends on the data being sent.
Appending that header information to your packet is a good approach to take. Another option you can do on the receive side, if the data is stored in some unsigned char* buffer of memory, is create a structure like so:
typedef struct network_packet
{
char id;
int message_size;
int data[];
} __attribute__((packed)) network_packet;
You can then simply "overlay" this structure on-top of your received buffer like so:
unsigned char* buffer;
//...fill the buffer and make sure it's the right size -- endian is taken care of
//via library
network_packet* packet_ptr = (network_packet*)buffer;
//access the 10th integer in the packet if the packet_ptr->message_size
//is long enough
if (packet_ptr->message_size >= 10)
int tenth_int = packet_ptr->data[9];
This will avoid you having to go through the expense of copying all the data back, which already exists in a buffer, back into another std::vector on the receive side.
I'm using C++ and wondering if I can just send an entire int array over a network (using basic sockets) without doing anything. Or do I have to split the data up and send it one at a time?
Yes.
An array will be laid out sequentially in memory so you are free to do this. Simply pass in the address of the first element and the amount of data and you'll send all data.
You could definitely send an array in one send, however you might want to do some additional work. There are issues with interpreting it correctly at the receiving end. For example, if using different machine architectures, you may want to convert the integers to network order (e.g., htonl).
Another thing to keep in mind is the memory layout. If it is a simple array of integers, then it would be contiguous in memory and a single send could successfully capture all the data. If, though, (and this is probably obvious), you have an array with other data, then the layout definitely needs consideration. A simple example would be if the array had pointers to other data such as a character string, then a send of the array would be sending pointers (and not data) and would be meaningless to the receiver.