Let's say I have an array of uint32_t, N elements.
The content of the array is only meaningful if I unpack the entire N*32 bits into small fragments.
The fragment size can be highly irregular: 1 segment of 5 bits, 2 segment of 7 bits, 1 segment of 48 bits, ... Any pattern is possible.
What is the fastest way to unpack the array?
I was thinking of using a large std::bitset<N*32>, convert to a string and the read the sub-strings; but it's kind of slow.
As another possibility, for each fragment, I was thinking to:
locate the index in the array of the starting "pointer"
locate the index in the array of the end "pointer"
locate, in the single located array element, where the starting/ending bit
do a loop and read bit by bit and merge
Working but I bet there must be something more simple.
Any idea?
Thanks
What is the the fastest way to unpack the array?
Bitwise operations.
This is the first time I'm working with large data sets in C++. Before, I'd just store whatever I want to in arrays. Because I'm generating heightmaps of 4000x4000 now, I want to be a bit more memory efficient, before my program is suddenly gulping up 100 MB (a little exorbitant for an indie fellow like me).
I'm going to load most of the data in chunks, but because my program needs a large amount of the data during execution, I'll still end up with large 2D-arrays in use. I want to find a way to make them occupy very little memory.
My heightmap is designed so that it only takes on small integer values; possibly small enough to fit into an Int8 but at the very least small enough for a Int16. So far, I've been using Int types, which on my implementation are of the Int32 variety.
If I were to switch my 2D-arrays and vectors to sets of the Int16 type, would this save me half the storage? Or would one element of the array still take one full byte and simply leave those untouched bits at zero?
If you are talking about the fixed-width integer types, then yes int16_t is guaranteed to be 16 bits without padding bits. You may also verify this yourself:
std::cout << sizeof(int16_t[1000]) << " " << sizeof(int32_t[1000]);
Switching to Int16 (assuming this is the same as int16_t) from Int32 (assuming this is the same as int32_t) would bring down the memory consumption that stems from your array by half (not considering any bookkeeping that goes along with the possibly dynamically sized array)
Also int16_t takes 2 bytes, and int32_t takes 4 bytes. The 16_t implies 16 bits.
There are no untouched bits, only those which you chose to not use at runtime. Unlike some pointer types, there are no unused bits with integers. 0s and 1s give identity to the integer's value
I am reading data from a serial port (in an Arduino) and framing it (syncing on a few bytes). To do that, I am reading the data into a big buffer.
Once I got the frame, I extract data and I want to send it to a different serial port using serial.write (Serial.write(buf, len)) which accepts a byte array and its size.
Since the data size can be random, I need something like a dynamic array (which is not recommended in Arduino). Any ideas?
Since the data size can be random, I need something like a dynamic array
In C you rarely need a dynamic array, because arrays passed to functions do not carry their size with them. That is why all functions that take an array also take length.
Let's say you have your data inside bigBuffer at position startPos, and you wish to send length bytes. All you need to do is
Serial.write(&bigBuffer[startPos], length);
or with pointer arithmetic syntax
Serial.write(bigBuffer+startPos, length);
I am trying write a function that reads PPM images and the function should return the contents.
PPM images have the following text format:
P3
numOfRows numOfColumns
maxColor
numOfRows-by-numOfColumns of RGB colors
Since the text format has a mixture of variable types, is there any way to store this all in an array? I remembered that C++ does not support arrays with different types. If not, then I am thinking of defining a class to store the PPM contents.
C++ does not support arrays with different types.
Correct.
You could:
Define a class as you say, like this: C++ Push Multiple Types onto Vector or this: Creating a vector that holds two different data types or classes or even this: Vector that can have 3 different data types C++.
Have a generic C-like array (or better yet, an std::vector) with void*.
C++ isn't Javascript.
The number of columns / number of rows must be integers. Maximum colour value might be either an integer or a float depending on the format details, as might the rgb values.
So you read the image dimensions first. Then you create a buffer to hold the image. Usually 32 bit rgba is what you want, so either allocate width * height * 4 with malloc() or use an std::vector and resize.
Then you loop through the data, reading the values and putting them into
the array.
Then you create an "Image" object, with integer members of width and height, and a pixel buffer of 32 bit rgbas (or whatever is your preferred pixel format).
I know it's ridiculous, but I need it for storage optimization. Is there any good way to implement it in C++?
It has to be flexible enough so that I can use it as a normal data type e.g Vector< int20 >, operator overloading, etc..
If storage is your main concern, I suspect you need quite a few 20-bit variables. How about storing them in pairs? You could create a class representing two such variables and store them in 2.5+2.5 = 5 bytes.
To access the variables conveniently you could override the []-operator so you could write:
int fst = pair[0];
int snd = pair[1];
Since you may want to allow for manipulations such as
pair[1] += 5;
you would not want to return a copy of the backing bytes, but a reference. However, you can't return a direct reference to the backing bytes (since it would mess up it's neighboring value), so you'd actually need to return a proxy for the backing bytes (which in turn has a reference to the backing bytes) and let the proxy overload the relevant operators.
As a metter of fact, as #Tony suggest, you could generalize this to have a general container holding N such 20-bit variables.
(I've done this myself in a specialization of a vector for efficient storage of booleans (as single bits).)
No... you can't do that as a single value-semantic type... any class data must be a multiple of the 8-bit character size (inviting all the usual quips about CHAR_BITS etc).
That said, let's clutch at straws...
Unfortunately, you're obviously handling very many data items. If this is more than 64k, any proxy object into a custom container of packed values will probably need a >16 bit index/handle too, but still one of the few possibilities I can see worth further consideration. It might be suitable if you're only actively working with and needing value semantic behaviour for a small subset of the values at one point in time.
struct Proxy
{
Int20_Container& container_; // might not need if a singleton
Int20_Container::size_type index_;
...
};
So, the proxy might be 32, 64 or more bits - the potential benefit is only if you can create them on the fly from indices into the container, have them write directly back into the container, and keep them short-lived with few concurrently. (One simple way - not necessarily the fastest - to implement this model is to use an STL bitset or vector as the Int20_Container, and either store 20 times the logical index in index_, or multiply on the fly.)
It's also vaguely possible that although your values range over a 20-bit space, you've less than say 64k distinct values in actual use. If you have some such insight into your data set, you can create a lookup table where 16-bit array indices map to 20-bit values.
Use a class. As long as you respect the copy/assign/clone/etc... STL semantics, you won't have any problem.
But it will not optimize the memory space on your computer. Especially if you put in in a flat array, the 20bit will likely be aligned on a 32bit boundary, so the benefit of a 20bit type there is useless.
In that case, you will need to define your own optimized array type, that could be compatible with the STL. But don't expect it to be fast. It won't be.
Use a bitfield. (I'm really surprised nobody has suggested this.)
struct int20_and_something_else {
int less_than_a_million : 20;
int less_than_four_thousand : 12; // total 32 bits
};
This only works as a mutual optimization of elements in a structure, where you can spackle the gaps with some other data. But it works very well!
If you truly need to optimize a gigantic array of 20-bit numbers and nothing else, there is:
struct int20_x3 {
int one : 20;
int two : 20;
int three : 20; // 60 bits is almost 64
void set( int index, int value );
int get( int index );
};
You can add getter/setter functions to make it prettier if you like, but you can't take the address of a bitfield, and they can't participate in an array. (Of course, you can have an array of the struct.)
Use as:
int20_x3 *big_array = new int20_x3[ array_size / 3 + 1 ];
big_array[ index / 3 ].set( index % 3, value );
You can use C++ std::bitset. Store everything in a bitset and access your data using the correct index (x20).
Your not going to be able to get exactly 20 bits as a type(even with a bit packed struct), as it will always be aligned (at smallest grainularity) to a byte. Imo the only way to go, if you must have 20 bits, is to create a bitstream to handle the data(which you can overload to accept indexing etc)
You can use the union keyword to create a bit field. I've used it way back when bit fields were a necessity. Otherwise, you can create a class that holds 3 bytes, but through bitwise operations exposes just the most significant 20.
As far as I know that isn't possible.
The easiest option would be to define a custom type, that uses an int32_t as the backing storage, and implements appropriate maths as override operators.
For better storage density, you could store 3 int20 in a single int64_t value.
Just an idea: use optimized storage (5 bytes for two instances), and for operations, convert it into 32-bit int and then back.
While its possible to do this a number of ways.
One possibilty would be to use bit twidling to store them as the left and right parts of a 5 byte array with a class to store/retrieve which converts yoiur desired array entry to an array entry in byte5[] array and extracts the left ot right half as appropriate.
However on most hardware requires integers to be word aligned so as well as the bit twiddling to extract the integer you would need some bit shifiting to align it properly.
I think it would be more efficient to increase your swap space and let virtual memory take care of your large array (after all 20 vs 32 is not much of a saving!) always assuming you have a 64 bit OS.