I am trying write a function that reads PPM images and the function should return the contents.
PPM images have the following text format:
P3
numOfRows numOfColumns
maxColor
numOfRows-by-numOfColumns of RGB colors
Since the text format has a mixture of variable types, is there any way to store this all in an array? I remembered that C++ does not support arrays with different types. If not, then I am thinking of defining a class to store the PPM contents.
C++ does not support arrays with different types.
Correct.
You could:
Define a class as you say, like this: C++ Push Multiple Types onto Vector or this: Creating a vector that holds two different data types or classes or even this: Vector that can have 3 different data types C++.
Have a generic C-like array (or better yet, an std::vector) with void*.
C++ isn't Javascript.
The number of columns / number of rows must be integers. Maximum colour value might be either an integer or a float depending on the format details, as might the rgb values.
So you read the image dimensions first. Then you create a buffer to hold the image. Usually 32 bit rgba is what you want, so either allocate width * height * 4 with malloc() or use an std::vector and resize.
Then you loop through the data, reading the values and putting them into
the array.
Then you create an "Image" object, with integer members of width and height, and a pixel buffer of 32 bit rgbas (or whatever is your preferred pixel format).
Related
I have an audio callback which is constantly writing an array of structures. The stuct has x,y,z members but also have another members after that.
Can I somehow transfer only the pixels of this array without re-arrangeing the array into new one lacking the additional members?
I see that PixelStore() can be set so TexImage2D() misses some bytes at the end of each row but is SKIP_PIXELS what i expect or something else?
Since you're reading from this array, you want UNPACK_SKIP_PIXELS.
Pixel skip is defined in pixels, not bytes or components. If for example your XYZ data is 32 bit float, so one 'pixel' = 12 bytes, the skip factor can only be 1 = 12 bytes, 2 = 24 bytes, etc. So if your struct size isn't a nice multiple of the XYZ size, it won't work.
I suggest glVertexAttribPointer is a better match for what you want to do.
I am working with 3D volumetric images, possibly large (256x256x256). I have 3 such volumes that I want to read in and operate on. Presently, each volume is stored as a text file of numbers which I read in using ifstream. I save it as a matrix (This is a class I have written by dynamic allocation of a 3D array). Then I perform operations on these 3 matrices, addition, multiplication and even Fourier transform. So far, everything works well, but, it takes a hell lot of time, especially the Fourier transform since it has 6 nested loops.
I want to know how I can speed this up. Also, whether the fact that I have stored the images in text files makes a difference. Should I save them as binary or in some other easier/faster to read in format? Is fstream the fastest way I can read in? I use the same 3 matrices each time without changing them. Does that make a difference? Also, is pointer to pointer to pointer the best way to store a 3D volume? If not what else can I do?
Also, is pointer to pointer to pointer best way to store a 3d volume?
Nope thats usually very ineficient.
If not what else can I do?
Its likely that you will get better performance if you store it in a contiguous block, and use computed offsets into the block.
I'd usually use a structure like this:
class DataBlock {
unsigned int nx;
unsigned int ny;
unsigned int nz;
std::vector<double> data;
DataBlock(in_nx,in_ny,in_nz) :
nx(in_nx), ny(in_ny), nz(in_nz) , data(in_nx*in_ny*in_nz, 0)
{}
//You may want to make this check bounds in debug builds
double& at(unsigned int x, unsigned int y, unsigned int z) {
return data[ x + y*nx + z*nx*ny ];
};
const double& at(unsigned int x, unsigned int y, unsigned int z) const {
return data[ x + y*nx + z*nx*ny ];
};
private:
//Dont want this class copied, so remove the copy constructor and assignment.
DataBlock(const DataBlock&);
DataBlock&operator=(const DataBlock&);
};
Storing a large (2563 elements) 3D image file as plain text is a waste of resources.
Without loss of generality, if you have a plain text file for your image and each line of your file consists of one value, you will have to read several characters until you find the end of the line (for a 3-digit number, these will be 4 bytes; 3 bytes for the digits, 1 byte for newline). Afterwards you will have to convert these single digits to a number. When using binary, you directly read a fixed amount of bytes and you will have your number. You could and should write and read it as a binary image.
There are several formats for doing so, the one I would recommend is the meta image file format of VTK. In this format, you have a plain text header file and a binary file with the actual image data. With the information from the header file you will know how large your image is and what datatype you will be using. In your program, you then directly read the binary data and save it to a 3D array.
If you really want to speed things up, use CUDA or OpenCL which will be pretty fast for your applications.
There are several C++ libraries that can help you with writing, saving and manipulating image data, including the before-mentioned VTK and ITK.
2563 is a rather large number. Parsing 2563 text strings will take a considerable amount of time. Using binary will make the reading/writing process much faster because it doesn't require converting a number to/from string, and using much less space. For example to read the number 123 as char from a text file the program will need to read it as a string and convert from decimal to binary using lots of multiplies by 10. Whereas if you had written it directly as the binary value 0b01111011 you only need to read that byte back again into memory, no conversion at all.
Using hexadecimal number may also increase reading speed since each hex digit can map directly to binary value but if you need more speed, binary file is the way to go. Just a fread command is enough to load the whole 2563 bytes = 16MB file into memory in less than 1 sec. And when you're done, just fwrite it back to file. To speedup you can use SIMD (SSE/AVX), CUDA or another parallel processing technique. You can improve the speed even further by multithreading or by only saving the non zero values because in many cases, most values will often be 0's.
Another reason maybe because your array is large and each dimension is a power of 2. This has been discussed in many questions on SO:
Why is there huge performance hit in 2048x2048 versus 2047x2047 array multiplication?
Why is my program slow when looping over exactly 8192 elements?
Why is transposing a matrix of 512x512 much slower than transposing a matrix of 513x513?
You may consider changing the last dimension to 257 and try again. Or better use another algorithm like divide and conquer that's more cache friendly
You should add timers around the load and the process so you know which is taking the most time, and focus your optimization efforts on it. If you control the file format, make one that is more efficient to read. If it is the processing, I'll echo what previous folks have said, investigate efficient memory layout as well as GPGPU computing. Good luck.
I'm encoding N-Dim image cubes into a different image format. I don't know the dimensions of the image until runtime and the library I'm using to read from the original image needs an N-dim array destination buffer as a parameter.
How can I declare such an array in C++? Thanks :)
The short answer is that you cannot declare such an array in C++. The dimensions of an array are part of the type (with a miscellaneous exception that sometimes the value of one of the dimensions can be unknown, for an extern array declaration). The number of dimensions is always part of the type, and the type must be known at compile time.
What you might be able to do instead is to use a "flat" array of appropriate size. For example, if you need a 3x3...x3 array then you can compute 3^n at runtime, dynamically allocate that many int (probably using a vector<int> for convenience), and you have memory with the same layout as an int[3][3]...[3]. You can refer to this memory via a void*.
I'm not certain that it's strictly legal in C++ to alias a flat array as a multi-dimensional array. But firstly the function you're calling might not actually alias it that way anyway given that it also doesn't know the dimension at compile-time. Secondly it will work in practice (if it doesn't, the function you're calling is either broken or else has some cunning way to deal with this that you should find out about and copy).
You can't use array in this case. Array is only for those data whose size and dimension are known at compile time. Try use an array of std::vector instead
I'm profiling the code that I have developed, and I see a bottleneck in my code when I use cvSet2D.
Is there some alternative to cvSet2D more efficient?
How can I write that pixels?
I recommend you to use the C++ functions and not the old C style functions.
The most efficient way to write to pixels is the following.
cv::Mat frame;
// load your img to your Mat
uchar* p = frame.ptr(row); // returns a row pointer
p[column] = x; // accesses value in the given column
One thing to note is that you might have more columns than you have pixel columns, e.g. on a 3 channel image you have 3 times the number of actual pixel columns you can access.
For more information on different ways to iterate over pixels, check this tutorial.
You need to get a pointer to the data field of the structure.
(C API) The IplImage structure has a char* field called data. Access (your_type*)image->data for the first element, and then use it like a regular C 1D array, but be careful to use the field stepWidth to jump from a line to the next (because lines of data may be aligned on multiples of 16 bits for memory access optimization).
(C++ API) Use T* cv::Mat::ptr<T>(int i) to get a pointer to the first element of the line you want to access. Then use as a regular C++ 1D array.
This should be the faster access pattern (see the book OpenCV2 Cookbook for a comparison of the different access patterns).
I know it's ridiculous, but I need it for storage optimization. Is there any good way to implement it in C++?
It has to be flexible enough so that I can use it as a normal data type e.g Vector< int20 >, operator overloading, etc..
If storage is your main concern, I suspect you need quite a few 20-bit variables. How about storing them in pairs? You could create a class representing two such variables and store them in 2.5+2.5 = 5 bytes.
To access the variables conveniently you could override the []-operator so you could write:
int fst = pair[0];
int snd = pair[1];
Since you may want to allow for manipulations such as
pair[1] += 5;
you would not want to return a copy of the backing bytes, but a reference. However, you can't return a direct reference to the backing bytes (since it would mess up it's neighboring value), so you'd actually need to return a proxy for the backing bytes (which in turn has a reference to the backing bytes) and let the proxy overload the relevant operators.
As a metter of fact, as #Tony suggest, you could generalize this to have a general container holding N such 20-bit variables.
(I've done this myself in a specialization of a vector for efficient storage of booleans (as single bits).)
No... you can't do that as a single value-semantic type... any class data must be a multiple of the 8-bit character size (inviting all the usual quips about CHAR_BITS etc).
That said, let's clutch at straws...
Unfortunately, you're obviously handling very many data items. If this is more than 64k, any proxy object into a custom container of packed values will probably need a >16 bit index/handle too, but still one of the few possibilities I can see worth further consideration. It might be suitable if you're only actively working with and needing value semantic behaviour for a small subset of the values at one point in time.
struct Proxy
{
Int20_Container& container_; // might not need if a singleton
Int20_Container::size_type index_;
...
};
So, the proxy might be 32, 64 or more bits - the potential benefit is only if you can create them on the fly from indices into the container, have them write directly back into the container, and keep them short-lived with few concurrently. (One simple way - not necessarily the fastest - to implement this model is to use an STL bitset or vector as the Int20_Container, and either store 20 times the logical index in index_, or multiply on the fly.)
It's also vaguely possible that although your values range over a 20-bit space, you've less than say 64k distinct values in actual use. If you have some such insight into your data set, you can create a lookup table where 16-bit array indices map to 20-bit values.
Use a class. As long as you respect the copy/assign/clone/etc... STL semantics, you won't have any problem.
But it will not optimize the memory space on your computer. Especially if you put in in a flat array, the 20bit will likely be aligned on a 32bit boundary, so the benefit of a 20bit type there is useless.
In that case, you will need to define your own optimized array type, that could be compatible with the STL. But don't expect it to be fast. It won't be.
Use a bitfield. (I'm really surprised nobody has suggested this.)
struct int20_and_something_else {
int less_than_a_million : 20;
int less_than_four_thousand : 12; // total 32 bits
};
This only works as a mutual optimization of elements in a structure, where you can spackle the gaps with some other data. But it works very well!
If you truly need to optimize a gigantic array of 20-bit numbers and nothing else, there is:
struct int20_x3 {
int one : 20;
int two : 20;
int three : 20; // 60 bits is almost 64
void set( int index, int value );
int get( int index );
};
You can add getter/setter functions to make it prettier if you like, but you can't take the address of a bitfield, and they can't participate in an array. (Of course, you can have an array of the struct.)
Use as:
int20_x3 *big_array = new int20_x3[ array_size / 3 + 1 ];
big_array[ index / 3 ].set( index % 3, value );
You can use C++ std::bitset. Store everything in a bitset and access your data using the correct index (x20).
Your not going to be able to get exactly 20 bits as a type(even with a bit packed struct), as it will always be aligned (at smallest grainularity) to a byte. Imo the only way to go, if you must have 20 bits, is to create a bitstream to handle the data(which you can overload to accept indexing etc)
You can use the union keyword to create a bit field. I've used it way back when bit fields were a necessity. Otherwise, you can create a class that holds 3 bytes, but through bitwise operations exposes just the most significant 20.
As far as I know that isn't possible.
The easiest option would be to define a custom type, that uses an int32_t as the backing storage, and implements appropriate maths as override operators.
For better storage density, you could store 3 int20 in a single int64_t value.
Just an idea: use optimized storage (5 bytes for two instances), and for operations, convert it into 32-bit int and then back.
While its possible to do this a number of ways.
One possibilty would be to use bit twidling to store them as the left and right parts of a 5 byte array with a class to store/retrieve which converts yoiur desired array entry to an array entry in byte5[] array and extracts the left ot right half as appropriate.
However on most hardware requires integers to be word aligned so as well as the bit twiddling to extract the integer you would need some bit shifiting to align it properly.
I think it would be more efficient to increase your swap space and let virtual memory take care of your large array (after all 20 vs 32 is not much of a saving!) always assuming you have a 64 bit OS.