I have a matrix array of doubles that I need to store into an array of chars. These 32-bit double values are guaranteed to be small enough to fit into an 8-bit char value. (The maximum double value in my program is 31). I've researched a bit and what I find are solutions to store a double as a char*, in other words convert a double to a c_string. This is NOT what I seek to achieve.
// I'm dealing with a 15*4 double array
double **d_array = new double*[15];
d_array[i] = new double[4];
// This creates a char array (That will have > 15*4 spaces)
unsigned char *c_array = new unsigned char [1024];
I can iterate a loop over the double matrix to store to the character matrix.
Say I had d_array[1][0] = 4. I want to have c_array[5] = 4. Because 4 is 00000100, it should be able to fit.
I think you should be able to just make the assignment in your loop and it will automatically be truncated and converted (you may get a compiler warning):
c_array[0] = d_array[0][0];
To be safe, you could do
c_array[0] = (char)(int)d_array[0][0];
You may want to use uint8_t since a char is either signed, unsigned or char.
You can use static_cast:
uint8_t value = static_cast<uint8_t>(d_array[i][j]);
If you want to copy the bytes of a floating point to a buffer:
uint8_t buffer[4096];
float f_value = 3.14;
uint8_t * p_float = static_cast<uint8_t *>(&f_value);
for (unsigned int i = 0; i < sizeof(float); ++i)
{
buffer[index + i] = p_float[i];
}
I am trying to read data from .wav and put it to fft.
To read wav file I am using sndfile library.
SNDFILE* infile;
SF_INFO sfinfo ;
memset (&sfinfo, 0, sizeof (sfinfo)) ;
infile = sf_open ("sound.wav", SFM_READ, &sfinfo);
double data [BUF_SIZE];
while (readcount = (int)sf_readf_double (infile, data, BUF_SIZE))
{
for (int i = 0; i < readcount; i++)
{
cout << data[i] << " ";
}
}
But every values in this (and other files) are between (-1 ; 1).
Is this correct? Why every values are so small? I was expected to read amplitude in time domain (volume of sound).
This is the canonical format of floating point samples. With float values, you get full 32-bit precision. Clipping is also easy to represent. If a sample value is higher than 1 or lower than -1, it means the sample clipped. With integer values, there's no way to know that.
Floating point is also an easy sample format to apply operations to. Mixing for example is trivial (you just add the sample values together.)
So even if it looks weird at first, it is the best format for audio sample representation. Once you applied the operations you need to the float values, you then convert them to the format you want for output (like 16-bit integers.) This operation is trivial. Here's a function that converts and clips float samples to any known integer sample format in use today:
#include <limits>
/* Convert and clip a float sample to an integer sample. This works for
* all usual integer sample types (8-bit, 16-bit, 32-bit, signed or
* unsigned.)
*/
template <typename T>
T floatSampleToInt(float src) noexcept
{
if (src >= 1.f)
return std::numeric_limits<T>::max();
if (src < -1.f)
return std::numeric_limits<T>::min();
return src * (float)(1UL << (sizeof(T) * 8 - 1))
+ ((float)(1UL << (sizeof(T) * 8 - 1))
+ (float)std::numeric_limits<T>::min());
}
If you want to convert a float sample to a signed 16-bit integer sample for example, you do:
int16_t intSample = floatSampleToInt<int16_t>(floatSample);
Note that 24-bit integer samples are covered by 32-bit. A 32-bit sample is also a valid 24-bit sample; its lower 8 bits are just truncated.
Supposing I am given an image of 2048x2048 and i want to know the total number of colors present in the image, what is the fastest possible algorithm? I came up with two algorithm but they are slow.
Algorithm 1:
Compare the current pixel an the next pixel and if they are different
Check a temporary variable, which contains all the detected colors, to see if the color is present or not
If not present add it to the array(List) and increment noOfColors.
This Algorithm works but is slow. For a 1600x1200 pixels image it takes around 3 sec.
Algorithm 2:
The obvious method of checking the each pixel with all other pixels and recording the no of occurences of the color and incrementing the count. This is very very slow, almost like a hung app. So is there any better approach? I need all the pixel info.
You could use std::set (or std::unordered_set), and simply do a single loop though the pixels, adding the colors to the set. Then the number of colors is the size of the set.
Well, this is suited for parallelization. Split the image in several parts and execute the algorithm for each part in a separate task. To avoid syncing each should have its own storage for the unique colors. When all tasks are done, you aggregate the results.
DRAM is dirt cheap. Use brute force. Fill a tab, count.
On a core2duo # 3.0GHz :
0.35secs for 4096x4096 32 bits rgb
0.20secs after some trivial parallelization (I do know nothing of omp)
However, if you are to use 64bit rgb (one channel = 16 bits) it is another question (not enough memory).
You shall probably need a good hash table function.
Using random pixels, same size takes 10 secs.
Remark: at 0.15 secs, the std::bitset<> solution is faster (it gets slower trivially parallelized !).
Solution, c++11
#include <vector>
#include <random>
#include <iostream>
#include <boost/chrono.hpp>
#define _16M 256*256*256
typedef union {
struct { unsigned char r,g,b,n ; } r_g_b_n ;
unsigned char rgb[4] ;
unsigned i_rgb;
} RGB ;
RGB make_RGB(unsigned char r, unsigned char g , unsigned char b) {
RGB res;
res.r_g_b_n.r = r;
res.r_g_b_n.g = g;
res.r_g_b_n.b = b;
res.r_g_b_n.n = 0;
return res;
}
static_assert(sizeof(RGB)==4,"bad RGB size not 4");
static_assert(sizeof(unsigned)==4,"bad i_RGB size not 4");
struct Image
{
Image (unsigned M, unsigned N) : M_(M) , N_(N) , v_(M*N) {}
const RGB* tab() const {return & v_[0] ; }
RGB* tab() {return & v_[0] ; }
unsigned M_ , N_;
std::vector<RGB> v_;
};
void FillRandom(Image & im) {
std::uniform_int_distribution<unsigned> rnd(0,_16M-1);
std::mt19937 rng;
const int N = im.M_ * im.N_;
RGB* tab = im.tab();
for (int i=0; i<N; i++) {
unsigned r = rnd(rng) ;
*tab++ = make_RGB( (r & 0xFF) , (r>>8 & 0xFF), (r>>16 & 0xFF) ) ;
}
}
size_t Count(const Image & im) {
const int N = im.M_ * im.N_;
std::vector<char> count(_16M,0);
const RGB* tab = im.tab();
#pragma omp parallel
{
#pragma omp for
for (int i=0; i<N; i++) {
count[ tab->i_rgb ] = 1 ;
tab++;
}
}
size_t nColors = 0 ;
#pragma omp parallel
{
#pragma omp for
for (int i = 0 ; i<_16M; i++) nColors += count[i];
}
return nColors;
}
int main() {
Image im(4096,4096);
FillRandom(im);
typedef boost::chrono::high_resolution_clock hrc;
auto start = hrc::now();
std::cout << " # colors " << Count(im) << std::endl ;
boost::chrono::duration<double> sec = hrc::now() - start;
std::cout << " took " << sec.count() << " seconds\n";
return 0;
}
The only feasible algorithm here is building a sort of a histogram of the image colors. The only difference in your case is that instead of calculating the population of each color you need just to know if it's zero or not.
Depending on which color space you work, you may use either an std::set to tag existing colors (as Joachim Pileborg suggested), or just use something like std::bitset, which is obviously faster. This depends on how much distinct colors exist in your color-space.
Also, like Marius Bancila noted, this procedure is a perfect match for parallelization. Calculated the histogram-like data for image parts, and then merge it. Naturally the image division should be based on its memory partition, not the geometric properties. In simple words - split the image vertically (by batches of scan lines), not horizontally.
And, if possible, you should either use some low-level library/code to run through pixels, or try to write your own. At least you must obtain a pointer to scan line and run on its pixels in a batch, rather than doing something like GetPixel for each pixel.
The point, here, is that the ideal representation of an image as 2D array of colors is not the one that happens the way the image is stored on memory (color components can be arranged in "planes", there could be "padding" etc. So getting the pixels using a GetPixel-like function may take time.
The question, then, may even be somehow meaningless if the image is not the result of a "vectorial draw": think to a photograph: between two nearby "greens" you find all the shade of green, so the colors -in this case- are no more no less the ones supported by the encoding of the image itself (2^24, or 256, or 16 or ...), so, unless you are interested on the color distribution (how differently used they are), just counting them makes very few sense.
A workaround can be:
Create an in-memory bitmap having pixel in a "single plane format"
Blit your image into that bitmap using BitBlt or similar (this let the OS to make pixel
conversion from the GPU,if any)
Get the bitmap-bits (this lets you
access the stored values)
Play your "counting algorithm" (whatever
it is) onto those values.
Note that step 1 and 2 can be avoided if you already know that the image is already in planar format.
If you have a multicore system, step 4 can also be assigned to different threads, each working part of the image.
You can use bitset which allows you to set individual bits and has a count function.
You have a bit for each colour, there are 256 values for each of RGB, so that's 256*256*256 bits (16,777,216 colours). The bitset will use a byte for every 8 bits so it will use 2MB.
Use the pixel colour as an index into the bitset:
bitset<256*256*256> colours;
for(int pixel: pixels) {
colours[pixel] = true;
}
colours.count();
This has linear complexity.
Late comer to this answer, but could not help it since this algorithm is brutally fast, developed about 2 or more decades ago, when it really mattered.
3-D Lookup Table Color Matching
http://www.ddj.com/cpp/184403257
Basically, it creates a 3d color loop up table and the search is very fast, I've done some modifications to suit my purpose for image binarization, so I reduced the color space from ff ff ff to f f f, and it's even 10 times faster. As it is right out of the box, I haven't found anything even close, including hash tables.
char * creatematcharray(struct rgb_color *palette, int palettesize)
{
int rval=16, gval=16, bval=16, len, r, g, b;
char *taken, *match, *same;
int i, set, sqstep, tp, maxtp, *entryr, *entryg, *entryb;
char *table;
len=rval*gval*bval;
// Prepare table buffers:
size_t size_of_table = len*sizeof(char);
table=(char *)malloc(size_of_table);
if (table==nullptr) return nullptr;
// Select colors to use for fill:
set=0;
size_t size_of_taken = (palettesize * sizeof(int) * 3) +
(palettesize*sizeof(char)) + (len * sizeof(char));
taken=(char *)malloc(size_of_taken);
same=taken + (len * sizeof(char));
entryr=(int*)(same + (palettesize * sizeof(char)));
entryg=entryr + palettesize;
entryb=entryg + palettesize;
if (taken==nullptr)
{
free((void *)table);
return nullptr;
}
std::memset((void *)taken, 0, len * sizeof(char));
// std::cout << "sizes: " << size_of_table << " " << size_of_taken << std::endl;
match=table;
for (i=0; i<palettesize; i++)
{
same[i]=0;
// Compute 3d-table coordinates of palette rgb color:
r=palette[i].r&0x0f, g=palette[i].g&0x0f, b=palette[i].b&0x0f;
// Put color in position:
if (taken[b*rval*gval+g*rval+r]==0) set++;
else same[match[b*rval*gval+g*rval+r]]=1;
match[b*rval*gval+g*rval+r]=i;
taken[b*rval*gval+g*rval+r]=1;
entryr[i]=r; entryg[i]=g; entryb[i]=b;
}
// ### Fill match_array by steps: ###
for (set=len-set, sqstep=1; set>0; sqstep++)
{
for (i=0; i<palettesize && set>0; i++)
if (same[i]==0)
{
// Fill all six sides of incremented cube (by pairs, 3 loops):
for (b=entryb[i]-sqstep; b<=entryb[i]+sqstep; b+=sqstep*2)
if (b>=0 && b<bval)
for (r=entryr[i]-sqstep; r<=entryr[i]+sqstep; r++)
if (r>=0 && r<rval)
{ // Draw one 3d line:
tp=b*rval*gval+(entryg[i]-sqstep)*rval+r;
maxtp=b*rval*gval+(entryg[i]+sqstep)*rval+r;
if (tp<b*rval*gval+0*rval+r)
tp=b*rval*gval+0*rval+r;
if (maxtp>b*rval*gval+(gval-1)*rval+r)
maxtp=b*rval*gval+(gval-1)*rval+r;
for (; tp<=maxtp; tp+=rval)
if (!taken[tp])
taken[tp]=1, match[tp]=i, set--;
}
for (g=entryg[i]-sqstep; g<=entryg[i]+sqstep; g+=sqstep*2)
if (g>=0 && g<gval)
for (b=entryb[i]-sqstep; b<=entryb[i]+sqstep; b++)
if (b>=0 && b<bval)
{ // Draw one 3d line:
tp=b*rval*gval+g*rval+(entryr[i]-sqstep);
maxtp=b*rval*gval+g*rval+(entryr[i]+sqstep);
if (tp<b*rval*gval+g*rval+0)
tp=b*rval*gval+g*rval+0;
if (maxtp>b*rval*gval+g*rval+(rval-1))
maxtp=b*rval*gval+g*rval+(rval-1);
for (; tp<=maxtp; tp++)
if (!taken[tp])
taken[tp]=1, match[tp]=i, set--;
}
for (r=entryr[i]-sqstep; r<=entryr[i]+sqstep; r+=sqstep*2)
if (r>=0 && r<rval)
for (g=entryg[i]-sqstep; g<=entryg[i]+sqstep; g++)
if (g>=0 && g<gval)
{ // Draw one 3d line:
tp=(entryb[i]-sqstep)*rval*gval+g*rval+r;
maxtp=(entryb[i]+sqstep)*rval*gval+g*rval+r;
if (tp<0*rval*gval+g*rval+r)
tp=0*rval*gval+g*rval+r;
if (maxtp>(bval-1)*rval*gval+g*rval+r)
maxtp=(bval-1)*rval*gval+g*rval+r;
for (; tp<=maxtp; tp+=rval*gval)
if (!taken[tp])
taken[tp]=1, match[tp]=i, set--;
}
}
}
free((void *)taken);`enter code here`
return table;
}
The answer: unordered_map
I use unordered_map, based on my testing.
You should test because your compiler / library may exhibit different performance Comment out #define USEHASH to use map instead.
On my machine, the vanilla unordered_map (a hash implementation) is about twice as fast as map. Inasmuch as different compilers, libraries can vary enormously, you must test to see which is better. In production, I build a fake image on first start of the app, run both algorithms on it and time them, save an indication of which one is faster, and then preferentially use that for all subsequent starts on that the machine. It's nit-picky, but hey, the user's time is valuable to them.
For a DSLR image with 12,106,244 pixels (about 12 megapixels, not a typo) and 11,857,131 distinct colors (also not a typo), map takes about 14 seconds, while unordered map takes about 7 seconds:
Test Code:
#define USEHASH 1
#ifdef USEHASH
#include <unordered_map>
#endif
size = im->xw * im->yw;
#ifdef USEHASH
// unordered_map is about twice as fast as map on my mac with qt5
// --------------------------------------------------------------
#include <unordered_map>
std::unordered_map<qint64, unsigned char> colors;
colors.reserve(size); // pre-allocate the hash space
#else
std::map<qint64, unsigned char> colors;
#endif
...use of either is in a loop where I build a 48-bit value of 0RGB in a 64-bit variable corresponding to the 16-bit RGB values of the image pixels, like so:
for (i=0; i<size; i++)
{
pel = BUILDPEL(i); // macro just shovels 0RGB into 64 bit pel from im
// You'd do the same for your image structure
// in whatever way is fastest for you
colors[pel] = 1;
}
cc = colors.size();
// time here: 14 secs for map, 7 secs for unordered_map with
// 12,106,244 pixels containing 11,857,131 colors on 12/24 core,
// 3 GHz, 64GB machine.
I have been searching along for a way to compress, using the zlib library (and the function compress) a struct containing float vars.
Every example I saw are showing how to compress a string, specifically an unsigned char*.
My struct is an easy one :
struct Particle{
float x;
float y;
float z;
};
And I am calling the compress function as below :
uLong initSize = sizeof(Particle);
uLongf destSize = initSize * 1.1 + 12;
Bytef *dataOriginal = (Bytef*)malloc( initSize );
Bytef *dataCompressed = (Bytef*)malloc( destSize );
Particle p;
memset( &p, 0, sizeof(Particle) );
p.x = 10.24;
p.y = 23.5;
p.z = 7.4;
memcpy( dataOriginal, &p, sizeof(p) );
compress( dataCompressed, &destSize, dataOriginal, initSize );
But when I try to uncompress my data to see what inside, I can't get back to my initial float value :
Bytef *decomp = (Bytef*)malloc( initSize );
uncompress( decomp, &initSize, dataCompressed, destSize );
for( int i = 0 ; i < initSize ; i++ ){
std::cout << (float)decomp[i] << std::endl;
}
If anyone have a solution to this problem, I'm on it since 2 days now...
You would need to copy the decompressed data back into the Particle struct, just like you copied it out in the first place. (Or you could just use casts instead of copies). Then you will recover the original floats in the struct. Whatever it is you think you're doing with 'decomp[i]` doesn't make any sense.
However there are several problems with this. First, this is only assured to work on the same machine, with the same compiler, and even then only within the same version of the compiler. If a different compiler or different version chooses to align the structure differently, then the compressed data will not be transferrable between the two. If there is a different representation of floats between different machines, the compressed data will not be transferrable.
Furthermore, you will not get any compression when compressing three floats. I presume that this is just a prelude to compressing a large array of such Particle structs. Then maybe you'll get somewhere with this.
Better would be to first convert the floats to the precision needed as integers. You should know the range and the useful number of bits for your application. This will compress before even using compress(), by using only the number of bits needed as opposed to 32 per float. Then convert those integers portably to a series of bytes with shift operations. You can then also apply differencing to successive Particles (e.g. x1-x2, y1-y2, z1-z2), which might improve compression if there is a correlation between successive Particles.
By the way, instead of * 1.1 + 12, you should use compressBound(), which does exactly what you want in a way that is assured by the zlib library for future versions.
Is there a way to convert a std::bitset<64> to a double without using any external library (Boost, etc.)? I am using a bitset to represent a genome in a genetic algorithm and I need a way to convert a set of bits to a double.
The C++11 road:
union Converter { uint64_t i; double d; };
double convert(std::bitset<64> const& bs) {
Converter c;
c.i = bs.to_ullong();
return c.d;
}
EDIT: As noted in the comments, we can use char* aliasing as it is unspecified instead of being undefined.
double convert(std::bitset<64> const& bs) {
static_assert(sizeof(uint64_t) == sizeof(double), "Cannot use this!");
uint64_t const u = bs.to_ullong();
double d;
// Aliases to `char*` are explicitly allowed in the Standard (and only them)
char const* cu = reinterpret_cast<char const*>(&u);
char* cd = reinterpret_cast<char*>(&d);
// Copy the bitwise representation from u to d
memcpy(cd, cu, sizeof(u));
return d;
}
C++11 is still required for to_ullong.
Most people are trying to provide answers that let you treat the bit-vector as though it directly contained an encoded int or double.
I would advise you completely avoid that approach. While it does "work" for some definition of working, it introduces hamming cliffs all over the place. You usually want your encoding to arrange things so that if two decoded values are near to one another, then their encoded values are near to one another as well. It also forces you to use 64-bits of precision.
I would manage the conversion manually. Say you have three variables to encode, x, y, and z. Your domain expertise can be used to say, for example, that -5 <= x < 5, 0 <= y < 100, and 0 <= z < 1, where you need 8 bits of precision for x, 12 bits for y, and 10 bits for z. This gives you a total search space of only 30 bits. You can have a 30 bit string, treat the first 8 as encoding x, the next 12 as y, and the last 10 as z. You are also free to gray code each one to remove the hamming cliffs.
I've personally done the following in the past:
inline void binary_encoding::encode(const vector<double>& params)
{
unsigned int start=0;
for(unsigned int param=0; param<params.size(); ++param) {
// m_bpp[i] = number of bits in encoding of parameter i
unsigned int num_bits = m_bpp[param];
// map the double onto the appropriate integer range
// m_range[i] is a pair of (min, max) values for ith parameter
pair<double,double> prange=m_range[param];
double range=prange.second-prange.first;
double max_bit_val=pow(2.0,static_cast<double>(num_bits))-1;
int int_val=static_cast<int>((params[param]-prange.first)*max_bit_val/range+0.5);
// convert the integer to binary
vector<int> result(m_bpp[param]);
for(unsigned int b=0; b<num_bits; ++b) {
result[b]=int_val%2;
int_val/=2;
}
if(m_gray) {
for(unsigned int b=0; b<num_bits-1; ++b) {
result[b]=!(result[b]==result[b+1]);
}
}
// insert the bits into the correct spot in the encoding
copy(result.begin(),result.end(),m_genotype.begin()+start);
start+=num_bits;
}
}
inline void binary_encoding::decode()
{
unsigned int start = 0;
// for each parameter
for(unsigned int param=0; param<m_bpp.size(); param++) {
unsigned int num_bits = m_bpp[param];
unsigned int intval = 0;
if(m_gray) {
// convert from gray to binary
vector<int> binary(num_bits);
binary[num_bits-1] = m_genotype[start+num_bits-1];
intval = binary[num_bits-1];
for(int i=num_bits-2; i>=0; i--) {
binary[i] = !(binary[i+1] == m_genotype[start+i]);
intval += intval + binary[i];
}
}
else {
// convert from binary encoding to integer
for(int i=num_bits-1; i>=0; i--) {
intval += intval + m_genotype[start+i];
}
}
// convert from integer to double in the appropriate range
pair<double,double> prange = m_range[param];
double range = prange.second - prange.first;
double m = range / (pow(2.0,double(num_bits)) - 1.0);
// m_phenotype is a vector<double> containing all the decoded parameters
m_phenotype[param] = m * double(intval) + prange.first;
start += num_bits;
}
}
Note that for reasons that probably don't matter to you, I wasn't using bit vectors -- just ordinary vector<int> to encoding things. And of course, there's a bunch of stuff tied into this code that isn't shown here, but you can probably get the basic idea.
One other note, if you're doing GPU calculations or if you have a particular problem such that 64 bits are the appropriate size anyway, it may be worth the extra overhead to stuff everything into native words. Otherwise, I would guess that the overhead you add to the search process will probably overwhelm whatever benefits you get by faster encoding and decoding.
Edit:: I've decided that I was being a bit silly with this. While you do end up with a double it assumes that the bitset holds an integer... which is a big assumption to make. You will end up with a predictable and repeatable value per bitset but still I don't think that this is what the author intended.
Well if you iterate over the bit values and do
output_double += pow( 2, 64-(bit_position+1) ) * bit_value;
That would work. As long as it is big-endian