Binary files: write with C++, read with MATLAB - c++

I could use your support on this. Here is my issue:
I've got a 2D buffer of floats (in a data object) in a C++ code, that I write in a binary file using:
ptrToFile.write(reinterpret_cast<char *>(&data->array[0][0]), nbOfEltsInArray * sizeof(float));
The data contains 8192 floats, and I (correctly ?) get a 32 kbytes (8192 * 4 bytes) file out of this line of code.
Now I want to read that binary file using MATLAB. The code is:
hdr_binaryfile = fopen(str_binaryfile_path,'r');
res2_raw = fread(hdr_binaryfile, 'float');
res2 = reshape(res2_raw, int_sizel, int_sizec);
But it's not happening as I expect it to happen. If I print the array of data in the C++ code using std::cout, I get:
pCarte_bin->m_size = 8192
pCarte_bin->m_sizel = 64
pCarte_bin->m_sizec = 128
pCarte_bin->m_p[0][0] = 1014.97
pCarte_bin->m_p[0][1] = 566946
pCarte_bin->m_p[0][2] = 423177
pCarte_bin->m_p[0][3] = 497375
pCarte_bin->m_p[0][4] = 624860
pCarte_bin->m_p[0][5] = 478834
pCarte_bin->m_p[1][0] = 2652.25
pCarte_bin->m_p[2][0] = 642077
pCarte_bin->m_p[3][0] = 5.33649e+006
pCarte_bin->m_p[4][0] = 3.80922e+006
pCarte_bin->m_p[5][0] = 568725
And on the MATLAB side, after I read the file using the little block of code above:
size(res2) = 64 128
res2(1,1) = 1014.9659
res2(1,2) = 323288.4063
res2(1,3) = 2652.2515
res2(1,4) = 457593.375
res2(1,5) = 642076.6875
res2(1,6) = 581674.625
res2(2,1) = 566946.1875
res2(3,1) = 423177.1563
res2(4,1) = 497374.6563
res2(5,1) = 624860.0625
res2(6,1) = 478833.7188
The size (lines, columns) is OK, as well as the very first item ([0][0] in C++ == [1][1] in MATLAB). But:
I'm reading the C++ line elements along the columns: [0][1] in C++ == [1][2] in MATLAB (remember that indexing starts at 1 in MATLAB), etc.
I'm reading one correct element out of two along the other dimension: [1][0] in C++ == [1][3] in MATLAB, [2][0] == [1][5], etc.
Any idea about this ?
Thanks!
bye

Leaving aside the fact there seems to be some precision difference (likely the display settings in MATLAB) the issue here is likely the difference between row major and column major ordering of data. Without more details it will be hard to be certain. In particular MATLAB is column major meaning that contiguous memory on disk is interpreted as detailing sequential elements in a column rather than a row.
The likely solution is to reverse the two sizes in your reshape, and access the elements with indices reversed. That is, swap the int_size1 and int_size2, and then read elements expecting
pCarte_bin->m_p[0][0] = res2(1,1)
pCarte_bin->m_p[0][1] = res2(2,1)
pCarte_bin->m_p[0][2] = res2(3,1)
pCarte_bin->m_p[0][3] = res2(4,1)
pCarte_bin->m_p[1][0] = res2(1,2)
etc.
You could also transpose the array in MATLAB after read, but for a large array that could be costly in itself

Related

How can I read from a .txt file to determine the size of a 2-D array?

I am working on a flood-fill recursion assignment where I have to read an ASCII art text file and fill it in. The assignment can be found here: https://faculty.utrgv.edu/robert.schweller/CS2380/homework/hw10.pdf
Recursion() //construcor
{
column = -1;
row = -1;
grid = new char*[size of art row];
for(int i = 0; i < size of art; i++)
{
board[row] = new char[size of art column]
}
}
I'm not sure if determining the size of the array should be in the constructor or not. I need to know the size of the array in order to know where the user wants to fill the art file in. Also, here is all of the code for a better context. https://pastebin.com/TSYH26Ci
I would handle your file as binary. I do not know which OS,API you are using so I will answer just in generic way...
get file size siz
usually seeking to 0 bytes from end of file will give you the file size.
allocate 1D array dat to store your entire file
dat = new BYTE[siz];
load your file into memory (1D array)
do not forget to use binary access as some ASCII arts could use control codes (ASCII below 32) which could be corrupted by text file access.
scan for end of line
so scan your array from 0 and stop when you find ASCII codes 13 or 10. Its position will give you the x resolution of your ASCII art
int xs;
for (xs=0;xs<siz;xs++)
if ((dat[xs]==10)||(dat[xs]==13))
break;
now xs should be holding your x resolution.
compute y resolution ys
the safest way is to count the number of end of lines (13 or 10). In such case you can even store the line start addresses to some pointer array BYTE **pixel=new (BYTE*)[ys]; which will enable you simple 2D access pixel[y][x]. If your ASCII art is aligned and have constant size per each line than you can compute ys from size..
ys = siz/(xs+eol_size)
where eol_size is 1 or 2 depending on the used line ending: ((10),(13),(13,10) or (10,13)) so:
eol_size=1;
if (xs<siz)
if ((dat[xs+1]==10)||(dat[xs+1]==13))
eol_size=2;
As we do not have access to any input file we can only guess... If you need to generate one see:
C++ Image to ASCII art conversion
Here example of binary file access in VCL (bullets #1,#2,#3):
Convert the Linux open, read, write, close functions to work on Windows

Why won't H5PYDataset.get_data() work within function?

I have a function which is supposed to unpack an H5PY dataset, temp.hdf5, which only contains one example so it can be evaluated:
def getprob():
test_set = H5PYDataset('temp.hdf5', which_sets=('test',))
handle = test_set.open()
test_data = test_set.get_data(handle, slice(0,1))
xx = test_data[0]
YY = test_data[1]
l, prob, rho_orig, rho_larger, rho_largest = f(xx)
return prob[9][0]
Where test_data[0] is 28x28 array of integers and test_data[1] is an integer between 0 and 9.
The problem is that, from within the function, test_data[0] is always a 28x28 array of zeros, even though it is not within 'temp.hdf5'. test_data[1] always loads properly, though.
When these lines of code are run outside of the function, everything works just fine.
What is going on here?

How can I create an array from a messy text file

I have a text file in the form below...
Some line of text
Some line of text
Some line of text
--
data entry 0 (i = 0, j = 0); value = 1.000000
data entry 1 (i = 0, j = 1); value = 1.000000
data entry 2 (i = 0, j = 2); value = 1.000000
data entry 3 (i = 0, j = 3); value = 1.000000
etc for quite a large number of lines. The total array ends up being 433 rows x 400 columns. There is a line of hyphens -- separating each new i value. So far I have the following code:
f = open('text_file_name', 'r')
lines = f.readlines()
which is simply opening the file and converting it to a list with each line as a separate string. I need to be able create an array with the given values for i and j positions - let's call the array A. The value of A[0,0] should be 1.000000. I don't know how I can get from a messy text file (at the stage I am, messy list) to a usable array
EDIT:
The expected output is a NumPy array. If I can get to that point, I can work through the rest of the tasks in the problem
UPDATE:
Thank you, Lukasz, for the suggestion below. I sort of understand the code you wrote, but I don't understand it well enough to use it. However, you have given me some good ideas on what to do. The data entries begin on line 12 of the text file. Values for i are within the 22nd and 27th character places, values for j are within the 33rd and 39th character places, and values for value are within the 49th and 62nd character places. I realize this is overly specific for this particular text file, but my professor is fine with that.
Now, I've written the following code using the formatting of this text file
for x in range(12,len(lines)):
if not lines[x].startswith(' data entry'):
continue
else:
i = int(lines[x][22:28])
j = int(lines[x][33:39])
r = int(lines[x][49:62])
matrix[i,j] = r
print matrix
and the following ValueError message is given:
r = int(lines[x][49:62])
ValueError: invalid literal for int() with base 10: '1.000000'
Can anyone explain why this is given (I should be able to convert the string '1.000000' to integer 1) and what I can do to correct the issue?
You may simply skip all lines that does not look like data line.
For retrieving indices simple regular expression is introduced.
import numpy as np
import re
def parse(line):
m = re.search('\(i = (\d+), j = (\d+)\); value = (\S+)', line)
if not m:
raise ValueError("Invalid line", line)
return int(m.group(1)), int(m.group(2)), float(m.group(3))
R = 433
C = 400
data_file = 'file.txt'
matrix = np.zeros((R, C))
with open(data_file) as f:
for line in f:
if not line.startswith('data entry'):
continue
i, j, v = parse(line)
matrix[i, j] = v
print matrix
Main trouble here is hardcoded matrix size. Ideally you' somehow detect a size of destination matrix prior to reading data, or use other data structure and rebuild numpy array from said structure.

Multidimensional array with different sizes using Arduino

I would like to have a multidimensional array that allows for different sizes.
Example:
int x[][][] = {{{1,2},{2,3}},{{1,2}},{{4,5},{2,7},{1,1}}};
The values will be known at compile time and will not change.
I would like to be able to access the values like val = x[2][0][1];
What is the best way to go about this? I'm used to java/php where doing something like this is trivial.
Thanks
I suppose you could do this "the old fashioned (uphill both ways) way":
#include <stdio.h>
int main(void){
int *x[3][3];
int y[12] = {1,2,3,4,5,6,7,8,9,10,11,12};
x[0][0] = &y[0];
x[0][1] = &y[2];
x[1][0] = &y[4];
x[2][0] = &y[6];
x[2][1] = &y[8];
x[2][2] = &y[10];
// testing:
printf("x[0][0][0] = %d\n", x[0][0][0]);
printf("x[0][0][1] = %d\n", x[0][0][1]);
printf("x[0][1][0] = %d\n", x[0][1][0]);
printf("x[0][1][1] = %d\n", x[0][1][1]);
printf("x[1][0][0] = %d\n", x[1][0][0]);
printf("x[1][0][1] = %d\n", x[1][0][1]);
printf("x[2][0][0] = %d\n", x[2][0][0]);
printf("x[2][0][1] = %d\n", x[2][0][1]);
printf("x[2][1][0] = %d\n", x[2][1][0]);
printf("x[2][1][1] = %d\n", x[2][1][1]);
printf("x[2][2][1] = %d\n", x[2][2][0]);
printf("x[2][2][1] = %d\n", x[2][2][1]);
return 0;
}
Basically, the array x is a little bit too big (3x3) and it points to the "right place" in the array y that contains your data (I am using the digits 1…12 because it's easier to see it is doing the right thing). For a small example like this, you end up with an array of 9 pointers in x (72 bytes), plus the 12 integers in y (48 bytes).
If you filled an int array with zeros where you didn't need values (or -1 if you wanted to indicate "invalid") you would end up with 18x4 = 72 bytes. So the above method is less efficient - because this array is not "very sparse". As you change the degree of raggedness, this gets better. If you really wanted to be efficient you would have an array of pointers-of-pointers, followed by n arrays of pointers - but this gets very messy very quickly.
Very often the right approach is a tradeoff between speed and memory size (which is always at a premium on the Arduino).
By the way - the above code does indeed produce the output
x[0][0][0] = 1
x[0][0][1] = 2
x[0][1][0] = 3
x[0][1][1] = 4
x[1][0][0] = 5
x[1][0][1] = 6
x[2][0][0] = 7
x[2][0][1] = 8
x[2][1][0] = 9
x[2][1][1] = 10
x[2][2][1] = 11
x[2][2][1] = 12
Of course it doesn't stop you from accessing an invalid array element - and doing so will generate a seg fault (since the unused elements in x are probably invalid pointers).
Thanks Floris.
I've decided to just load all values into a single array, like
{1,2,2,3,1,2,4,5,2,7,1,1}
and have a second array which stores the length of each first dimension, like
{2,1,3}
The third dimension always has a length of 2, so I will just multiply the number by 2. I'm going to make a helper class so I can just do something like getX(2,0) which would return 4, and have another function like getLength(0) which would return 2.

calculating w coefficients for iir filter

I am trying to implement an IIR filter I have designed in Matlab into a c++ program to filter out an unwanted signal from a wave file. The fdatool in Matlab generated this C header to use (it is a bandstop filter):
#include "tmwtypes.h"
/*
* Expected path to tmwtypes.h
* C:\Program Files (x86)\MATLAB\R2013a Student\extern\include\tmwtypes.h
*/
const int al = 7;
const real64_T a[7] = {
0.9915141178644, -5.910578456199, 14.71918523779, -19.60023964796,
14.71918523779, -5.910578456199, 0.9915141178644
};
const int bl = 7;
const real64_T b[7] = {
1, -5.944230431733, 14.76096188047, -19.60009655976,
14.67733658492, -5.877069568864, 0.9831002459245
};
After hours of exhausting research, I still can't figure out the proper way to use these values to determine the W values and then how to use those W values to properly calculate my Y outputs. If anyone has any insight into the ordering these values should be used to do all these conversions, it would be a major help.
All the methods I've developed and tried to this point do not generate a valid wave file, the header values all translate correctly, but everything beyond cannot be evaluated by a media player.
Thanks.
IIR filters work this way:
Assuming an array of samples A and and array of ceof named 'c' the result array B will be:
B[i] = (A[i] * c[0]) + (B[i-1] * c[1]) + ... + (B[n] * c[n])
Note that only the newest element is taken from A.
This is easier to do in-place, just update A as you move along.
These filter coefs are very violent, are you sure you got them right?
The first one is also symmetrical which probably indicates it's an FIR filter.
It appears to me that you have a 3 pole IIR filter with the coefficients given for an Nth order implementation (as opposed to a series of 2nd order sections). Since this is a band reject (or band pass) the polynomial order is twice the pole count.
I am not sure what you mean by W values, unless you are trying to evaluate the frequency response of this filter.
To calculate the Y values, as you put it, see this link for code on implementing IIR filters. See the Nth order implementation code in particular.
http://www.iowahills.com/A7ExampleCodePage.html
BTW: I assumed these were Nth order coefficients and simulated them. I got a 10 dB notch at 0.05 Pi. Sound about right?
where
B6 = 0.9915141178644
.
.
.
b0 = 0.9915141178644
a6 = 0.9831002459245
.
.
.
a0 = 1
Also, you may want to post a question like this on:
https://dsp.stackexchange.com/