I am a novice at c++, currently trying to create a c extension to a python library using ctypes. My main issue is that I am generating two unknown lengthed vectors inside the c code and don't know the best way to initialize variables on the python code side to interface with the c code.
My approach is to initialize two double pointer float arrays on python side and point the vector data to them on the c side.
my_python_code.py:
float_ptr_ptr = ctypes.POINTER(ctypes.POINTER(ctypes.c_float))
# initialize the array getter function
dll = ctypes.cdll.LoadLibrary("my_c_lib.so")
dll.get_arrays.argtypes = [float_ptr_ptr, float_ptr_ptr]
dll.get_arrays.restype = ctypes.c_int
# initialize the function that deletes float array data
dll.delete_float_array.argtypes = [float_ptr_ptr]
dll.delete_float_array.restype = None
# initialize double pointer to float arrays
output_arr1_py = ctypes.POINTER(ctypes.c_float)()
output_arr2_py = ctypes.POINTER(ctypes.c_float)()
# call c function and save it to numpy arrays
vec_size = dll.get_arrays(ctypes.byref(output_arr1_py), ctypes.byref(output_arr2_py))
np_arr1 = np.fromiter(output_arr1_py, dtype=np.float, count=vec_size.value)
np_arr2 = np.fromiter(output_arr2_py, dtype=np.float, count=vec_size.value)
# deallocate float arrays in c
dll.delete_float_array(output_arr1_py)
dll.delete_float_array(output_arr2_py)
On c++ side I have a function get_arrays() that takes in two double-pointer to float array. get_arrays calls set_vector_values which populates two float vectors (output_vec_1, output_vec_2) and I want to then copy the data from output_vec_1 to output_arr_1 and output_vec_2 to output_arr_2. I have two implementations:
my_c_lib.cc (implementation 1):
int get_arrays(float **output_arr_1, float **output_arr_2) {
std::vector<float> output_vec_1; # same size as output_vec_2
std::vector<float> output_vec_2;
if (!set_vector_values(&output_vec_1, &output_vec_2)) {return -1; # if set_vector_values fails exit with retcode -1}
*(output_arr_1) = output_vec_1.data();
*(output_arr_2) = output_vec_2.data();
return output_vec_1.size();
}
my_c_lib.cc (implementation 2):
int get_arrays(float **output_arr_1, float **output_arr_2) {
std::vector<float> output_vec_1; # same size as output_vec_2
std::vector<float> output_vec_2;
if (!set_vector_values(&output_vec_1, &output_vec_2)) {return -1; # if set_vector_values fails exit with retcode -1}
float *tmp1 = new float[output_vec_1.size()];
for (int32_t i = 0; i < output_vec_1.size(); ++i) {
tmp1[i] = output_vec_1[i];
}
float *tmp2 = new float[output_vec_2.size()];
for (int32_t i = 0; i < output_vec_2.size(); ++i) {
tmp2[i] = output_vec_2[i];
}
*(output_arr_1) = tmp1;
*(output_arr_2) = tmp2;
return output_vec_1.size();
}
void delete_float_array(float **arr){
delete[] *(arr);
}
From my understanding c++ automatically handles memory de-allocation of vectors if I go out of scope how does this affect pointers to vector.data()? For implementation 1 for example, would output_arr_1 still point to output_vec_1.data() or did that disappear once I move out of scope from get_arrays?
implementation 2 is me trying to deep copy the vector data into a temp float array so that output_arr_1 and _2 does not point to the vectors' data. once I am done with using the array I will then call delete_float_array to free up the array memory to prevent memory leak.
please let me know if there are better ways of doing what I am trying to accomplish.
I think it is a trivial question, but I couldn't find a specific solution to it. I'm trying to append array into a vector, using push_back() function. Here is the code:
int main()
{
std::vector<int*> matchVector;
int msmTemp[3];
msmTemp[0] = 1;
msmTemp[1] = 2;
msmTemp[2] = 3;
matchVector.push_back(msmTemp);
msmTemp[0] = 4;
msmTemp[1] = 7;
msmTemp[2] = 0;
matchVector.push_back(msmTemp);
for(auto i : matchVector)
{
for(int j = 0; j<3; j++)
{
cout<<i[j]<<", ";
}
cout<<"\n";
}
return 0;
}
The output I'm getting is 4,7,0 two times. I don't understand as to why I'm not able to see the previous values, namely 1,2,3? Is it because of the type of vector matchVector defined above? I think it needs to be array only.
A int* is a pointer to an integer.
An int[3] is an array of 3 integers.
An array of 3 integers "decays" at the drop of a hat to a pointer to the first element.
When you do push_back(msmTemp), you push a pointer to the first element of msmTemp into the vector.
Pointers in C++ do not own what they point to. The vector afte the two push_backs contains two pointers, both to the same array msmTemp.
When you later iterate over the vector, you get two pointers in turn. Each points to msmTemp.
You then use [] to index those pointers. When you have a pointer to the first element of an array, you can use [] to access the other elements of the array. [0] is the first element, [1] the second, etc.
So you look at the 3 elements in msmTemp (luckily it has 3) and look at them twice, because you have two pointers into it in the vector.
You can inject elements like this:
std::vector<int> matchVector;
int msmTemp[3];
msmTemp[0]={1};
msmTemp[1]={2};
msmTemp[2]={3};
matchVector.insert( matchVector.end(), std::begin(msmTemp), std::end(msmTemp) );
etc. This ends up with a vector containing 6 elements, not two arrays.
If you want arrays as values you need std::array:
std::vector< std::array<int,3> > matchVector;
std::array<int, 3> msmTemp;
and then your code works as written. std::array is a library type that acts sort of like a raw array, but it doesn't have the decay-to-pointer problems of a raw array.
Forget that int[3] names a type. C arrays don't behave like sensible values. Arrays are named std::array<type, count>.
#include <vector>
#include <array>
int main()
{
std::vector<std::array<int, 3>> matchVector;
std::array<int, 3> msmTemp;
msmTemp[0] = 1;
msmTemp[1] = 2;
msmTemp[2] = 3;
matchVector.push_back(msmTemp);
msmTemp[0] = 4;
msmTemp[1] = 7;
msmTemp[2] = 0;
matchVector.push_back(msmTemp);
for(auto & arr : matchVector)
{
for(auto i : arr)
{
std::cout << i <<", ";
}
std::cout<<"\n";
}
return 0;
}
The other answers already explain how to fix your code. I think it's also good to explain why your code behaves the way it does:
Here you tell your compiler to create an std::vector that holds pointers to int:
std::vector<int*> matchVector;
Here you tell your compiler to allocate some space on the stack that fits 3 ints:
int msmTemp[3];
Here you tell your compiler to write the values 1, 2 and 3 into the memory previously allocated:
msmTemp[0] = 1;
msmTemp[1] = 2;
msmTemp[2] = 3;
Here you tell your compiler to take the address of that allocated space, treat it as a pointer and pass it to push_back. This is called array decaying:
matchVector.push_back(msmTemp);
Your matchVector now contains 1 element, which is a pointer to the address of the memory on your stack that was allocated to hold 3 ints.
Here you tell your compiler to write the values 4, 7 and 0 in the memory previously allocated. Note that this is still the same memory block as before:
msmTemp[0] = 4;
msmTemp[1] = 7;
msmTemp[2] = 0;
Here you tell your compiler to again take the address of the allocated space, treat it as a pointer and pass it to push_back:
matchVector.push_back(msmTemp);
Thus matchVector now contains 2 identical values, each a pointer to the same memory location. Specifically the memory location that you last wrote 4, 7 and 0 into.
I have the following piece of code, which is only half on the entire code:
// Declare map elements using an enumeration
enum entity_labels {
EMPTY = 0,
WALL
};
typedef entity_labels ENTITY;
// Define an array of ASCII codes to use for visualising the map
const int TOKEN[2] = {
32, // EMPTY
178 // WALL
};
// create type aliases for console and map array buffers
using GUI_BUFFER = CHAR_INFO[MAP_HEIGHT][MAP_WIDTH];
using MAP_BUFFER = ENTITY[MAP_HEIGHT][MAP_WIDTH];
//Declare application subroutines
void InitConsole(unsigned int, unsigned int);
void ClearConsole(HANDLE hStdOut);
WORD GetKey();
void DrawMap(MAP_BUFFER & rMap);
/**************************************************************************
* Initialise the standard output console
*/
HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
if (hStdOut != INVALID_HANDLE_VALUE)
{
ClearConsole(hStdOut);
// Set window title
SetConsoleTitle(TEXT("Tile Map Demo"));
// Set window size
SMALL_RECT srWindowRect;
srWindowRect.Left = 0;
srWindowRect.Top = 0;
srWindowRect.Bottom = srWindowRect.Top + MAP_HEIGHT;
srWindowRect.Right = srWindowRect.Left + MAP_WIDTH;
SetConsoleWindowInfo(hStdOut, true, &srWindowRect);
// Set screen buffer size
COORD cWindowSize = { MAP_WIDTH, MAP_HEIGHT };
SetConsoleScreenBufferSize(hStdOut, cWindowSize);
}
/*************************************************************************/
/*************************************************************************
* Initialise the tile map with appropriate ENTITY values
*/
MAP_BUFFER tileMap;
for (unsigned int row = 0; row < MAP_HEIGHT; row++)
{
for (unsigned int col = 0; col < MAP_WIDTH; col++)
{
tileMap [row][col] = WALL;
}
}
Essentially the entire code is used to create a tile map and output it to screen but I'm attempting to make tileMap a dynamic array in runtime.
I have tried creating one down where the tileMap is being created.
I've tried creating one just after "entity_lables" are given the typedef "ENTITY".
I've tried creating one after the "MAP_BUFFER" and "GUI_BUFFER" become aliases.
But still I'm at a loss, I have no idea on how to successfully implement a dynamic array to tileMap, and I certainly don't know the best spot to put it.
Any help would be greatly appreciated.
The syntax you are using for defining your array is for a constant sized C array. In general you should shy away from C arrays unless the size of the data is determined at compile time(and never needs to change) and the array never leaves the scope(because a C array does not retain information on its own size.)
In place of constant or dynamically sized C arrays I would suggest to use the Vector container. The Vector is a dynamically sized container that fills up from the back, the last element you have added to
std::vector<std::vector<ENTITY>>
To add the vector container to your project add the line
#include <vector>
To fill the container your loop could look like:
MAP_BUFFER tileMap;
for (unsigned int row = 0; row < MAP_HEIGHT; row++)
{
std::vector<ENTITY> column; // A column of the tile map
for (unsigned int col = 0; col < MAP_WIDTH; col++)
{
column.push_back(WALL); // Add one element to the column
}
tileMap.push_back(column); // Add the column to the tile map
}
or you could initialize the Vector to the size you want at the beginning and use your current loop to assign the tile values:
using TILE_MAP = vector<vector<ENTITY>>;
// MAP_WIDTH x MAP_HEIGHT multidimensional vector
TILE_MAP tileMap(MAP_WIDTH, vector<ENTITY>(MAP_HEIGHT));
for (unsigned int row = 0; row < MAP_HEIGHT; row++)
{
for (unsigned int col = 0; col < MAP_WIDTH; col++)
{
tileMap [row][col] = WALL;
}
}
Calling an element of a vector after it has been filled has the same syntax as an array.
tileMap[2][4]
You can also check the length of the vector:
int rows = tileMap.size();
if( rows > 0 )
int columnsInRow0 = tileMap[0].size()
While you are at it you should look into other containers like Maps and Sets since they make your life easier.
Edit:
Since you want to know how to make a dynamic array not using a vector I will give you an answer: std::vector is the C++ defined dynamically sized array. C arrays will not change size after they are defined, vector will.
However I think you are asking about the ability to define runtime constant sized arrays. So I will explain what they are and why you should not use them.
When you define the C array you are probably getting a warning saying that the expression needs to be constant.
A C array is a pointer to the stack. And the implementation of the compiletime C array is that it needs to be a constant size at compile time.
int compiletimeArray[] = { 1, 2, 3 };
// turns out c arrays are pointers
int* ptr = compiletimeArray;
// prints 2
std::cout << compiletimeArray[1];
// prints 2
std::cout << ptr[1];
// prints 2
std::cout << *(compiletimeArray + 1);
// also prints 2
std::cout << *(ptr + 1); //move pointer 1 element and de-reference
Pointers are like a whiteboard with a telephone number written on it. The same kind of issues occur as with telephone numbers; number on whiteboard has been erased, number on whiteboard has changed, recipient does not exist, recipient changed their number, service provider running out of available numbers to give new users... Keep that in mind.
To get create a runtime constant sized array you need to allocate the array on the heap and assign it to a pointer.
int size = 4;
int* runtimeArray = new int[size]; // this will work
delete[] runtimeArray; // de-allocate
size = 8; // change size
runtimeArray = new int[size]; // allocate a new array
The main difference between the stack and heap is that the stack will de-allocate the memory used by a variable when the program exits the scope the variable was declared in, on the other hand anything declared on the heap will still remain in memory and has to be explicitly de-allocated or you will get a memory leak.
// You must call this when you are never going to use the data at the memory address again
// release the memory from the heap
delete[] runtimeArray; // akin to releasing a phone number to be used by someone else
If you do not release memory from the heap eventually you will run out.
// Try running this
void crashingFunction() {
while(true)
{
// every time new[] is called ptr is assigned a new address, the memory at the old address is not freed
// 90001 ints worth of space(generally 32 or 64 bytes each int) is reserved on the heap
int* ptr = new int[90001]; // new[] eventually crashes because your system runs out of memory space to give
}
}
void okFunction() {
// Try running this
while(true)
{
// every time new[] is called ptr is assigned a new address, the old is not freed
// 90001 ints worth of space is reserved on the heap
int* ptr = new int[90001]; // never crashes
delete[] ptr; // reserved space above is de-allocated
}
}
Why use std::vector? Because std::vector internally manages the runtime array.
// allocates for you
vector(int size) {
// ...
runtimeArray = new runtimeArray[size];
}
// When the vector exits scope the deconstructor is called and it deletes allocated memory
// So you do not have to remember to do it yourself
~vector() {
// ...
delete[] runtimeArray;
}
So if you had the same scenario as last time
void vectorTestFunction() {
// Try running this
while(true)
{
std::vector<int> vec(9001); // internally allocates memory
} // <-- deallocates memory here because ~vector is called
}
If you want to use a runtime constant array I suggest the std:array container. It is like vector in that it manages its internal memory but is optimized for if you never need to add new elements. It is declared just like vector but does not contain resizing functions after its constructor.
I have a short clip of C++ code that should theoretically work to create and return a torch.IntTensor object, but when I call it from Torch I get garbage data.
Here is my code (note this snippet leaves out the function registering, but suffice it to say that it registers fine--I can provide it if necessary):
static int ltest(lua_State* L)
{
std::vector<int> matches;
for (int i = 0; i < 10; i++)
{
matches.push_back(i);
}
performMatching(dist, matches, ratio_threshold);
THIntStorage* storage = THIntStorage_newWithData(&matches[0], matches.size());
THIntTensor* tensorMatches = THIntTensor_newWithStorage1d(storage, 0, matches.size(), 1);
// Push result to Lua stack
luaT_pushudata(L, (void*)tensorMatches, "torch.IntTensor");
return 1;
}
When I call this from Lua, I should get a [torch.IntTensor of size 10] and I do. However, the data appears to be either memory addresses or junk:
29677072
0
16712197
3
0
0
29677328
0
4387616
0
[torch.IntTensor of size 10]
It should have been the numbers [0,9].
Where am I going wrong?
For the record, when I test it in C++
for (int i = 0; i < storage->size; i++)
std::cout << *(storage->data+i) << std::endl;
prints the proper values.
As does
for (int i = 0; i < tensorMatches->storage->size; i++)
std::cout << *(tensorMatches->storage->data+i) << std::endl;
so it seems clear to me that the problem lies in the exchange between C++ and Lua.
So I got an answer elsewhere--the Google group for Torch7--but I'll copy and paste it here for anyone who may need it.
From user #alban desmaison:
Your problem is actually memory management.
When your C++ function return, you vector<int> is free, and so is its content.
From that point onward, the tensor is pointing to free memory and when you access it, you access freed memory.
You will have to either:
Allocate memory on the heap with malloc (as an array of ints) and use THIntStorage_newWithData as you currently do (the pointer that you give to newWithData will be freeed when it is not used anymore by Torch).
Use a vector<int> the way you currently do but create a new Tensor with a given size with THIntTensor_newWithSize1d(matches.size()) and then copy the content of the vector into the tensor.
For the record, I couldn't get it to work with malloc but the copying memory approach worked just fine.
i've a problem with an array (called "Inputs" of type "GeneralInput") on Arduino,basically,no matter which element i try to have access to,the code always returns me the last element of that array.
Here's part of the code:
//...include statements
//other initializations
GeneralInput *Inputs[19];
void setup()
{
//...
//...
InitializeInputs();
}
void InitializeInputs()
{
//type 0 = pedal switch; 1 = volume pedal
//type 2 = potentiometer; 3= switch;
//pedal switches
Inputs[0] = &GeneralInput(0,0,true,false,NULL,10);
Inputs[1] = &GeneralInput(1,0,true,false,NULL,9);
Inputs[2] = &GeneralInput(2,0,true,false,NULL,6);
Inputs[3] = &GeneralInput(3,0,true,false,NULL,5);
//volume pedal
Inputs[4] = &GeneralInput(4,1,false,false,NULL,A2);
//potentiometer
Inputs[5] = &GeneralInput(5,2,false,true,mux2,5);
Inputs[6] = &GeneralInput(6,2,false,true,mux2,6);
Inputs[7] = &GeneralInput(7,2,false,true,mux2,7);
Inputs[8] = &GeneralInput(8,2,false,true,mux2,8);
Inputs[9] = &GeneralInput(9,2,false,true,mux2,9);
Inputs[10] = &GeneralInput(10,2,false,true,mux2,10);
Inputs[11] = &GeneralInput(11,2,false,true,mux2,11);
//switch
Inputs[12] = &GeneralInput(12,3,true,true,mux2,15);
Inputs[13] = &GeneralInput(13,3,true,true,mux2,14);
Inputs[14] = &GeneralInput(14,3,true,true,mux2,13);
Inputs[15] = &GeneralInput(15,3,true,true,mux2,12);
//joystick
Inputs[16] = &GeneralInput(16,3,true,true,mux1,2); //switch
Inputs[17] = &GeneralInput(17,2,false,true,mux1,1); //x axis
Inputs[18] = &GeneralInput(18,2,false,true,mux1,3); //y axis
}
void loop()
{
int length=0;
//cycle through different inputs
int startIndex=0,endIndex=0;
//temp arrays
byte toSendTmp[30];
for(int i=0;i<30;i++)
toSendTmp[i]=0;
//...
//..
int packetIndex=0;
for(int i=startIndex;i<endIndex;i++)
{
//if the input is updated,fill the array with the new data
/*
* When i try to have access to the i-element i always get
* the last one instead.
*/
if(Inputs[i]->Update())
{
toSendTmp[(packetIndex*3)] = Inputs[i]->GetID();
toSendTmp[(packetIndex*3)+1] = Inputs[i]->GetType();
toSendTmp[(packetIndex*3)+2] = Inputs[i]->GetValue();
packetIndex++;
}
}
//....
//...
}
And if needed here's the GeneralInput.h and GeneralInput.cpp code.
NOTE: I can't tell if the array is always returning the last item or if every slot of the array is filled with a pointer to the same object (the last created).
Any idea on what i'm doing wrong?
Thanks in advance.
Your &GeneralInput are incorrect, in fact you create temporary objects and store their adresses in an array, but as soon as your GeneralInput object get destroy (same line as creation), a new object takes place at the same address:
// Create GeneralInput at address #
Inputs[0] = &GeneralInput(0,0,true,false,NULL,10);
// End of your temporary object, the `GeneralInput` object is destroyed but you still
// points to its address...
/* etc. */
You're getting the last value because the compiler always create the GeneralInput at the same address, so all Inputs[] point to the same address.
You need to dynamically create your GeneralInput:
Inputs[0] = new GeneralInput(0,0,true,false,NULL,10);
Every slot in the array has a pointer to the same memory location which is the occupied by the last element you create. By doing &GeneralInput(...) you are creating a GeneralInput object on the stack and retrieving its stack address. But since the GeneralInput object itself is never assigned to a variable, the memory it occupies is immediately available for reuse. This means that every GeneralInput object is created at the same address on the stack. The solution, however, isn't to change your code to something like
GeneralInput genInput = GeneralInput(...);
Inputs[...] = &genInput;
Code like that will still be filling your array with pointers to stack addresses. Those pointers will immediately become invalid when the function returns. You should be filling your array with something like
Inputs[...] = (GeneralInput*)malloc(sizeof(GeneralInput));
*Inputs[...] = GeneralInput(...);
Using this method make sure that if your Inputs array ever reaches a point where you don't use it anymore loop over it freeing every element.
Edit: Arduino using C, so doesn't have new. Use malloc and free instead.
As others have said, the problem is with the address of the temporary variables. You can get around the "new" problem by having default parameters.
class GeneralInput
{
public:
GeneralInput(int a = 0, int b = 0, bool c = true, bool d = true, int* e = NULL, int f = 0);
...
};
Then declare your array - this takes GeneralInput with the default parameters
GeneralInput inputs[20];
Then in Initialize - then you won't have the new problem or the problem of temporaries disappearing at the end of the routine.
void InitializeInputs()
{
inputs[0] = GeneralInput(0,0,true,false,NULL,10);
...
}
I don't know what the NULL points to but you might want to put in a copy operator for this if it is anything else other than copying the value. Not very efficient because it calls the constructor twice but that only happens at initialization.