memory management problem in mexFunction - c++

I'm reading from a binary data file which is written by invoking the following lines in Matlab m-file:
disp(sprintf('template = %d', fwrite(fid, template_1d, 'uint8')));
AFAIK, uint8 is the same size as the types BYTE, unsigned char, and unsigned short. Hence I have written the following code in a file-reading method in a C++ class instantiated in the mexfunction called by Matlab:
template1D = (unsigned short*) malloc(Nimgs*sizeof(unsigned short));
printf("template1D = %d\n", fread(template1D, sizeof(unsigned short), Nimgs, dfile));
and the following is how I deallocated this member variable in the class destructor's helper function:
free((void*) template1D);
In the main mexfunction, when I did not instantiate the class object to persist in memory after mex-function completes by calling mexMakeMemoryPersistent() function, template1D gets cleared properly without segmentation error messages from Matlab. However, if I did instantiate the class to persist in memory as follows:
if (!dasani)
{
dasani = new NeedleUSsim;
mexMakeMemoryPersistent((void*) dasani);
mexAtExit(ExitFcn);
}
with ExitFcn being:
void ExitFcn()
{
delete dasani;
}
then when I'm at the line of free((void*) template1D);, Matlab gives me an error message about the segmentation fault. I have checked the memory sizes and they seem to be consistent. For the malloc/calloc/free functions, I'm using Matlab's mxMalloc/mxCalloc/mxFree functions when I'm executing the C++ project as a Matlab mex function.
Based on this description, what further suggestions would you have for me to solve this problem and ensure this doesn't happen in the future (or at least know how to deal with similar problems like this in the future)?
Thanks in advance.
----------------------------Additions------------------------------------------------------
The following block of code basically shows the jists of my mex file. A mex file is basically an executable that is run in Matlab and compiled from C/C++ code with some Matlab headers.
void ExitFcn()
{
delete dasani;
}
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
needle_info pin;
// check number of i/o if they are correct
if (nrhs != NUMIN)
{
mexErrMsgTxt("Invalid number of input arguments");
}
else if (nlhs != NUMOUT)
{
mexErrMsgTxt("Invalid number of output arguments");
}
// check if the input is noncomplex
if (mxIsComplex(NEEDLE))
{
mexErrMsgTxt("Input must be a noncomplex scalar integer.");
}
// check if the dimensions of the needle information is valid
int needlerows, needlecols;
needlerows = mxGetM(NEEDLE);
needlecols = mxGetN(NEEDLE);
if (needlerows < 1 || needlecols < 6)
{
mexErrMsgTxt("Needle information's dimensions are invalid");
}
float *needlePoint, *yPoint ;
// retrieving current needle information
// order of the variables are always as follows:
// r, theta, l, rho, alpha, beta
needlePoint = (float*) mxGetData(NEEDLE) ;
pin.r = needlePoint[0];
pin.theta = needlePoint[1];
pin.l = needlePoint[2];
pin.rho = needlePoint[3];
pin.alpha = needlePoint[4];
pin.beta = needlePoint[5];
//// read the file inputs
**//if (!dasani)
//{
// dasani = new NeedleUSsim;
// mexMakeMemoryPersistent((void*) dasani);
// mexAtExit(ExitFcn);
//}
dasani = new NeedleUSsim;
delete dasani;**
// sending an useless output for now (get rid of this if not conceptually needed
plhs[0] = mxCreateNumericMatrix(1,1,mxSINGLE_CLASS,mxREAL) ;
yPoint = (float*) mxGetData(plhs[0]) ;
*yPoint = 1;
}
This code would run after build/compilation if the user invokes "mexfunction" anywhere from the command line or m-file script. The snippet enclosed by "**" (when I was trying to bold the snippet) is the problem that I'm looking at. From a second look at the snippet, I may be allocating the memory for dasani pointer in a different memory from the Matlab memory (as there is the memory with scope limited to the C++ mex function only, and another memory space with scope visible to the Matlab program). Otherwise, I'm not sure why Matlab is complaining about this problem.

The MEX API supports C as well as C++. Because C has no try/catch or destructors, there is no way for a C MEX-function to clean-up memory directly in the event of an error. Therefore, MATLAB tracks the results of the memory allocation routines (mxMalloc, mxCalloc, mxRealloc, mxFree, and all the mxCreate* functions that return mxArrays) on an internal list. If an error occurs during the MEX-function execution (either by calling mexErrMsgIdAndTxt directly, or using something like mexEvalString to call MATLAB code that errors), then MATLAB will automatically free any mx-based allocated memory. But, also, when a MEX-function terminates normally, MATLAB will
also free any mx-based memory allocated by the MEX-function. Before the days of destructors, this was a convenience for MEX authors, though in the modern C++ world it can get really annoying.
Sometimes, as in the case of this question, you don't want MATLAB to auto-free memory. In that case, you have to use mexMakeMemoryPersistent, or mexMakeArrayPersistent for mxArrays.
You should only ever pass a pointer to mexMakeMemoryPersistent if it was originally allocated with mxMalloc, mxCalloc, or mxRealloc. So this code
dasani = new NeedleUSsim;
mexMakeMemoryPersistent((void*) dasani);
is bad with a capital 'B', unless you have overloaded NeedleUSsim::operator new() to use mxMalloc, which I wouldn't recommend. But if the fields of dasani are allocated with mxMalloc et al., then you would want to pass those to mexMakeMemoryPersistent. I'd recommend doing something like that in the NeedleUSsim constructor if at all possible, to keep it near the mxMalloc call.

It feels like it's mexMakeMemoryPersistent() that is causing all this trouble. I guess you have to use it to instruct matlab do not delete the memory once it's done. But why should matlab delete the dasani pointer? How is that pointer provided to matlab and what does matlab need it for?

On top of making dasani to be a persistent pointer, I also need to make its member variables with memory allocated by mxMalloc/mxCalloc to be persistent too, for example:
if (!dasani)
{
dasani = new NeedleUSsim;
mexMakeMemoryPersistent((void*) dasani->tplL);
mexMakeMemoryPersistent((void*) dasani->tplR);
mexMakeMemoryPersistent((void*) dasani->tplRho_deg);
mexMakeMemoryPersistent((void*) dasani->tplAlpha_deg);
mexMakeMemoryPersistent((void*) dasani->tplBeta_deg);
mexMakeMemoryPersistent((void*) dasani->hashTb);
mexMakeMemoryPersistent((void*) dasani->template1D);
mexAtExit(ExitFcn);
}
With the destructor as shown:
void NeedleUSsim::Deallocate()
{
free((void*) tplR); free((void*) tplL);
free((void*) tplRho_deg); free((void*) tplAlpha_deg);
free((void*) tplBeta_deg);
free((void*) hashTb);
free((void*) template1D);
}

Related

Calling function within C++ classs not working

I have been working on this simply hobbyist OS, and I have decided to add some C++ support. Here is the simple script I wrote. When I compile it, I get this message:
cp.o: In function `caller':
test.cpp:(.text+0x3a): undefined reference to `__stack_chk_fail'
Here is the script:
class CPP {
public:
int a;
void test(void);
};
void CPP::test(void) {
// Code here
}
int caller() {
CPP caller;
caller.test();
return CPP.a;
}
Try it like this.
class CPP {
public:
int a;
void test(void);
};
void CPP::test(void) {
CPP::a = 4;
}
int caller() {
CPP caller;
caller.test();
return caller.a;
}
int main(){
int called = caller();
std::cout << called << std::endl;
return 0;
}
It seems to me that the linker you are using can't find the library containing a security function crashing the program upon detecting stack smashing. (It may be that the compiler doesn't include the function declaration for some reason? I am not familiar who actually defies this specific function.) Try compiling with -fno-stack-protector or equivalent.
What is the compiler used? A workaround might be defining the function as something like exit(1); or similar. That would produce the intended effect yet fix the problem for now.
I created a test program to show how this actually plays out. Test program:
int main(){
int a[50]; // To have the compiler manage the stack
return 0;
}
With only -O0 as the flag ghidra decompiles this to:
undefined8 main(void){
long in_FS_OFFSET;
if (*(long *)(in_FS_OFFSET + 0x28) != *(long *)(in_FS_OFFSET + 0x28)) {
/* WARNING: Subroutine does not return */
__stack_chk_fail();
}
return 0;
}
With -fno-stack-protector:
undefined8 main(void){
return 0;
}
The array was thrown out by ghidra in decompilation, but we see that the stack protection is missing if you use the flag. There are also some messed up parts of this in ghidra (e.g. int->undefined8), but this is standard in decompilation.
Consequences of using the flag
Compiling without stack protection is not good per se, but it shouldn't affect you in much. If you write some code (that the compiler shouts you about) you can create a buffer overflowable program, which should not be that big of an issue in my optinion.
Alternative
Alternatively have a look at this. They are talking about embedded systems, but the topic seems appropriate.
Why is the code there
Look up stack smashing, but to my knowledge I will try to explain. When the program enters a function (main in this case) it stores the location of the next instruction in the stack.
If you write an OS you probably know what the stack is, but for completeness: The stack is just some memory onto which you can push and off which you can pop data. You always pop the last pushed thing off the stack. C++ and other languages also use the stack as a way to store local variables. The stack is at the end of memory and when you push something, the new thing will be further forward rather than back, it fills up 'backwards'.
You can initialise buffers as a local variable e.g. char[20]. If you filled the buffer without checking the length you might overfill this, and overwrite things in the stack other than the buffer. The return address of the next instruction is in the stack as well. So if we have a program like this:
int test(){
int a;
char buffer[20];
int c;
// someCode;
}
Then the stack will look something like this at someCode:
[ Unused space, c, buffer[0], buffer[1] ..., buffer[19], a, Return Address, variables of calling function ]
Now if I filled the buffer without checking the length I can overwrite a (which is a problem as I can modify how the program runs) or even the return address (which is a major flaw as I might be able to execute malicious shellcode, by injecting it into the buffer). To avoid this compilers insert a 'stack cookie' between a and the return address. If that variable is changed then the program should terminate before calling return, and that is what __stack_chk_fail() is for. It seems that it is defined in some library as well so you might not be able use this, despite technically the compiler being the one that uses this.

save pointer object to system and load it in c++(not c)

I am trying to create a python object in c++ and put it in a file to be used by another program. the problem I am facing that PyObject is a pointer and I don't know how to save a pointer to a file and reload it later. I have tried multiple solutions but nothing works for me, boost for example.
func = PyDict_GetItemString(dict, "create_and_train");
pValue = PyObject_CallFunctionObjArgs(func, PyUnicode_FromString(file_path.c_str()), PyLong_FromLong(size), NULL);
//return feature_values, training_set
if (pValue != NULL)
{
PyObject *object = PyTuple_GetItem(pValue, 0);
}
I want to save object pointer to a file and load it later.
Could any one help me? thanks in advance.
Pointers are basically addresses of memory. Since they are dynamically allocated (meaning where ever there is space in the memory when you plan on executing it). This means that unless you are guaranteed that the location will not change whatsoever, you cant use a saved pointer.
One option is to copy the object's memory footprint to the file in binary. This means that you can load the whole object from the file at once at runtime. But be careful because if the object has pointers, it might not work properly because of the reason I stated before.
Saving the object's memory footprint is pretty simple,
std::ofstream file("sim.bin", std::ios::binary);
if (!file.is_open()) return; // Bail if failed.
// Dummy object as an example.
struct DummyObject { int x = 100; };
DummyObject obj;
file.write(reinterpret_cast<char*>(&obj), sizeof(DummyObject));
file.close();
And when loading the content,
std::ifstream iFile("sim.bin", std::ios::binary);
if (!iFile.is_open()) return; // Bail if failed.
DummyObject iobj;
iFile.read(reinterpret_cast<char*>(&iobj), sizeof(DummyObject));
iFile.close();
Now if that DummyObject had a pointer to some other thing, then we need to copy the content of that data to the file too. It'll slightly complicate things.

How to return base64 decoded image bytes through a methods formal parameter of type char**

Being new to C++, I am still struggling with pointers-to-pointers and I am not sure if my method below is returning decoded image bytes properly.
This method gets a base64 encoded image string from API. The method has to follow this signature as it is part of legacy code that is not allowed to abbreviate from the way it was written originally. So the signature has to stay the same. Also, I have omitted here async calls and continuations, exceptions etc for code simplicity.
int __declspec(dllexport) GetInfoAndPicture(CString uid, char **image, long *imageSize)
{
CString request = "";
request.Format(url);
http_client httpClient(url);
http_request msg(methods::POST);
...
http_response httpResponse;
httpResponse = httpClient.request(msg).get(); //blocking
web::json::value jsonValue = httpResponse.extract_json().get();
if (jsonValue.has_string_field(L"img"))
{
web::json::value base64EncodedImageValue = jsonValue.at(L"img");
utility::string_t imageString = base64EncodedImageValue.as_string();
std::vector<unsigned char> imageBytes = utility::conversions::from_base64(imageString);
image = (char**)&imageBytes; //Is this the way to pass image bytes back?
*imageSize = imageBytes.size();
}
...
}
The caller calls this method like so:
char mUid[64];
char* mImage;
long mImageSize;
...
resultCode = GetInfoAndPicture(mUid, &mImage, &mImageSize);
//process image given its data and its size
I know what pointer to pointer is, my question is specific to this line
image = (char**)&imageBytes;
Is this the correct way to return the image decoded from base64 into the calling code via the char** image formal parameter given the above method signature and method call?
I do get error "Program .... File: minkernel\crts\ucrt\src\appcrt\convert\isctype.cpp ... "Expression c >= -1 && c <= 255"" which I believe is related to the fact that this line is not correctly passing data back.
Give the requirements there isn't any way to avoid allocating more memory and copying the bytes. You cannot use the vector directly because that is local to the GetInfoAndPicture function and will be destroyed when that function exits.
If I understand the API correctly then this is what you need to do
//*image = new char[imageBytes.size()]; //use this if caller calls delete[] to deallocate memory
*image = (char*)malloc(imageBytes.size()); //use this if caller calls free(image) to deallocate memory
std::copy(imageBytes.begin(), imageBytes.end(), *image);
*imageSize = imageBytes.size();
Maybe there is some way in your utility::conversions functions of decoding directly to a character array instead of to a vector, but only you would know about that.
The problem is with allocating (and freeing) memory for that image; who is responsible for that?
You can't (shouldn't) allocate memory in one module and free it in another.
Your two options are:
Allocate large enough buffer on the caller side, and have DLL use it utility::conversions::from_base64(). The issue here is: what is large enough? Some Win APIs provide an additional method to query the required size. Doesn't fit this scenario as the DLL would either have to get that image for the second time, or hold it (indefinitely) until you ask for it.
Allocate required buffer in the DLL and return a pointer to it. You need to ensure that it won't be freed until the caller request to free it (in a separate API).

FlasCC: interop between Vector.<Number> and double[]

Assume that I am new to AS3 and FlasCC and try to port some legacy C++ code to Flash. What I have is some number-crunching routine that takes an array of type double (the array is of fixed size). The C++ prototype is, say:
double doNumberCrunching(double input[512]);
Now I would like to build an SWC and later call doNumberCrunching() from AS3. SWIG generates the AS3 interface as:
doNumberCrunching(var _input:*):Number;
However, I have no idea how to feed it with Vector.<Number>(512) without working on raw bytes.
My question is: how to solve it neatly? Do you know a cheap way to access double* underlying the Vector.<Number>? Or something even better?
If only your number crunching function was operating on ints! Then you would be able to use CModule.writeIntVector to write vector contents to DomainMemory and just pass over a pointer to the C++ code.
But since it's doubles, you will have to iterate through Vector and convert each element from AS3 Number to C++ double using CModule.writeDouble. Then you can manually expose your function to AS3 with an interface that accepts a DomainMemory pointer:
void doNumberCrunchingAS() __attribute__((used,
annotate("as3sig:public function doNumberCrunching(inputPtr:int):Number"),
annotate("as3package:mypackage"))
));
double doNumberCrunching( double* input )
{
// Actual implementation here
return 0;
}
void doNumberCrunchingAS()
{
// Read the AS3 argument and convert it to a convenient C++ form.
int memoryOffset = 0;
AS3_GetScalarFromVar(memoryOffset, inputPtr);
double* inputPtr = reinterpret_cast<double*>( memoryOffset );
// Call an implementation routine.
double result = doNumberCrunching( inputPtr );
// Return result to AS3 code.
AS3_Return( result );
}
AS3 code:
package
{
using mypackage.doNumberCrunching;
function testNumberCrunching():void
{
var ptr:int = CModule.malloc(blobSize);
// a loop of writeDouble should be here
doNumberCrunching(ptr);
CModule.free(ptr);
}
}

Pointer object in C++

I have a very simple class that looks as follows:
class CHeader
{
public:
CHeader();
~CHeader();
void SetCommand( const unsigned char cmd );
void SetFlag( const unsigned char flag );
public:
unsigned char iHeader[32];
};
void CHeader::SetCommand( const unsigned char cmd )
{
iHeader[0] = cmd;
}
void CHeader::SetFlag( const unsigned char flag )
{
iHeader[1] = flag;
}
Then, I have a method which takes a pointer to CHeader as input and looks
as follows:
void updateHeader(CHeader *Hdr)
{
unsigned char cmd = 'A';
unsigned char flag = 'B';
Hdr->SetCommand(cmd);
Hdr->SetFlag(flag);
...
}
Basically, this method simply sets some array values to a certain value.
Afterwards, I create then a pointer to an object of class CHeader and pass it to
the updateHeader function:
CHeader* hdr = new CHeader();
updateHeader(hdr);
In doing this, the program crashes as soon as it executes the Hdr->SetCommand(cmd)
line. Anyone sees the problem, any input would be really appreciated
When you run into a crash, act like a crime investigator: investigate the crime scene.
what is the information you get from your environment (access violation? any debug messages? what does the memory at *Hdr look like? ...)
Is the passed-in Hdr pointer valid?
Then use logical deduction, e.g.:
the dereferencing of Hdr causes an access violation
=> passed in Hdr points to invalid memory
=> either memory wasn't valid to start with (wrong pointer passed in), or memory was invalidated (object was deleted before passing in the pointer, or someone painted over the memory)
...
It's probably SEGFAULTing. Check the pointers.
After
your adding some source code
your comment that the thing runs on another machine
the fact that you use the term 'flag' and 'cmd' and some very small datatypes
making me assume the target machine is quite limited in capacity, I suggest testing the result of the new CHeader for validity: if the system runs out of resources, the resulting pointer will not refer to valid memory.
There is nothing wrong with the code you've provided.
Are you sure the pointer you've created is the same same address once you enter the 'updateHeader' function? Just to be sure, after new() note the address, fill the memory, sizeof(CHeader), with something you know is unique like 0XDEAD, then trace into the updateHeader function, making sure everything is equal.
Other than that, I wonder if it is an alignment issues. I know you're using 8 bit values, but try changing your array to unsigned ints or longs and see if you get the same issue. What architecture are you running this on?
Your code looks fine. The only potential issue I can see is that you have declared a CHeader constructor and destructor in your class, but do not show the implementation of either. I guess you have just omitted to show these, else the linker should have complained (if I duplicate this project in VC++6 it comes up with an 'unresolved external' error for the constructor. It should also have shown the same error for the destructor if you had a... delete hdr; ...statement in your code).
But it is actually not necessary to have an implementation for every method declared in a class unless the methods are actually going to get called (any unimplemented methods are simply ignored by the compiler/linker if never called). Of course, in the case of an object one of the constructor(s) has to be called when the object is instantiated - which is the reason the compiler will create a default constructor for you if you omit to add any constructors to your class. But it will be a serious error for your compiler to compile/link the above code without the implementation of your declared constructor, so I will really be surprised if this is the reason for your problem.
But the symptoms you describe definitely sounds like the 'hdr' pointer you are passing to the updateHeader function is invalid. The reason being that the 1st time you are dereferencing this pointer after the updateHeader function call is in the... Hdr->SetCommand(cmd); ...call (which you say crashes).
I can only think of 2 possible scenarios for this invalid pointer:
a.) You have some problem with your heap and the allocation of memory with the 'new' operator failed on creation of the 'hdr' object. Maybe you have insufficient heap space. On some embedded environments you may also need to provide 'custom' versions of the 'new' and 'delete' operator. The easiest way to check this (and you should always do) is to check the validity of the pointer after the allocation:
CHeader* hdr = new CHeader();
if(hdr) {
updateHeader(hdr);
}
else
//handle or throw exception...
The normal behaviour when 'new' fails should actually be to throw an exception - so the following code will cater for that as well:
try{
CHeader* hdr = new CHeader();
} catch(...) {
//handle or throw specific exception i.e. AfxThrowMemoryException() for MFC
}
if(hdr) {
updateHeader(hdr);
}
else
//handle or throw exception...
}
b.) You are using some older (possibly 16 bit and/or embedded) environment, where you may need to use a FAR pointer (which includes the SEGMENT address) for objects created on the heap.
I suspect that you will need to provide more details of your environment plus compiler to get any useful feedback on this problem.