I have a python object which accesses and downloads some text via HTTP.
I'm running this python object, and processing that text, using a c++ code.
I.e.
/* CPPCode.cxx */
int main(...) {
for(int i = 0; i < numURLs; i++) {
// Python method returns a string
PyObject *pyValue = PyObject_CallMethod(pyObjectInstance, pyFunctionName, par1, par2....);
string valString = PyString_AsString(pHistValue);
// ... process string ...
}
}
/* PyObject.py */
class PyClass:
def PyFunction(...):
try: urlSock = urllib.urlopen(urlName)
except ...
while(...) :
dataStr = urlSock.readline()
# do some basic string processing....
return dataStr
Most URLs work fine---the c++ code gets the proper string, I can process it, all is happy and well. A few particular URLs which look (basically) the same as the others on a browser, lead to a segfault in the PyString_AsString() method:
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x00000000000000b2
0x000000010007716d in PyString_AsString ()
If I print out the string that should be returned by the python method ('dataStr' in the pseudo-code above), it looks fine! I have no idea what could be causing this problem---any tips on how to procede would be appreciated!
Thanks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SOLUTION:
The template code I was using had a call to
Py_DECREF(pyValue)
before I called
PyString_AsString(pyValue)
Why it was being deallocated for certain particular function calls, I have no idea. As 'Gecco' says in the comments below,
'PyString_AsString documentation says: "The pointer refers to the internal buffer of string, not a copy. The data must not be modified in any way, unless the string was just created using PyString_FromStringAndSize(NULL, size). It must not be deallocated." '
PyString_AsString documentation says: "The pointer refers to the internal buffer of string, not a copy. The data must not be modified in any way, unless the string was just created using PyString_FromStringAndSize(NULL, size). It must not be deallocated."
Please ensure you do not deallocate this buffer
If you compile your C code with the -g debug flag (in GCC at least) then you can run your python code using the gnu debugger gdb:
$ gdb /path/to/python/compiled/against
... blah ...
(gdb) run PyObject.py
and you should catch your segfault.
My guess is the Py_DECREF is somehow getting a NULL value.
Related
I have a transformer routine written in C++ that is set to clear all whitespace and map to a value if the input string is either null or empty. The c++ code compiles and has tested properly, but I am having trouble getting the routine to work in Datastage.
As per instructions, I have copied the exact compiler options that I have in my DS Environment as below.
g++ -c -O -fPIC -Wno-deprecated -m64 -mtune=generic -mcmodel=small BlankToValue.cpp
g++ -shared -m64 BlankToValue.so BlankToValue.o
When testing the routine in a job however I get the following error.
Sequential_File_36,0: Internal Error: (shbuf): iomgr/iomgr.C: 2649
Is there a different set of options I should be using for compilation?
For reference, the c++ code.
#include <stdlib.h>
#include <stdio.h>
#include <algorithm>
#include <locale.h>
#include <locale>
char * BlankToValue(char *InStr, char *RepStr)
{
if (InStr[0] == '\0') // Check for null pointer at first character of input string.
{
return RepStr; // Return replacement string if true. This is to prevent unnecessary processing.
} else
{
const char* checkstr = InStr; // Establish copy of inputstring stored in checkstring.
do { // Start outer loop.
while (isspace(*checkstr)) { // Inner loop while current checkstring byte is whitespace.
++checkstr; // Increment to next checkstring byte.
}
} while ((*InStr++ = *checkstr++)); // Set inputstring byte to current checkstring and iterate both. Breaks when either string evaluates to null.
*InStr = '\0'; // Set null terminator for input string at current byte location.
if (InStr[0] == '\0') // Checks first character of cleaned input string for null pointer.
{
return RepStr; // Return replacement string if true.
} else
{
return InStr; // Return new input string if false.
}
}
}
William,
in your DataStage routine definition that points to this custom function, did you select routine type as object (.o file that is compiled into transformer stage at job run time) or a library (a lib.so file that is loaded at job run time but has requirements on library naming convention and that library is located in library path). Your code above suggested you are creating a *.so file but not prefixed with lib. Here is an example:
https://www.ibm.com/support/pages/node/403041
Additionally, if the first error in job log was not a library load error but rather was the internal error (shbuf) error, I found a couple of cases where that has occurred in the past with custom routines:
Custom routine involved null handling, as does yours, and began to fail after upgrading to Information Server 8.5 when null handling rules changed in our product. The changes are explained here:
https://www.ibm.com/support/pages/node/433863
You could test if this is the issue by running the job with new job level environment variable: APT_TRANSFORM_COMPILE_OLD_NULL_HANDLING=1
In another case, the shbuf error in custom routine was the result of transformer stage receiving a large record (larger than could be handled by the datatype defined in the custom routine). Does the job still fail when using only a single sample input record with small values in all string type fields.
Thanks.
Also, I noticed that the error is coming from sequential file stage, rather than the transformer stage that is using the custom routine. Thus may also need to consider what is the output datatype for your custom routine and ensure that it is exiting with valid value that is not too large for the datatype and also not larger that default transport buffer size used between stages (defaults to 128k).
After a day or two of multiple attempts to try different compile and code approaches I found the solution to my problem. The below code was throwing a segmentation fault when passed a null column. Which makes sense in retrospect.
if (InStr[0] == '\0')
It has been corrected to the below and now everything works properly.
if ((InStr == NULL) || (InStr[0] == '\0'))
I have a .dll file that's being injected into a game's process. It tries to read a small string from a specific area of memory, but whenever that string is accessed as an std::string, the game crashes. I've only just begun scratching the surface of how program memory works, so I'm lost.
In this game, there are instances of objects that have specific properties. Those properties always have the same offset. The property I am trying to read is the Name property, which has an offset of 0x28 for every possible instance. This is the function I'm using to get the name from the instance:
std::string* GetName(int Instance) {
return (std::string*)(*(int*)(Instance + 0x28));
}
I've used a debugger/disassembler to verify that the instance is valid and that the resulting pointer has the value that I need. However, as soon as I try to do something with it, the game crashes. For example, this:
std::string objName = *GetName(obj);
Or this:
std::cout << *GetName(obj);
Or even this (when I got desperate enough):
std::string* objName = (std::string*) obj + 0x28;
std::cout << *objName;
I tried attaching my IDE's debugger to the game, and the most useful information I got was "segmentation fault" when the crash happened. Even my much more knowledgeable friends who helped me out with this are stumped as to what could be wrong.
If there isn't enough information here to find out what the problem is, could I at least get a nudge in the right direction? What could possibly be causing this?
As pointed out by MacroVirus in a comment, the solution was to store the value in a char* then convert that to a new std::string instance. It now works flawlessly. Here is the revised function:
char* GetName(int Instance) {
return (char*)(*(int*)(Instance + 0x28));
}
Thank you for the help! This was driving me crazy.
I am trying to make a socket on windows to connect to a server.
I am using the code from msdn's website, the winsock client code.
(Link: msdn.microsoft.com/en-us/library/windows/desktop/ms737591(v=vs.100).aspx )
In any case, when I try debugging said code I get the error message:
Unhandled exception at 0x58a714cf (msvcr100d.dll) in Application.exe: 0xC0000005: Access violation reading location 0x00000032.
It asks me if I want to break or continue, if I continue the same error message simply keeps popping up. If I press break it takes me to line 1643 in the file output.c .
Simply put, I have no idea about what to do to make it work, and I'd appreciate any help given.
EDIT:
A reply to all comments given thus far:
The surrounding relevant code in output.c is:
if (flags & (FL_LONG|FL_WIDECHAR)) {
if (text.wz == NULL) /* NULL passed, use special string */
text.wz = __wnullstring;
bufferiswide = 1;
pwch = text.wz;
while ( i-- && *pwch )
++pwch;
textlen = (int)(pwch - text.wz);
/* textlen now contains length in wide chars */
} else {
if (text.sz == NULL) /* NULL passed, use special string */
text.sz = __nullstring;
p = text.sz;
while (i-- && *p) //THIS IS WHERE IT BREAKS
++p;
textlen = (int)(p - text.sz); /* length of the string */
}
This is not code that I have written but innate code that already exists.
EDIT NR 2:
This is a printscreen displaying my call stack.
I do not have 10 reputation so I cannot show the image, so here is a link to the image:
http://tinypic.com/r/5n6ww9/5
On it you can see my call stack
The file output.c has the code that handles the printf family of functions.
The fact that you have an error here is probably due to a malformed printf function call in your code. Maybe you have specified an invalid print format or have not provided enough arguments.
When your program crashes, click Break and look at the call stack in the debugging windows to see where - in your code - the function is called, and with what arguments.
I suspect you are trying to print a NULL string or something. When you have found the printf call (if that's what it is), edit your question to show that section of source code and/or use the debugger to examine the variables used a arguments to the function and make sure they are all correct.
Without seeing the code that you've written, at the location of the crash, it's not possible to give a more precise answer.
Or you may insert a null string to CString.format like below:
CString str;
str.format("%s"); //A null string
where it should be
str.format("%s",mes);
where mes is char mes[20] for example
or any other variable
So the point is there may be some mistake in CString.format or Printf
Good Luck.
I have valgrind 3.6.0, I've searched everywhere and found nothing.
The problem is that when I'm trying to access a float number while using valgrind, I get a segfault, but when I run the program as is, without valgrind, everythings goes as expected.
This is the piece of code:
class MyClass {
public:
void end() {
float f;
f = 1.23;
std::stringstream ss;
ss << f;
std::cout << ss.str();
}
};
extern "C" void clean_exit_on_sig(int sig) {
//Code logging the error
mc->end();
exit(1);
}
MyClass *mc;
int main(int argc, char *argv[]) {
signal(SIGINT , clean_exit_on_sig);
signal(SIGABRT , clean_exit_on_sig);
signal(SIGILL , clean_exit_on_sig);
signal(SIGFPE , clean_exit_on_sig);
signal(SIGSEGV, clean_exit_on_sig);
signal(SIGTERM , clean_exit_on_sig);
mc = new MyClass();
while(true) {
// Main program loop
}
}
When I press Control+C, the program catches the signal correctly and everything goes fine, but when I run the program using valgrind, when tries to execute this command ss << f; // (Inside MyClass) a segfault is thrown :-/
I've tried this too:
std::string stm = boost::lexical_cast<std::string>(f);
But I keep on receiving a segfault signal when boost acceses the float number too.
This is the backtrace when I get segfault with boost:
./a.out(_Z17clean_exit_on_sigi+0x1c)[0x420e72]
/lib64/libc.so.6(+0x32920)[0x593a920]
/usr/lib64/libstdc++.so.6(+0x7eb29)[0x51e6b29]
/usr/lib64/libstdc++.so.6(_ZNKSt7num_putIcSt19ostreambuf_iteratorIcSt11char_traitsIcEEE15_M_insert_floatIdEES3_S3_RSt8ios_baseccT_+0xd3)[0x51e8f43]
/usr/lib64/libstdc++.so.6(_ZNKSt7num_putIcSt19ostreambuf_iteratorIcSt11char_traitsIcEEE6do_putES3_RSt8ios_basecd+0x19)[0x51e9269]
/usr/lib64/libstdc++.so.6(_ZNSo9_M_insertIdEERSoT_+0x9f)[0x51fc87f]
./a.out(_ZN5boost6detail26lexical_stream_limited_srcIcSt15basic_streambufIcSt11char_traitsIcEES4_E9lcast_putIfEEbRKT_+0x8f)[0x42c251]
./a.out(_ZN5boost6detail26lexical_stream_limited_srcIcSt15basic_streambufIcSt11char_traitsIcEES4_ElsEf+0x24)[0x42a150]
./a.out(_ZN5boost6detail12lexical_castISsfLb0EcEET_NS_11call_traitsIT0_E10param_typeEPT2_m+0x75)[0x428349]
./a.out(_ZN5boost12lexical_castISsfEET_RKT0_+0x3c)[0x426fbb]
./a.out(This line of code corresponds to the line where boost tries to do the conversion)
and this is with the default stringstream conversion:
./a.out(_Z17clean_exit_on_sigi+0x1c)[0x41deaa]
/lib64/libc.so.6(+0x32920)[0x593a920]
/usr/lib64/libstdc++.so.6(+0x7eb29)[0x51e6b29]
/usr/lib64/libstdc++.so.6(_ZNKSt7num_putIcSt19ostreambuf_iteratorIcSt11char_traitsIcEEE15_M_insert_floatIdEES3_S3_RSt8ios_baseccT_+0xd3)[0x51e8f43]
/usr/lib64/libstdc++.so.6(_ZNKSt7num_putIcSt19ostreambuf_iteratorIcSt11char_traitsIcEEE6do_putES3_RSt8ios_basecd+0x19)[0x51e9269]
/usr/lib64/libstdc++.so.6(_ZNSo9_M_insertIdEERSoT_+0x9f)[0x51fc87f]
./a.out(This line of code corresponds to the line where I try to do the conversion)
a.out is my program, and I run valgrind this way: valgrind --tool=memcheck ./a.out
Another weird thing is that when I call mc->end(); while the program runs fine (Any signal received, Object just finished his work), I don't get segfault in any way (as is and with valgrind).
Please, don't tell me 'Don't close your program with Control+C blah blah...' this piece of code is for logging any error the program possibly have without losing data in case of segfault, killing it because of deadlock or something else.
EDIT: Maybe is a valgrind bug (I don't know, searched on google but found nothing, don't kill me), any workaround will be accepted too.
EDIT2: Just realized that boost calls ostream too (Here is clearer than using vim :-/), going to try sprintf float conversion.
EDIT3: Tried this sprintf(fl, "%.1g", f); but still crashes, backtrace:
./a.out(_Z17clean_exit_on_sigi+0x40)[0x41df24]
/lib64/libc.so.6(+0x32920)[0x593a920]
/lib64/libc.so.6(sprintf+0x56)[0x5956be6]
./a.out(Line where sprintf is)
Ok, after some hours of reading and research, I found the problem, I'm going to answer my own question because noone does, only a comment by #Kerrek SB [ https://stackoverflow.com/users/596781/kerrek-sb ] but I cannot accept a comment. (Thank you)
It's as easy as inside a signal handler you only can call a bunch of functions safely: http://pubs.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html
If you call some non-async-safe functions, they can work, but not always.
If you want to call non-async-safe functions inside a signal handler, you can do this:
Create 2 pipes. int pip1[2]; int pip2[2]; pipe(pip1); pipe(pip2);
Create a new thread and make the thread wait to receive some data from the 1rst pipe read(pip1[0], msg, 1);
When signal handler is called, use write async-safe function to write to the 1rst pipe write(pip1[1], "0", 1);
Then make the signal wait for the second pipe with read(pip2[0], msg, 1);
The thread will wake up and do all the job he has to do (saving data to database in this case), after that, make the thread write data to the second pipe write(pip2[1], "0", 1);
Now main thread will wake up and finish with _Exit(1) or something else.
Info:
I'm using 2 pipes because if I write to a pipe and just after that I read it, it's possible that the 2nd thread never wakes up because the main thread have read the data have just written. And I'm using a secondary pipe to block the main thread because I don't want it to exit while the 2nd thread is saving data.
Keep in mind that signal handler maybe has been called while modifying a shared resource, if your 2nd thread acceses that resource is possible that you encounter a second segfault, so be careful when accesing shared resources with your 2nd thread (Global variables or something else).
If you are testing with valgrind and don't want to receive 'false' memory leaks when receiving a signal you can do this before exiting pthread_join(2ndthread, NULL) and exit(1) instead of _Exit(1). These are non-async-safe functions, but at least you can test memory leaks and close you app with a signal without receiving 'false' memory leaks.
Hope this helps someone. Thanks again #Kerrek SB.
Debuggers and stuff sometimes toss signals to the process that you don't normally get. I had to alter a function that used recv to work under gdb for example. Check to see what your signal is and verify that mc is not null before trying to use it. See if that starts getting you closer to an answer.
I am thinking perhaps your use of new (or something else maybe) is possibly causing valgrind to send a signal that is being caught by your handler before mc is initialized.
It's also clear you didn't paste actual code because your use of 'class' without making the end() function public means this should not compile.
I have a program that makes heavy use of QSharedPointer. When I execute my program it runs fine, but when I debug it with GDB it starts throwing errors. "Invalid Address specified to RtlFreeHeap" is thrown in the following code:
QSharedPointer<PersistentList> p =
PersistentList::createEx(wrap("abc")).dynamicCast<PersistentList>();
QSharedPointer<IPersistentCollection> c = p->empty(); // Error thrown after this line
QSharedPointer<IPersistentCollection> ASeq::empty()
{
return QSharedPointer<EmptyList>(new EmptyList());
}
If I disable the p->empty() line the program runs just fine. Any ideas?
The problem is likely elsewhere in your code. Run it under Valgrind and see if you're touching any memory that's not yours, or using uninitialized data, etc.