i'm having a weird situation. i'm trying to implement a 10+ years old pci camera device SDK to my camera management software. Manifacturer no longer in business and i have no chance to get official help. So here I am, looking for some help to my ugly problem.
SDK comes with Visual Studio 6.0 samples. One of the include files has a structure ending with a one byte array like below;
typedef struct AVData {
...
BYTE audioVideoData[1];
}AVDATA, *PAVDATA;
But this single byte allocated byte array receives video frames and weird enough, it works fine with Visual Studio 6.0 version. If I try it with Visual Studio 2005/2008/2010, i start getting Memory Access Violation error messages which actully makes sense since it shouldn't be possible to allocate space to a fixed size array afterwards, no? But same code runs fine with VS 6.0?! It's probably caused by either compiler or c++ runtime differences but i'm not very experienced on this subject so it's hard to tell the certain reason for me.
I tried changing the size to an expected maximum number of bytes like below;
typedef struct AVData {
...
BYTE audioVideoData[20000];
}AVDATA, *PAVDATA;
This helped it get working but from time to time i get memory access violation problems when trying to destroy the decoder object of the library.
There is something definitely wrong with this. I don't have the source codes of the SDK, only the DLL, Lib and Header files. My questions are:
1) Is it really legal to allocate space to a fixed size array in Visual Studio 6.0 version?
2) Is there any possible way (a compiler option etc.) to make the same code work with newer VS versions / C++ runtimes?
3) Since my workaround of editing the header file works up to a point but still having problems, do you know any better way to get around of this problem?
IIRC its an old trick to create a struct that is variable in size.
consider
struct {
int len;
char name[1];
} s;
the 'name' can now be of variable length if the appropriate allocation is done and it will be sequentially laid out in memory :
char* foo = "abc";
int len = strlen(foo);
struct s* p = malloc( sizeof(int) + len + 1 );
p->len = len;
strcpy(p->name, foo );
I think the above should work fine in newer versions of visual studio as well, maybe it is a matter of packing, have you done #pragma pack(1) to get structs on byte boundaries? I know that VS6 had that as default.
A one-element array in a C structure like this often means that the size is unknown until runtime. (For a Windows example, see BITMAPINFO.)
Usually, there will be some other information (possibly in the struct) that tells you how large the buffer needs to be. You would never allocate one of these directly, but instead allocate the right size block of memory, then cast it:
int size = /* calculate frame size somehow */
AVDATA * data = (AVDATA*) malloc(sizeof(AVDATA) + size);
// use data->audioVideoData
The code almost certainly exhibits undefined behaviour in some way, and there is no way to fix this except to fix the interface or source code of the SDK. As it is no longer in business, this is impossible.
Related
I've got two modules. One calls other one and passes as an argument a structure.
struct {
char* szDGRTag;
bool bTagEx;
} ADPTAG;
Master is written in C++ from '98 in Visual C++ 6.0. Slave is written in C++ 11 in Visual Studio 2010 Professional.
Function that is called in Slave:
long lCheckPresenceOfFields (char* szName, ADPTAG* AdpTagList, long lNbVar)
In Master:
long lNbVar = 2;
ADPTAG* AdpTagList = NULL;
AdpTagList = new ADPTAG[lNbVar];
AdpTagList[0].szTag = new char[32];
AdpTagList[0].bTagEx = true;
memset(AdpTagList[0].szTag, 0x0, 32+1);
AdpTagList[0].szTag = NULL;
AdpTagList[1].szTag = new char[32];
AdpTagList[1].bTagEx = true;
memset(AdpTagList[1].szTag, 0x0, 32+1);
AdpTagList[1].szTag = NULL;
int size = sizeof(AdpTagList[0]);
AdpTagList[0].szTag = "DDD";
AdpTagList[1].szTag = "AAA";
long pres = pGetFieldsPresence(szPath, AdpTagList, 2);
I've checked size of AdpTagList[0] in Master and it was 5 bytes, but in Slave it is 8 bytes, so the first object is OK, but any next is bad, because pointer points to wrong memory area.
What can be the issue here? Different compilator? Probably not as structure with only char* and int works fine between these two modules.
Regardlesss of size of 1st object [0], second [1] is null-pointer.
You're running into a packing discrepancy. You can modify the packing with the "/Zp" compiler option.
Since by default Visual Studio:
Packs structures on 8-byte boundaries
It's very likely that your Visual Studio 2010 compiler does not specify packing. Your Visual C++ 6.0 application likely did specify 1-byte packing as a structure containing an int and a char should pack into that.
Using 1-byte packing optimizes a programs system memory footprint, because no memory is lost to padding.
Using 8-byte packing optimizes speed because it increases the chance that structures can be fetched in a single load.
For more on packing you can read here: https://msdn.microsoft.com/en-us/library/71kf49f1.aspx
But I'd go with Microsoft's statement:
You should not use this option unless you have specific alignment requirements
Meaning since you have to change one, I'd recommend changing the packing on the Visual C++ 6.0 application.
Yes, its the compiler difference that's to blame. There is no standardisation at all of C++ at the binary level. In particular this means any compiler is free to choose its own padding rules for structures, which is what is affecting you here. Clearly the more recent compiler has decided to add 3 bytes of padding to make the structure 64-bit aligned, but the older compiler doesn't do this.
In general there is no way to ensure binary compatibility between data produced by two different compilers. You can either switch to using byte streams, or use the same compiler for both.
I've converted some legacy Fortran code to C using the f2c converter (f2c), and I've created a Visual Studio 10 solution on Windows 7 (64-bit). I've also had to link my C++ program (test.cpp, containing my main function) with the f2c library (built on my system using nmake).
The program runs, but once the end of the main function is reached, I receive the following Debug error:
Stack around the variable 'qq' was corrupted
Stack around the variable 'pf' was corrupted
Stack around the variable 'ampls' was corrupted
I am wondering if this might be due to a "correction" made by the f2c converter in the converted C (from Fortran) file:
/* Parameter adjustments */
--x1;
--xabs;
--ximag;
--xreal;
--work4;
--work3;
--work2;
--work1;
--ampls;
--pf;
--qq;
--tri;
This seems a bit odd, since all of these variables are C arrays, and I think that the f2c program is simply doing some pointer arithmetic so that index 0 in the array becomes index 1, in a similar fashion to Fortran.
I don't know if this could also be due to something going wrong with the converted code accessing an element of the array that has not been allocated.
What is the best way to debug this error and fix it?
Potential reasons:
This error is usually related to writing outside the bounds of an array (dynamic or static array). This error can occur by writing\getting a value in a -ve index or index >= size_of_array.
This error also accurs if your pointer is not set to its correct location. (e.g. ptr = 0, ptr = 55, points to deleted (released; or has been free) memory, or any invalid address)
Best way to debug your error in my mind is to debug your prorgam step-by-step and watch those pointer values. There must be some wrong with them.
What you say could be true. I would suggest to create a very small program that uses an array and decrements the pointer exactly as f2c does. Something like
int aa[10];
int *pa = aa;
--pa;
pa[1] = ...
That is, test the suspected code in the small scale. You might isolate the cause to the problem this way. (Finding a workaround is a different story)
Are you compiling with the debug versions of the crt? That might give you some more information.
Also, is it possible that your library is built as C and your application is written as C++?
Those errors you mention are sometimes because of different calling conventions. You do state it's a 64bit application, so it shouldn't be an issue (all 64bit apps use the same calling convention), but it's worth looking into.
Is it possible to add all the fortran converted code to visual studio and not do a seperate make?
I have the following code segment . For a vector of size 176000 the loop takes upto 8 minutes to execute . I am not sure what is taking so much time
XEPComBSTR bstrSetWithIdsAsString; //Wrapper class for BSTR
std::vector<__int64>::const_iterator it;
for(it = vecIds.begin();
it != vecIds.end();
it++)
{
__int64 i64Id = (*it);
__int64 i64OID = XPtFunctions::GetOID(i64Id);
// set ',' between two set members
if (it != vecIds.begin())
bstrSetWithIdsAsString.Append(XEPComBSTR(L","));
wchar_t buf[20];
_i64tow_s(i64OID, buf, 20, 10);
bstrSetWithIdsAsString.Append(buf);
}
__int64 GetOID( const __int64 &i64Id)
{
__int64 numId = i64Id;
numId <<= 16;
numId >>= 16;
return numId;
}
I think your bottleneck is the Append function. You see, the string has some allocated memory inside, and when you try to append something that won't fit, it reallocates more memory, which takes a lot of time. Try allocating necessary memory once in the beginning. HTH
I am not sure what this is doing:
bstrSetWithIdsAsString.Append(buf);
but I guess this is where the slowness is, particularly if it has to work out where the end of the buffer is every time by looking for the first zero byte, and possibly needs to do a lot of reallocating.
Why not use wostringstream?
The only way to find out what takes up all this time is to profile the application. Some editions of Visual Studio come with a fully functional profiler.
Alternatively, simply run the program in the debugger, and break it at random intervals, and note where in the code you are.
But I can see a few potential trouble spots:
you perform a lot of string appends. Do they allocate new memory each time? Does your string type allow you to reserve memory in advance, like std::string can do? In general, is the string class efficient?
you loop over iterators, and given the horrible Hungarian Notation you appear to use, I am assuming you are working on Windows, probably using MSVC. Some versions of MSVC enable a lot of runtime checking of STL iterators even in release builds, unless you explicitly disable it. VS2005 and 2008 specifically are guilty of this. 2010 only enables this checking in debug mode.
and, of course, you are building with optimizations enabled, right?
But I'm just pointing out what seems like it could potentially slow down your code. I have no clue what's actually happening. To be sure, I'd have to profile your code. You can do that. I can't. So do it.
Apparently this function in SDL_Mixer keeps dying, and I'm not sure why. Does anyone have any ideas? According to visual studio, the crash is caused by Windows triggering a breakpoint somewhere in the realloc() line.
The code in question is from the SVN version of SDL_Mixer specifically, if that makes a difference.
static void add_music_decoder(const char *decoder)
{
void *ptr = realloc(music_decoders, num_decoders * sizeof (const char **));
if (ptr == NULL) {
return; /* oh well, go on without it. */
}
music_decoders = (const char **) ptr;
music_decoders[num_decoders++] = decoder;
}
I'm using Visual Studio 2008, and music_decoders and num_decoders are both correct (music_decoders contains one pointer, to the string "WAVE", and music_decoders. ptr is 0x00000000, and the best I can tell, the crash seems to be in the realloc() function. Does anyone have any idea how I could handle this crash problem? I don't mind having to do a bit of refactoring in order to make this work, if it comes down to that.
For one thing, it's not valid to allocate an array of num_decoders pointers, and then write to index num_decoders in that array. Presumably the first time this function was called, it allocated 0 bytes and wrote a pointer to the result. This could have corrupted the memory allocator's structures, resulting in a crash/breakpoint when realloc is called.
Btw, if you report the bug, note that add_chunk_decoder (in mixer.c) is broken in the same way.
I'd replace
void *ptr = realloc(music_decoders, num_decoders * sizeof (const char **));
with
void *ptr = realloc(music_decoders, (num_decoders + 1) * sizeof(*music_decoders));
Make sure that the SDL_Mixer.DLL file and your program build are using the same C Runtime settings. It's possible that the memory is allocated using one CRT, and realloc'ed using another CRT.
In the project settings, look for C/C++ -> Code Generation. The Runtime Library setting there should be the same for both.
music_decoders[num_decoders++] = decoder;
You are one off here. If num_decoders is the size of the array then the last index is num_decoders - 1. Therefore you should replace the line with:
music_decoders[num_decoders-1] = decoder;
And you may want to increment num_decoders at the beginning of the function, not at the end since you want to reallow for the new size, not for the old one.
One additional thing: you want to multiply the size with sizeof (const char *), not with double-star.
Ah, the joys of C programming. A crash in realloc (or malloc or free) can be triggered by writing past the bounds of a memory block -- and this can happen anywhere else in your program. The approach I've used in the past is some flavor of debugging malloc package. Before jumping in with a third party solution, check the docs to see if Visual Studio provides anything along these lines.
Crashes are not generally triggered by breakpoints. Are you crashing, breaking due to a breakpoint or crashing during the handling of the breakpoint?
The debug output window should have some information as to why a CRT breakpoint is being hit. For example, it might notice during the memory operations that guard bytes around the original block have been modified (due to a buffer overrun that occurred before add_music_decoder was even invoked). The CRT will check these guard pages when memory is freed and possibly when realloced too.
I'm pretty inexperienced using C++, but I'm trying to compile version 2.0.2 of the SBML toolbox for matlab on a 64-bit XP platform. The SBML toolbox depends upon Xerces 2.8 and libsbml 2.3.5.
I've been able to build and compile the toolbox on a 32-bit machine, and it works when I test it. However, after rebuilding it on a 64-bit machine (which is a HUGE PITA!), I get a segmentation fault when I try to read long .xml files with it.
I suspect that the issue is caused by pointer addresses issues.
The Stack Trace from the the segmentation fault starts with:
[ 0] 000000003CB3856E libsbml.dll+165230 (StringBuffer_append+000030)
[ 6] 000000003CB1BFAF libsbml.dll+049071 (EventAssignment_createWith+001631)
[ 12] 000000003CB1C1D7 libsbml.dll+049623 (SBML_formulaToString+000039)
[ 18] 000000003CB2C154 libsbml.dll+115028 (
So I'm looking at the StringBuffer_append function in the libsbml code:
LIBSBML_EXTERN
void
StringBuffer_append (StringBuffer_t *sb, const char *s)
{
unsigned long len = strlen(s);
StringBuffer_ensureCapacity(sb, len);
strncpy(sb->buffer + sb->length, s, len + 1);
sb->length += len;
}
ensureCapacity looks like this:
LIBSBML_EXTERN
void
StringBuffer_ensureCapacity (StringBuffer_t *sb, unsigned long n)
{
unsigned long wanted = sb->length + n;
unsigned long c;
if (wanted > sb->capacity)
{
/**
* Double the total new capacity (c) until it is greater-than wanted.
* Grow StringBuffer by this amount minus the current capacity.
*/
for (c = 2 * sb->capacity; c < wanted; c *= 2) ;
StringBuffer_grow(sb, c - sb->capacity);
}
}
and StringBuffer_grow looks like this:
LIBSBML_EXTERN
void
StringBuffer_grow (StringBuffer_t *sb, unsigned long n)
{
sb->capacity += n;
sb->buffer = (char *) safe_realloc(sb->buffer, sb->capacity + 1);
}
Is it likely that the
strncpy(sb->buffer + sb->length, s, len + 1);
in StringBuffer_append is the source of my segfault?
If so, can anyone suggest a fix? I really don't know C++, and am particularly confused by pointers and memory addressing, so am likely to have no idea what you're talking about - I'll need some hand-holding.
Also, I put details of my build process online here, in case anyone else is dealing with trying to compile C++ for 64-bit systems using Microsoft Visual C++ Express Edition.
Thanks in advance!
-Ben
Try printing or using a debugger to see what values your getting for some of your intermediate variables. In StringBuffer_append() O/P len, in StringBuffer_ensureCapacity() observe sb->capacity and c before and in the loop. See if the values make sense.
A segmentation fault may be caused by accessing data beyond the end of the string.
The strange fact that it worked on a 32-bit machine and not a 64-bit O/S is also a clue. Is the physical and pagefile memory size the same for the two machines? Also, in a 64-bit machine the kernel space may be larger than the 32-bit machine, and eating some available memory space that was in the user part of the memory space for 32-bit O/S. For XML the entire document must fit into memory. There are probably some switches to set the size if this is the problem. The difference in machines being the cause of the problem should only be the case if you are working with a very large string. If the string is not huge, it may be some problem with library or utility method that doesn't work well in a 64-bit environment.
Also, use a simple/small xml file to start with if you have nothing else to try.
Where do you initialize sb->length. Your problem is likely in strncpy(), though I don't know why the 32bit -> 64-bit O/S change would matter. Best bet is looking at the intermediate values and your problem will then be obvious.
being one of the developers of libsbml i just stumbled over this. Is this still a problem for you? In the meantime we have released libsbml 5, with separate 64bit and 32bit versions and improved testing a lot, please have a look at:
http://sf.net/projects/sbml/files/libsbml
The problem could be pretty much anything. True, it might be that strncpy does something bad, but most likely, it is simply passed a bad pointer. Which could originate from anywhere. A segfault (or access violation in Windows) simply means that the application tried to read or write to an address it did not have permission to access. So the real question is, where did that address come from? The function that tried to follow the pointer is probably ok. But it was passed a bad pointer from somewhere else. Probably.
Unfortunately, debugging C code is not trivial at the best of time. If the code isn't your own, that doesn't make it easier. :)
StringBuffer is defined as follows:
/**
* Creates a new StringBuffer and returns a pointer to it.
*/
LIBSBML_EXTERN
StringBuffer_t *
StringBuffer_create (unsigned long capacity)
{
StringBuffer_t *sb;
sb = (StringBuffer_t *) safe_malloc(sizeof(StringBuffer_t));
sb->buffer = (char *) safe_malloc(capacity + 1);
sb->capacity = capacity;
StringBuffer_reset(sb);
return sb;
}
A bit more of the stack trace is:
[ 0] 000000003CB3856E libsbml.dll+165230 (StringBuffer_append+000030)
[ 6] 000000003CB1BFAF libsbml.dll+049071 (EventAssignment_createWith+001631)
[ 12] 000000003CB1C1D7 libsbml.dll+049623 (SBML_formulaToString+000039)
[ 18] 000000003CB2C154 libsbml.dll+115028 (Rule::setFormulaFromMath+000036)
[ 20] 0000000001751913 libmx.dll+137491 (mxCheckMN_700+000723)
[ 25] 000000003CB1E7B2 libsbml.dll+059314 (KineticLaw_getFormula+000018)
[ 37] 0000000035727749 TranslateSBML.mexw64+030537 (mexFunction+009353)
Seems if it was in any of the StringBuffer_* functions, that would be in the stack trace. I disagree with how _ensureCapacity and _grow are implemented. None of the functions check if realloc works or not. Realloc failure will certainly cause a segfault. I would insert a check for null after _ensureCapacity. With the way _ensureCapacity and _grow are, it seems possible to get an off-by-one error. If you're running on Windows, the 64-bit and 32-bit systems may have different page protection mechanisms that cause it to fail. (You can often live through off-by-one errors in malloc'ed memory on systems with weak page protection.)
Let's assume that safe_malloc and safe_realloc do something sensible like aborting the program when they can't get the requested memory. That way your program won't continue executing with invalid pointers.
Now let's look at how big StringBuffer_ensureCapacity grows the buffer to, in comparison to the wanted capacity. It's not an off-by-one error. It's an off-by-a-factor-of-two error.
How did your program ever work in x32, I can't guess.
In response to bk1e's comment on the question - unfortunately, I need version 2.0.2 for use with the COBRA toolbox, which doesn't work with the newer version 3. So, I'm stuck with this older version for now.
I'm also hitting some walls attempting to debug - I'm building a .dll, so in addition to recompiling xerces to make sure it has the same debugging settings in MSVC++, I also need to attach to the Matlab process to do the debugging - it's a pretty big jump for my limited experience in this environment, and I haven't dug into it very far yet.
I had hoped there was some obvious syntax issue with the buffer allocation or expansion. Looks like I'm in for a few more days of pain, though.
unsigned long is not a safe type to use for sizes on a 64-bit machine in Windows. Unlike Linux, Windows defines long to be 32 bits on both 32- and 64-bit architectures. So if the buffer being appended to grows beyond 4 GB in size (or if you're trying to append a string whose length is >4GB), you need to change the unsigned long type declarations to size_t (which is 64 bits on 64-bit architectures, in all operating systems).
However, if you're only reading a 1.5 MB file, I don't see how you'd ever get a StringBuffer to exceed 4 GB in size, so this might be a blind alley.