MSVS 2010 C++ Compiler and Stack alignment issue? - c++

My problem is MSVS 2010 C++ compiler is generating code in a way after returning from a function call resolved in runtime(GetProcAddress+GetModuleHandle) from another dll the compiler then tries to align stack this way:
CALL DWORD PTR DS:[2000367C] ; apiresolvedinruntime.dll
ADD ESP,12 ; <- this is the stack alignment
This is of course overwriting the return address and my program crashes, can someone explain me why compiler aligning the stack when it really shouldn't do it?

You didn't call the runtime loaded function using the correct calling convention. Calling convention specifies the default handling of what happens to the stack. Most likely, the DLL was compiled using the __stdcall calling convention (which is what e.g. the Windows DLLs use), which specifies that the called function is supposed to clean up the stack, but the calling code was declared with a function pointer using the __cdecl calling convention (which is the default). Under __cdecl, functions support variadic arguments, so the caller needs to do the cleanup of the stack, because the called function does not know how many arguments are passed.
You need to verify that the DLL and the calling code are compiled using the same calling conventions.

Related

“The value of ESP was not properly saved across a function call.”

I have a DLL written in Delphi 7 that I need to use in Visual C++ 2008.
From documentation that came with DLL, I can see that function is declared as (Delphi 7):
function ReadInfo(pCOM, pBuf, pErr: Pointer):boolean;
where pCom is pointer to data structure:
TCOM = record
dwBaudRate: Longword;
nCom,
nErr,
nLang : Byte;
end;
pBuf is pointer to "array of byte" (as it is written in DLL's documentation).
pErr - not used.
So now in c++ (after successfully loading DLL with LoadLibrary), I call:
myFunc = (MY_FUNC_POINTER)GetProcAddress(dllHandle, "ReadInfo");
which also doesn't return any errors.
MY_FUNC_POINTER is defined as:
typedef bool (*MY_FUNC_POINTER)(TCOM*, BYTE*, void*);
where TCOM is:
struct TCOM
{
unsigned long dwBaudRate;
BYTE nComm;
BYTE nError;
BYTE nLanguage;
};
I defined:
TCOM MyCom;
BYTE *myRes;
myRes = new BYTE[1024*1024];
But after calling
myFunc(&MyCom, myRes, NULL)
I get “The value of ESP was not properly saved across a function call.” error.
There would appear to be a calling convention mismatch. On the face of it, the function declares no calling convention in the Delphi, so the default Borland register convention is used. Your C++ code does not declare a calling convention for the import so the default of cdecl is used. But it is plausible that the Delphi code and documentation are not aligned and the Delphi code actually uses a different calling convention. Check the Delphi code, or contact the vendor. No matter what, the error message that you report does indicate a binary mis-match across the boundary between your module and the other module.
If the function really does use the Borland register calling convention (but see below for more), then you cannot readily call the function from languages other than Delphi. In that case you'd need a bridge to adapt that to a standard calling convention such as stdcall. By that I mean a Delphi DLL that can call the original DLL and expose it's functionality a way suited to interop. A better solution would be to fix the root problem and build the DLL again using standard calling conventions.
In fact, I now suspect that all the other commentators are correct. I suspect that the Delphi documentation does not match the Delphi code. I suspect that the function really is stdcall. So you can, probably, solve your problem by changing the function pointer typedef to be as follows:
typedef bool (__stdcall *MY_FUNC_POINTER)(TCOM*, BYTE*, void*);
My reasoning for this is that in stdcall the callee is responsible for cleaning the stack. That's not the case for cdecl, and since all the parameters, and the return value, fit in registers, it's not the case for Delphi register calling convention, for this function. Since there is a stack pointer mis-match, it follows that the most likely explanation is that the Delphi function is stdcall.
All the same, it's not comfortable to be working out calling conventions this way. If you cannot get any help from the DLL vendor then I'd be inclined to dig a little deeper by looking at the DLL function's code under a disassembler.

stdcall and cdecl

There are (among others) two types of calling conventions - stdcall and cdecl. I have few questions on them:
When a cdecl function is called, how does a caller
know if it should free up the stack ? At the call site, does the
caller know if the function being called is a cdecl or a stdcall
function ? How does it work ? How does the caller know if it should
free up the stack or not ? Or is it the linkers responsibility ?
If a function which is declared as stdcall calls a function(which
has a calling convention as cdecl), or the other way round, would
this be inappropriate ?
In general, can we say that which call will be faster - cdecl or
stdcall ?
Raymond Chen gives a nice overview of what __stdcall and __cdecl does.
(1) The caller "knows" to clean up the stack after calling a function because the compiler knows the calling convention of that function and generates the necessary code.
void __stdcall StdcallFunc() {}
void __cdecl CdeclFunc()
{
// The compiler knows that StdcallFunc() uses the __stdcall
// convention at this point, so it generates the proper binary
// for stack cleanup.
StdcallFunc();
}
It is possible to mismatch the calling convention, like this:
LRESULT MyWndProc(HWND hwnd, UINT msg,
WPARAM wParam, LPARAM lParam);
// ...
// Compiler usually complains but there's this cast here...
windowClass.lpfnWndProc = reinterpret_cast<WNDPROC>(&MyWndProc);
So many code samples get this wrong it's not even funny. It's supposed to be like this:
// CALLBACK is #define'd as __stdcall
LRESULT CALLBACK MyWndProc(HWND hwnd, UINT msg
WPARAM wParam, LPARAM lParam);
// ...
windowClass.lpfnWndProc = &MyWndProc;
However, assuming the programmer doesn't ignore compiler errors, the compiler will generate the code needed to clean up the stack properly since it'll know the calling conventions of the functions involved.
(2) Both ways should work. In fact, this happens quite frequently at least in code that interacts with the Windows API, because __cdecl is the default for C and C++ programs according to the Visual C++ compiler and the WinAPI functions use the __stdcall convention.
(3) There should be no real performance difference between the two.
In CDECL arguments are pushed onto the stack in revers order, the caller clears the stack and result is returned via processor registry (later I will call it "register A"). In STDCALL there is one difference, the caller doeasn't clear the stack, the calle do.
You are asking which one is faster. No one. You should use native calling convention as long as you can. Change convention only if there is no way out, when using external libraries that requires certain convention to be used.
Besides, there are other conventions that compiler may choose as default one i.e. Visual C++ compiler uses FASTCALL which is theoretically faster because of more extensive usage of processor registers.
Usually you must give a proper calling convention signature to callback functions passed to some external library i.e. callback to qsort from C library must be CDECL (if the compiler by default uses other convention then we must mark the callback as CDECL) or various WinAPI callbacks must be STDCALL (whole WinAPI is STDCALL).
Other usual case may be when you are storing pointers to some external functions i.e. to create a pointer to WinAPI function its type definition must be marked with STDCALL.
And below is an example showing how does the compiler do it:
/* 1. calling function in C++ */
i = Function(x, y, z);
/* 2. function body in C++ */
int Function(int a, int b, int c) { return a + b + c; }
CDECL:
/* 1. calling CDECL 'Function' in pseudo-assembler (similar to what the compiler outputs) */
push on the stack a copy of 'z', then a copy of 'y', then a copy of 'x'
call (jump to function body, after function is finished it will jump back here, the address where to jump back is in registers)
move contents of register A to 'i' variable
pop all from the stack that we have pushed (copy of x, y and z)
/* 2. CDECL 'Function' body in pseudo-assembler */
/* Now copies of 'a', 'b' and 'c' variables are pushed onto the stack */
copy 'a' (from stack) to register A
copy 'b' (from stack) to register B
add A and B, store result in A
copy 'c' (from stack) to register B
add A and B, store result in A
jump back to caller code (a, b and c still on the stack, the result is in register A)
STDCALL:
/* 1. calling STDCALL in pseudo-assembler (similar to what the compiler outputs) */
push on the stack a copy of 'z', then a copy of 'y', then a copy of 'x'
call
move contents of register A to 'i' variable
/* 2. STDCALL 'Function' body in pseaudo-assembler */
pop 'a' from stack to register A
pop 'b' from stack to register B
add A and B, store result in A
pop 'c' from stack to register B
add A and B, store result in A
jump back to caller code (a, b and c are no more on the stack, result in register A)
I noticed a posting that say that it does not matter if you call a __stdcall from a __cdecl or visa versa. It does.
The reason: with __cdecl the arguments that are passed to the called functions are removed form the stack by the calling function, in __stdcall, the arguments are removed from the stack by the called function. If you call a __cdecl function with a __stdcall, the stack is not cleaned up at all, so eventually when the __cdecl uses a stacked based reference for arguments or return address will use the old data at the current stack pointer. If you call a __stdcall function from a __cdecl, the __stdcall function cleans up the arguments on the stack, and then the __cdecl function does it again, possibly removing the calling functions return information.
The Microsoft convention for C tries to circumvent this by mangling the names. A __cdecl function is prefixed with an underscore. A __stdcall function prefixes with an underscore and suffixed with an at sign “#” and the number of bytes to be removed. Eg __cdecl f(x) is linked as _f, __stdcall f(int x) is linked as _f#4 where sizeof(int) is 4 bytes)
If you manage to get past the linker, enjoy the debugging mess.
I want to improve on #adf88's answer. I feel that pseudocode for the STDCALL does not reflect the way of how it happens in reality. 'a', 'b', and 'c' aren't popped from the stack in the function body. Instead they are popped by the ret instruction (ret 12 would be used in this case) that in one swoop jumps back to the caller and at the same time pops 'a', 'b', and 'c' from the stack.
Here is my version corrected according to my understanding:
STDCALL:
/* 1. calling STDCALL in pseudo-assembler (similar to what the compiler outputs) */
push on the stack a copy of 'z', then copy of 'y', then copy of 'x'
call
move contents of register A to 'i' variable
/* 2. STDCALL 'Function' body in pseaudo-assembler */
copy 'a' (from stack) to register A
copy 'b' (from stack) to register B
add A and B, store result in A
copy 'c' (from stack) to register B
add A and B, store result in A
jump back to caller code and at the same time pop 'a', 'b' and 'c' off the stack (a, b and
c are removed from the stack in this step, result in register A)
It's specified in the function type. When you have a function pointer, it's assumed to be cdecl if not explicitly stdcall. This means that if you get a stdcall pointer and a cdecl pointer, you can't exchange them. The two function types can call each other without issues, it's just getting one type when you expect the other. As for speed, they both perform the same roles, just in a very slightly different place, it's really irrelevant.
The caller and the callee need to use the same convention at the point of invokation - that's the only way it could reliably work. Both the caller and the callee follow a predefined protocol - for example, who needs to clean up the stack. If conventions mismatch your program runs into undefined behavior - likely just crashes spectacularly.
This is only required per invokation site - the calling code itself can be a function with any calling convention.
You shouldn't notice any real difference in performance between those conventions. If that becomes a problem you usually need to make less calls - for example, change the algorithm.
Those things are Compiler- and Platform-specific. Neither the C nor the C++ standard say anything about calling conventions except for extern "C" in C++.
how does a caller know if it should free up the stack ?
The caller knows the calling convention of the function and handles the call accordingly.
At the call site, does the caller know if the function being called is a cdecl or a stdcall function ?
Yes.
How does it work ?
It is part of the function declaration.
How does the caller know if it should free up the stack or not ?
The caller knows the calling conventions and can act accordingly.
Or is it the linkers responsibility ?
No, the calling convention is part of a function's declaration so the compiler knows everything it needs to know.
If a function which is declared as stdcall calls a function(which has a calling convention as cdecl), or the other way round, would this be inappropriate ?
No. Why should it?
In general, can we say that which call will be faster - cdecl or stdcall ?
I don't know. Test it.
a) When a cdecl function is called by the caller, how does a caller know if it should free up the stack?
The cdecl modifier is part of the function prototype (or function pointer type etc.) so the caller get the info from there and acts accordingly.
b) If a function which is declared as stdcall calls a function(which has a calling convention as cdecl), or the other way round, would this be inappropriate?
No, it's fine.
c) In general, can we say that which call will be faster - cdecl or stdcall?
In general, I would refrain from any such statements. The distinction matters eg. when you want to use va_arg functions. In theory, it could be that stdcall is faster and generates smaller code because it allows to combine popping the arguments with popping the locals, but OTOH with cdecl, you can do the same thing, too, if you're clever.
The calling conventions that aim to be faster usually do some register-passing.
Calling conventions have nothing to do with the C/C++ programming languages and are rather specifics on how a compiler implements the given language. If you consistently use the same compiler, you never need to worry about calling conventions.
However, sometimes we want binary code compiled by different compilers to inter-operate correctly. When we do so we need to define something called the Application Binary Interface (ABI). The ABI defines how the compiler converts the C/C++ source into machine-code. This will include calling conventions, name mangling, and v-table layout. cdelc and stdcall are two different calling conventions commonly used on x86 platforms.
By placing the information on the calling convention into the source header, the compiler will know what code needs to be generated to inter-operate correctly with the given executable.

What is the meaning and usage of __stdcall?

I've come across __stdcall a lot these days.
MSDN doesn't explain very clearly what it really means, when and why should it be used, if at all.
I would appreciate if someone would provide an explanation, preferably with an example or two.
This answer covers 32-bit mode. (Windows x64 only uses 2 conventions: the normal one (which is called __fastcall if it has a name at all) and __vectorcall, which is the same except for how SIMD vector args like __m128i are passed).
Traditionally, C function calls are made with the caller pushing some parameters onto the stack, calling the function, and then popping the stack to clean up those pushed arguments.
/* example of __cdecl */
push arg1
push arg2
push arg3
call function
add esp,12 ; effectively "pop; pop; pop"
Note: The default convention — shown above — is known as __cdecl.
The other most popular convention is __stdcall. In it the parameters are again pushed by the caller, but the stack is cleaned up by the callee. It is the standard convention for Win32 API functions (as defined by the WINAPI macro in <windows.h>), and it's also sometimes called the "Pascal" calling convention.
/* example of __stdcall */
push arg1
push arg2
push arg3
call function // no stack cleanup - callee does this
This looks like a minor technical detail, but if there is a disagreement on how the stack is managed between the caller and the callee, the stack will be destroyed in a way that is unlikely to be recovered.
Since __stdcall does stack cleanup, the (very tiny) code to perform this task is found in only one place, rather than being duplicated in every caller as it is in __cdecl. This makes the code very slightly smaller, though the size impact is only visible in large programs.
(Optimizing compilers can sometimes leave space for args allocated across multiple cdecl calls made from the same function and mov args into it, instead of always add esp, n / push. That saves instructions but can increase code-size. For example gcc -maccumulate-outgoing-args always does this, and was good for performance on older CPUs before push was efficient.)
Variadic functions like printf() are impossible to get right with __stdcall, because only the caller really knows how many arguments were passed in order to clean them up. The callee can make some good guesses (say, by looking at a format string), but it's legal in C to pass more args to printf than the format-string references (they'll be silently ignored). Hence only __cdecl supports variadic functions, where the caller does the cleanup.
Linker symbol name decorations:
As mentioned in a bullet point above, calling a function with the "wrong" convention can be disastrous, so Microsoft has a mechanism to avoid this from happening. It works well, though it can be maddening if one does not know what the reasons are.
They have chosen to resolve this by encoding the calling convention into the low-level function names with extra characters (which are often called "decorations"), and these are treated as unrelated names by the linker. The default calling convention is __cdecl, but each one can be requested explicitly with the /G? parameter to the compiler.
__cdecl (cl /Gd ...)
All function names of this type are prefixed with an underscore, and the number of parameters does not really matter because the caller is responsible for stack setup and stack cleanup. It is possible for a caller and callee to be confused over the number of parameters actually passed, but at least the stack discipline is maintained properly.
__stdcall (cl /Gz ...)
These function names are prefixed with an underscore and appended with # plus the number of bytes of parameters passed. By this mechanism, it's not possible to call a function with the wrong amount of parameters. The caller and callee definitely agree on returning with a ret 12 instruction for example, to pop 12 bytes of stack args along with the return address.
You'll get a link-time or runtime DLL error instead of having a function return with ESP pointing somewhere the caller isn't expecting. (For example if you added a new arg and didn't recompile both the main program and the library. Assuming you didn't fool the system by making an earlier arg narrower, like int64_t -> int32_t.)
__fastcall (cl /Gr ...)
These function names start with an # sign and are suffixed with the #bytes count, much like __stdcall. The first 2 args are passed in ECX and EDX, the rest are passed on the stack. The byte count includes the register args. As with __stdcall, a narrow arg like char still uses up a 4-byte arg-passing slot (a register, or a dword on the stack).
Examples:
Declaration -----------------------> decorated name
void __cdecl foo(void); -----------------------> _foo
void __cdecl foo(int a); -----------------------> _foo
void __cdecl foo(int a, int b); -----------------------> _foo
void __stdcall foo(void); -----------------------> _foo#0
void __stdcall foo(int a); -----------------------> _foo#4
void __stdcall foo(int a, int b); -----------------------> _foo#8
void __fastcall foo(void); -----------------------> #foo#0
void __fastcall foo(int a); -----------------------> #foo#4
void __fastcall foo(int a, int b); -----------------------> #foo#8
Note that in C++, the normal name-mangling mechanism that allows function overloading is used instead of #8, not as well. So you'll only see actual numbers in extern "C" functions. For example, https://godbolt.org/z/v7EaWs for example.
All functions in C/C++ have a particular calling convention. The point of a calling convention is to establish how data is passed between the caller and callee and who is responsible for operations such as cleaning out the call stack.
The most popular calling conventions on windows are
__stdcall, Pushes parameters on the stack, in reverse order (right to left)
__cdecl, Pushes parameters on the stack, in reverse order (right to left)
__clrcall, Load parameters onto CLR expression stack in order (left to right).
__fastcall, Stored in registers, then pushed on stack
__thiscall, Pushed on stack; this pointer stored in ECX
Adding this specifier to the function declaration essentially tells the compiler that you want this particular function to have this particular calling convention.
The calling conventions are documented here
https://learn.microsoft.com/en-us/cpp/cpp/calling-conventions
Raymond Chen also did a long series on the history of the various calling conventions (5 parts) starting here.
https://devblogs.microsoft.com/oldnewthing/20040102-00/?p=41213
__stdcall is a calling convention: a way of determining how parameters are passed to a function (on the stack or in registers) and who is responsible for cleaning up after the function returns (the caller or the callee).
Raymond Chen wrote a blog about the major x86 calling conventions, and there's a nice CodeProject article too.
For the most part, you shouldn't have to worry about them. The only case in which you should is if you're calling a library function that uses something other than the default -- otherwise the compiler will generate the wrong code and your program will probably crash.
Unfortunately, there is no easy answer for when to use it and when not.
__stdcall means that the arguments to a function are pushed onto the stack from the first to the last. This is as opposed to __cdecl, which means that the arguments are pushed from last to first, and __fastcall, which places the first four (I think) arguments in registers, and the rest go on the stack.
You just need to know what the callee expects, or if you are writing a library, what your callers are likely expect, and make sure you document your chosen convention.
That's a calling convention that WinAPI functions need to be called properly. A calling convention is a set of rules on how the parameters are passed into the function and how the return value is passed from the function.
If the caller and the called code use different conventions you run into undefined behaviour (like such a strange-looking crash).
C++ compilers don't use __stdcall by default - they use other conventions. So in order to call WinAPI functions from C++ you need to specify that they use __stdcall - this is usually done in Windoes SDK header files and you also do it when declaring function pointers.
It specifies a calling convention for a function. A calling convention is a set of rules how parameters are passed to a function: in which order, per address or per copy, who is to clean up the parameters (caller or callee) etc.
__stdcall denotes a calling convention (see this PDF for some details). This means it specifies how function arguments are pushed and popped from the stack, and who is responsible.
__stdcall is just one of several calling conventions, and is used throughout the WINAPI. You must use it if you provide function pointers as callbacks for some of those functions. In general, you do not need to denote any specific calling convention in your code, but just use the compiler's default, except for the case noted above (providing callbacks to 3rd party code).
simply put when you call function, it gets loaded in stack/register. __stdcall is one convention/way(right argument first, then left argument ...), __decl is another convention that are used to load the function on the stack or registers.
If you use them you instruct the computer to use that specific way to load/unload the function during linking and hence you would not get a mismatch/crash.
Otherwise the function-callee and function-caller might use different conventions causing program to crash.
__stdcall is the calling convention used for the function. This tells the compiler the rules that apply for setting up the stack, pushing arguments and getting a return value. There are a number of other calling conventions like __cdecl, __thiscall, __fastcall and __naked.
__stdcall is the standard calling convention for Win32 system calls.
More details can be found on Wikipedia.

Why do thread functions need to be declared as '__cdecl'?

Sample code that shows how to create threads using MFC declares the thread function as both static and __cdecl. Why is the latter required? Boost threads don't bother with this convention, so is it just an anachronism?
For example (MFC):
static __cdecl UINT MyFunc(LPVOID pParam)
{
...
}
CWinThread* pThread = AfxBeginThread(MyFunc, ...);
Whereas Boost:
static void func()
{
...
}
boost::thread t;
t.create(&func);
(the code samples might not be 100% correct as I am nowhere near an IDE).
What is the point of __cdecl? How does it help when creating threads?
__cdecl tells the compiler to use the C calling convention (as opposed to the stdcall, fastcall or whatever other calling convention your compiler supports). I believe, VC++ uses stdcall by default.
The calling convention affects things such as how arguments are pushed onto the stack (or registers, in the case of fastcall) and who pops arguments off the stack (caller or callee).
In the case of Boost. I believe it uses template specialization to figure out the appropriate function type and calling convention.
Look at the prototype for AfxBeginThread():
CWinThread* AfxBeginThread(
AFX_THREADPROC pfnThreadProc,
LPVOID pParam,
int nPriority = THREAD_PRIORITY_NORMAL,
UINT nStackSize = 0,
DWORD dwCreateFlags = 0,
LPSECURITY_ATTRIBUTES lpSecurityAttrs = NULL
);
AFX_THREADPROC is a typedef for UINT(AFX_CDECL*)(LPVOID). When you pass a function to AfxBeginThread(), it must match that prototype, including the calling convention.
The MSDN pages on __cdecl and __stdcall (as well as __fastcall and __thiscall) explain the pros and cons of each calling convention.
The boost::thread constructor uses templates to allow you to pass a function pointer or callable function object, so it doesn't have the same restrictions as MFC.
Because your thread is going to be called by a runtime function that manages this for you, and that function expects it to be that way. Boost designed it a different way.
Put a breakpoint at the start of your thread function and look at the stack when it gets called, you'll see the runtime function that calls you.
C/C++ compilers by default use the C calling convention (pushing rightmost param first on the stack) for it allows working with functions with variable argument number as printf.
The Pascal calling convention (aka "fastcall") pushes leftmost param first. This is quicker though costs you the possibility of easy variable argument functions (I read somewhere they're still possible, though you need to use some tricks).
Due to the speed resulting from using the Pascal convention, both Win32 and MacOS APIs by default use that calling convention, except in certain cases.
If that function has only one param, in theory using either calling convention would be legal, though the compiler may enforce the same calling convention is used to avoid any problem.
The boost libraries were designed with an eye on portability, so they should be agnostic as to which caller convention a particular compiler is using.
The real answer has to do with how windows internally calls the thread proc routine, and it is expecting the function to abide by a specific calling convention, which in this case is a macro, WINAPI, which according to my system is defined as:
#define WINAPI __stdcall
This means that the called function is responsible for cleaning up the stack. The reason why boost::thread is able to support arbitrary functions is that it passes a pointer to the function object used in the call to thread::create function to CreateThread. The threadproc associated with the thread simply calls operator() on the function object.
The reason MFC requires __cdecl therefore has to do with the way it internally calls the function passed in to the call to AfxBeginThread. There is no good reason to do this unless they were planning on allowing vararg parameters...

Weird MSC 8.0 error: "The value of ESP was not properly saved across a function call..."

We recently attempted to break apart some of our Visual Studio projects into libraries, and everything seemed to compile and build fine in a test project with one of the library projects as a dependency. However, attempting to run the application gave us the following nasty run-time error message:
Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function pointer declared with a different calling convention.
We have never even specified calling conventions (__cdecl etc.) for our functions, leaving all the compiler switches on the default. I checked and the project settings are consistent for calling convention across the library and test projects.
Update: One of our devs changed the "Basic Runtime Checks" project setting from "Both (/RTC1, equiv. to /RTCsu)" to "Default" and the run-time vanished, leaving the program running apparently correctly. I do not trust this at all. Was this a proper solution, or a dangerous hack?
This debug error means that the stack pointer register is not returned to its original value after the function call, i.e. that the number of pushes before the function call were not followed by the equal number of pops after the call.
There are 2 reasons for this that I know (both with dynamically loaded libraries). #1 is what VC++ is describing in the error message, but I don't think this is the most often cause of the error (see #2).
1) Mismatched calling conventions:
The caller and the callee do not have a proper agreement on who is going to do what. For example, if you're calling a DLL function that is _stdcall, but you for some reason have it declared as a _cdecl (default in VC++) in your call. This would happen a lot if you're using different languages in different modules etc.
You would have to inspect the declaration of the offending function, and make sure it is not declared twice, and differently.
2) Mismatched types:
The caller and the callee are not compiled with the same types. For example, a common header defines the types in the API and has recently changed, and one module was recompiled, but the other was not--i.e. some types may have a different size in the caller and in the callee.
In that case, the caller pushes the arguments of one size, but the callee (if you're using _stdcall where the callee cleans the stack) pops the different size. The ESP is not, thus, returned to the correct value.
(Of course, these arguments, and others below them, would seem garbled in the called function, but sometimes you can survive that without a visible crash.)
If you have access to all the code, simply recompile it.
I read this in other forum
I was having the same problem, but I just FIXED it. I was getting the same error from the following code:
HMODULE hPowerFunctions = LoadLibrary("Powrprof.dll");
typedef bool (*tSetSuspendStateSig)(BOOL, BOOL, BOOL);
tSetSuspendState SetSuspendState = (tSuspendStateSig)GetProcAddress(hPowerfunctions, "SetSuspendState");
result = SetSuspendState(false, false, false); <---- This line was where the error popped up.
After some investigation, I changed one of the lines to:
typedef bool (WINAPI*tSetSuspendStateSig)(BOOL, BOOL, BOOL);
which solved the problem. If you take a look in the header file where SetSuspendState is found (powrprof.h, part of the SDK), you will see the function prototype is defined as:
BOOLEAN WINAPI SetSuspendState(BOOLEAN, BOOLEAN, BOOLEAN);
So you guys are having a similar problem. When you are calling a given function from a .dll, its signature is probably off. (In my case it was the missing WINAPI keyword).
Hope that helps any future people! :-)
Cheers.
Silencing the check is not the right solution. You have to figure out what is messed up with your calling conventions.
There are quite a few ways to change the calling convetion of a function without explicitly specifying it. extern "C" will do it, STDMETHODIMP/IFACEMETHODIMP will also do it, other macros might do it as well.
I believe if run your program under WinDBG (http://www.microsoft.com/whdc/devtools/debugging/default.mspx), the runtime should break at the point where you hit that problem. You can look at the call stack and figure out which function has the problem and then look at its definition and the declaration that the caller uses.
I saw this error when the code tried to call a function on an object that was not of the expected type.
So, class hierarchy: Parent with children: Child1 and Child2
Child1* pMyChild = 0;
...
pMyChild = pSomeClass->GetTheObj();// This call actually returned a Child2 object
pMyChild->SomeFunction(); // "...value of ESP..." error occurs here
I was getting similar error for AutoIt APIs which i was calling from VC++ program.
typedef long (*AU3_RunFn)(LPCWSTR, LPCWSTR);
However, when I changed the declaration which includes WINAPI, as suggested earlier in the thread, problem vanished.
Code without any error looks like:
typedef long (WINAPI *AU3_RunFn)(LPCWSTR, LPCWSTR);
AU3_RunFn _AU3_RunFn;
HINSTANCE hInstLibrary = LoadLibrary("AutoItX3.dll");
if (hInstLibrary)
{
_AU3_RunFn = (AU3_RunFn)GetProcAddress(hInstLibrary, "AU3_WinActivate");
if (_AU3_RunFn)
_AU3_RunFn(L"Untitled - Notepad",L"");
FreeLibrary(hInstLibrary);
}
It's worth pointing out that this can also be a Visual Studio bug.
I got this issue on VS2017, Win10 x64. At first it made sense, since I was doing weird things casting this to a derived type and wrapping it in a lambda. However, I reverted the code to a previous commit and still got the error, even though it wasn't there before.
I tried restarting and then rebuilding the project, and then the error went away.
I was getting this error calling a function in a DLL which was compiled with a pre-2005 version of Visual C++ from a newer version of VC (2008).
The function had this signature:
LONG WINAPI myFunc( time_t, SYSTEMTIME*, BOOL* );
The problem was that time_t's size is 32 bits in pre-2005 version, but 64 bits since VS2005 (is defined as _time64_t). The call of the function expects a 32 bit variable but gets a 64 bit variable when called from VC >= 2005. As parameters of functions are passed via the stack when using WINAPI calling convention, this corrupts the stack and generates the above mentioned error message ("Run-Time Check Failure #0 ...").
To fix this, it is possible to
#define _USE_32BIT_TIME_T
before including the header file of the DLL or -- better -- change the signature of the function in the header file depending on the VS version (pre-2005 versions don't know _time32_t!):
#if _MSC_VER >= 1400
LONG WINAPI myFunc( _time32_t, SYSTEMTIME*, BOOL* );
#else
LONG WINAPI myFunc( time_t, SYSTEMTIME*, BOOL* );
#endif
Note that you need to use _time32_t instead of time_t in the calling program, of course.
I was having this exact same error after moving functions to a dll and dynamically loading the dll with LoadLibrary and GetProcAddress. I had declared extern "C" for the function in the dll because of the decoration. So that changed calling convention to __cdecl as well. I was declaring function pointers to be __stdcall in the loading code. Once I changed the function pointer from __stdcall to__cdecl in the loading code the runtime error went away.
Are you creating static libs or DLLs? If DLLs, how are the exports defined; how are the import libraries created?
Are the prototypes for the functions in the libs exactly the same as the function declarations where the functions are defined?
do you have any typedef'd function prototypes (eg int (*fn)(int a, int b) )
if you dom you might be have gotten the prototype wrong.
ESP is an error on the calling of a function (can you tell which one in the debugger?) that has a mismatch in the parameters - ie the stack has restored back to the state it started in when you called the function.
You can also get this if you're loading C++ functions that need to be declared extern C - C uses cdecl, C++ uses stdcall calling convention by default (IIRC). Put some extern C wrappers around the imported function prototypes and you may fix it.
If you can run it in the debugger, you'll see the function immediatey. If not, you can set DrWtsn32 to create a minidump that you can load into windbg to see the callstack at the time of the error (you'll need symbols or a mapfile to see the function names though).
Another case where esp can get messed up is with an inadvertent buffer overflow, usually through mistaken use of pointers to work past the boundary of an array. Say you have some C function that looks like
int a, b[2];
Writing to b[3] will probably change a, and anywhere past that is likely to hose the saved esp on the stack.
You would get this error if the function is invoked with a calling convention other than the one it is compiled to.
Visual Studio uses a default calling convention setting thats decalred in the project's options. Check if this value is the same in the orignal project settings and in the new libraries. An over ambitious dev could have set this to _stdcall/pascal in the original since it reduces the code size compared to the default cdecl. So the base process would be using this setting and the new libraries get the default cdecl which causes the problem
Since you have said that you do not use any special calling conventions this seems to be a good probability.
Also do a diff on the headers to see if the declarations / files that the process sees are the same ones that the libraries are compiled with .
ps : Making the warning go away is BAAAD. the underlying error still persists.
This happened to me when accessing a COM object (Visual Studio 2010). I passed the GUID for another interface A for in my call to QueryInterface, but then I cast the retrieved pointer as interface B. This resulted in making a function call to one with an entirely signature, which accounts for the stack (and ESP) being messed up.
Passing the GUID for interface B fixed the problem.
In my MFC C++ app I am experiencing the same problem as reported in Weird MSC 8.0 error: “The value of ESP was not properly saved across a function call…”. The posting has over 42K views and 16 answers/comments none of which blamed the compiler as the problem. At least in my case I can show that the VS2015 compiler is at fault.
My dev and test setup is the following: I have 3 PCs all of which run Win10 version 10.0.10586. All are compiling with VS2015, but here is the difference. Two of the VS2015s have Update 2 while the other has Update 3 applied. The PC with Update 3 works, but the other two with Update 2 fail with the same error as reported in the posting above. My MFC C++ app code is exactly the same on all three PCs.
Conclusion: at least in my case for my app the compiler version (Update 2) contained a bug that broke my code. My app makes heavy use of std::packaged_task so I expect the problem was in that fairly new compiler code.
ESP is the stack pointer. So according to the compiler, your stack pointer is getting messed up. It is hard to say how (or if) this could be happening without seeing some code.
What is the smallest code segment you can get to reproduce this?
If you're using any callback functions with the Windows API, they must be declared using CALLBACK and/or WINAPI. That will apply appropriate decorations to make the compiler generate code that cleans the stack correctly. For example, on Microsoft's compiler it adds __stdcall.
Windows has always used the __stdcall convention as it leads to (slightly) smaller code, with the cleanup happening in the called function rather than at every call site. It's not compatible with varargs functions, though (because only the caller knows how many arguments they pushed).
Here's a stripped down C++ program that produces that error. Compiled using (Microsoft Visual Studio 2003) produces the above mentioned error.
#include "stdafx.h"
char* blah(char *a){
char p[1];
strcat(p, a);
return (char*)p;
}
int main(){
std::cout << blah("a");
std::cin.get();
}
ERROR:
"Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention."
I had this same problem here at work. I was updating some very old code that was calling a FARPROC function pointer. If you don't know, FARPROC's are function pointers with ZERO type safety. It's the C equivalent of a typdef'd function pointer, without the compiler type checking.
So for instance, say you have a function that takes 3 parameters. You point a FARPROC to it, and then call it with 4 parameters instead of 3. The extra parameter pushed extra garbage onto the stack, and when it pops off, ESP is now different than when it started. So I solved it by removing the extra parameter to the invocation of the FARPROC function call.
Not the best answer but I just recompiled my code from scratch (rebuild in VS) and then the problem went away.