Today, I was checking source code of the CRT library of Visual Studio 2008 and 2010 and I found a bug into the file mtdll.h. The problem is with the macro FLS_GETVALUE. On x86 systems, this macro call directly TlsGetValue instead of making a call to the function assigned to the variable gpFlsGetValue.
First of all, it's a problem because FlsAlloc, FlsGetValue, FlsFree and FlsSetValue are not available on all system( Windows Vista+ and Windows Server 2003+). These functions behave like TlsAlloc, TlsGetValue, TlsFree and TlsSetValue but support Fibers(kind of user thread). Because we should prefer using Fls* functions instead of Tls functions, the C runtime library of VS check if the Fls functions are availables when the process or the dll is loaded and initialise the gpFls* variables. If the Fls* are not available, the CRT initialise these variables with the equivalent Tls* functions.
You cannot use the macro FLS_GETVALUE directly into your code because it's used internally by the runtime library. The runtime use this macro to initialise per thread data of the runtime library.
My question is about the impact of this bug. I know that fiber are not widely used, but if you develop a DLL that is used inside an application that use fiber, what can be the impact? Does this bug can cause a crash of the application or only produce false results? Does this problem can cause problem with widely used application like SQL Server? What do you think? Does this bug can be a security risk? Does IIS or ASP.Net use fiber that can cause a crash?
For curious people, here the current source of the problematic line in mtdll.h:
#define FLS_GETVALUE ((PFLS_GETVALUE_FUNCTION)TlsGetValue(__getvalueindex))
Here a fixed version of the macro FLS_GETVALUE :
#define FLS_GETVALUE (((PFLS_GETVALUE_FUNCTION)DecodePointer(gpFlsGetValue))(__getvalueindex))
For now, I should find how to submit a bug to Microsoft.
I'm not sure about that. The CRT seems to handle FlsGetValue differently from the other FLS functions. It looks like the CRT is storing a pointer to the FlsGetValue function in TLS (pseudo-code):
void init_fls()
{
FLS_GET_VALUE_PROC proc = GetProcAddress(kernel32, "FlsGetValue");
if(!proc)
{
// FlsGetValue not implemented on this platform
// use alternative implementation provided by CRT
proc = __crtFlsGetValue;
}
// store pointer to FlsGetValue proc in TLS
TlsSetValue(fls_get_value_index, proc);
}
void* get_fls_value(int index)
{
// retrieve pointer to FlsGetValue proc from TLS
FLS_GET_VALUE_PROC proc = TlsGetValue(fls_get_value_index);
return proc(index);
}
Related
I have a library of code I'm working on upgrading from x86 to x64 for a Windows application.
Part of the code took advantage of MSVC inline assembly blocks. I'm not looking to go through and interpret the assembly but I am looking to keep functionality from this part of the application.
Can I compile the functions using the inline assembly using GCC to make a DLL and link that to the rest of the library?
EDIT 1:(7/7/21) The flexibility with which compiler the project uses is open and I am currently looking into using Clang for use with MSVC.(also the Intel C++ compiler as another possibility) As stated in the first sentence it is a Windows application that I want to keep on Windows and the purpose of using another compiler is due to me 1.) not wanting to rewrite the large amount of assembly and 2.) because I know that MSVC does not support x64 inline assembly. So far clang seems to be working with a couple issues of how it declares comments inside of the assembly block and a few commands. The function is built around doing mathematical operations on a block of data, in what was supposed to be as fast as possible when it was developed but now that it works as intended I'm not looking to upgrade just maintain functionality. So, any compiler that will support inline assembly is an option.
EDIT 2:(7/7/21) I forgot to mention in the first edit, I'm not necessarily looking to load the 32-bit DLL into another process because I'm worried about copying data into an out of shared memory. I've done a similar solution for another project but the data set is around 8 MB and I'm worried that slow copy times for the function would cause the time constraint on the math to cause issues in the runtime of the application.(slow, laggy, and buffering are effects I'm trying to avoid.) I'm not trying to make it any faster but it definitely can't get any slower.
In theory, if you manage to create a plain C interface for that DLL (all exported symbols from DLL are standard C functions) and don't use memory management functions across "border" (no mixed memory management) then you should be able to dynamically load that DLL from another another (MSVC) process and call its functions, at least.
Not sure about statically linking against it... probably not, because the compiler and linker must go hand in hand (MSVC compiler+MSVC linker or GCC compiler+GCC linker) . The output of GCC linker is probably not compatible with MSVC at least regarding name mangling.
Here is how I would structure it (without small details):
Header.h (separate header to be included in both DLL and EXE)
//... remember to use your preferred calling convention but be consistent about it
struc Interface{
void (*func0)();
void (*func1)(int);
//...
};
typedef Interface* (*GetInterface)();
DLL (gcc)
#include "Header.h"
//functions implementing specific functionality (not exported)
void f0)(){/*...*/}
void f1)(int){/*...*/}
//...
Interface* getInterface(){//this must be exported from DLL (compiler specific)
static Interface interface;
//initialize functions pointers from interface with corresponding functions
interface.func0 = &f0;
interface.func1 = &f1;
//...
return &interface;
}
EXE (MSVC)
#include "Header.h"
int main(){
auto dll = LoadLibrary("DLL.dll");
auto getDllInterface = (GetInstance)GetProcAddress(dll, "getInterface");
auto* dllInterface = getDllInterface();
dllInterface->func0();
dllInterface->func1(123);
//...
return 0;
}
I have a C++ client to a C++/CLI DLL, which initializes a series of C# dlls.
This used to work. The code that is failing has not changed. The code that has changed is not called before the exception is thrown. My compile environment has changed, but recompiling on a machine with an environment similar to my old one still failed. (EDIT: as we see in the answer this is not entirely true, I was only recompiling the library in the old environment, not the library and client together. The client projects had been upgraded and couldn't easily go back.)
Someone besides me recompiled the library, and we started getting memory management issues. The pointer passed in as a String must not be in the bottom 64K of the process's address space. I recompiled it, and all worked well with no code changes. (Alarm #1) Recently it was recompiled, and memory management issues with strings re-appeared, and this time they're not going away. The new error is Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
I'm pretty sure the problem is not located where I see the exception, the code didn't change between the successful and failing builds, but we should review that to be complete. Ignore the names of things, I don't have much control over the design of what it's doing with these strings. And sorry for the confusion, but note that _bridge and bridge are different things. Lots of lines of code missing because this question is already too long.
Defined in library:
struct Config
{
std::string aye;
std::string bee;
std::string sea;
};
extern "C" __declspec(dllexport) BridgeBase_I* __stdcall Bridge_GetConfiguredDefaultsImplementationPointer(
const std::vector<Config> & newConfigs, /**< new configurations to apply **/
std::string configFolderPath, /**< folder to write config files in **/
std::string defaultConfigFolderPath, /**< folder to find default config files in **/
std::string & status /**< output status of config parse **/
);
In client function:
GatewayWrapper::Config bridge;
std::string configPath("./config");
std::string defaultPath("./config/default");
GatewayWrapper::Config gwtransport;
bridge.aye = "bridged.dll";
bridge.bee = "1.0";
bridge.sea = "";
configs.push_back(bridge);
_bridge = GatewayWrapper::Bridge_GetConfiguredDefaultsImplementationPointer(configs, configPath, defaultPath, status);
Note that call to library that is crashing is in the same scope as the vector declaration, struct declaration, string assignment and vector push-back
There are no threading calls in this section of code, but there are other threads running doing other things. There is no pointer math here, there are no heap allocations in the area except perhaps inside the standard library.
I can run the code up to the Bridge_GetConfiguredDefaultsImplementationPointer call in the debugger, and the contents of the configs vector look correct in the debugger.
Back in the library, in the first sub function, where the debugger don't shine, I've broken down the failing statement into several console prints.
System::String^ temp
List<CConfig^>^ configs = gcnew List<CConfig ^>((INT32)newConfigs.size());
for( int i = 0; i< newConfigs.size(); i++)
{
std::cout << newConfigs[i].aye<< std::flush; // prints
std::cout << newConfigs[i].aye.c_str() << std::flush; // prints
temp = gcnew System::String(newConfigs[i].aye.c_str());
System::Console::WriteLine(temp); // prints
std::cout << "Testing string creation" << std::endl; // prints
std::cout << newConfigs[i].bee << std::flush; // crashes here
}
I get the same exception on access of bee if I move the newConfigs[i].bee out above the assignment of temp or comment out the list declaration/assignment.
Just for reference that the std::string in a struct in a vector should have arrived at it's destination ok
Is std::vector copying the objects with a push_back?
std::string in struct - Copy/assignment issues?
http://www.cplusplus.com/reference/vector/vector/operator=/
Assign one struct to another in C
Why this exception is not caught by my try/catch
https://stackoverflow.com/a/918891/2091951
Generic AccessViolationException related questions
How to handle AccessViolationException
Programs randomly getting System.AccessViolationException
https://connect.microsoft.com/VisualStudio/feedback/details/819552/visual-studio-debugger-throws-accessviolationexception
finding the cause of System.AccessViolationException
https://msdn.microsoft.com/en-us/library/ms164911.aspx
Catching access violation exceptions?
AccessViolationException when using C++ DLL from C#
Suggestions in above questions
Change to .net 3.5, change target platform - these solutions could have serious issues with a large mult-project solution.
HandleProcessCorruptedStateExceptions - does not work in C++, this decoration is for C#, catching this error could be a very bad idea anyway
Change legacyCorruptedStateExceptionsPolicy - this is about catching the error, not preventing it
Install .NET 4.5.2 - can't, already have 4.6.1. Installing 4.6.2 did not help. Recompiling on a different machine that didn't have 4.5 or 4.6 installed did not help. (Despite this used to compile and run on my machine before installing Visual Studio 2013, which strongly suggests the .NET library is an issue?)
VSDebug_DisableManagedReturnValue - I only see this mentioned in relation to a specific crash in the debugger, and the help from Microsoft says that other AccessViolationException issues are likely unrelated. (http://connect.microsoft.com/VisualStudio/feedbackdetail/view/819552/visual-studio-debugger-throws-accessviolationexception)
Change Comodo Firewall settings - I don't use this software
Change all the code to managed memory - Not an option. The overall design of calling C# from C++ through C++/CLI is resistant to change. I was specifically asked to design it this way to leverage existing C# code from existing C++ code.
Make sure memory is allocated - memory should be allocated on the stack in the C++ client. I've attempted to make the vector be not a reference parameter, to force a vector copy into explicitly library controlled memory space, did not help.
"Access violations in unmanaged code that bubble up to managed code are always wrapped in an AccessViolationException." - Fact, not a solution.
but it was the mismatch, not the specific version that was the problem
Yes, that's black letter law in VS. You unfortunately just missed the counter-measures that were built into VS2012 to turn this mistake into a diagnosable linker error. Previously (and in VS2010), the CRT would allocate its own heap with HeapAlloc(). Now (in VS2013), it uses the default process heap, the one returned by the GetProcessHeap().
Which is in itself enough to trigger an AVE when you run your app on Vista or higher, allocating memory from one heap and releasing it from another triggers an AVE at runtime, a debugger break when you debug with the Debug Heap enabled.
This is not where it ends, another significant issue is that the std::string object layout is not the same between the versions. Something you can discover with a little test program:
#include <string>
#include <iostream>
int main()
{
std::cout << sizeof(std::string) << std::endl;
return 0;
}
VS2010 Debug : 32
VS2010 Release : 28
VS2013 Debug : 28
VS2013 Release : 24
I have a vague memory of Stephen Lavavej mentioning the std::string object size reduction, very much presented as a feature, but I can't find it back. The extra 4 bytes in the Debug build is caused by the iterator debugging feature, it can be disabled with _HAS_ITERATOR_DEBUGGING=0 in the Preprocessor Definitions. Not a feature you'd quickly want to throw away but it makes mixing Debug and Release builds of the EXE and its DLLs quite lethal.
Needless to say, the different object sizes seriously bytes when the Config object is created in a DLL built with one version of the standard C++ library and used in another. Many mishaps, the most basic one is that the code will simply read the Config::bee member from the wrong offset. An AVE is (almost) guaranteed. Lots more misery when code allocates the small flavor of the Config object but writes the large flavor of std::string, that randomly corrupts the heap or the stack frame.
Don't mix.
I believe 2013 introduced a lot of changes in the internal data formats of STL containers, as part of a push to reduce memory usage and improve perf. I know vector became smaller, and string is basically a glorified vector<char>.
Microsoft acknowledges the incompatibility:
"To enable new optimizations and debugging checks, the Visual Studio
implementation of the C++ Standard Library intentionally breaks binary
compatibility from one version to the next. Therefore, when the C++
Standard Library is used, object files and static libraries that are
compiled by using different versions can't be mixed in one binary (EXE
or DLL), and C++ Standard Library objects can't be passed between
binaries that are compiled by using different versions."
If you're going to pass std::* objects between executables and/or DLLs, you absolutely must ensure that they are using the same version of the compiler. It would be well-advised to have your client and its DLLs negotiate in some way at startup, comparing any available versions (e.g. compiler version + flags, boost version, directx version, etc.) so that you catch errors like this quickly. Think of it as a cross-module assert.
If you want to confirm that this is the issue, you could pick a few of the data structures you're passing back and forth and check their sizes in the client vs. the DLLs. I suspect your Config class above would register differently in one of the fail cases.
I'd also like to mention that it is probably a bad idea in the first place to use smart containers in DLL calls. Unless you can guarantee that the app and DLL won't try to free or reallocate the internal buffers of the other's containers, you could easily run into heap corruption issues, since the app and DLL each have their own internal C++ heap. I think that behavior is considered undefined at best. Even passing const& arguments could still result in reallocation in rare cases, since const doesn't prevent a compiler from diddling with mutable internals.
You seem to have memory corruption. Microsoft Application Verifier is invaluable in finding corruption. Use it to find your bug:
Install it to your dev machine.
Add your exe to it.
Only select Basics\Heaps.
Press Save. It doesn't matter if you keep application verifier open.
Run your program a few times.
If it crashes, debug it and this time, the crash will point to your problem, not just some random location in your program.
PS: It's a great idea to have Application Verifier enabled at all times for your development project.
I have the problem with passing by reference std::string to function in dll.
This is function call:
CAFC AFCArchive;
std::string sSSS = std::string("data\\gtasa.afc");
AFCER_PRINT_RET(AFCArchive.OpenArchive(sSSS.c_str()));
//AFCER_PRINT_RET(AFCArchive.OpenArchive(sSSS));
//AFCER_PRINT_RET(AFCArchive.OpenArchive("data\\gtasa.afc"));
This is function header:
#define AFCLIBDLL_API __declspec(dllimport)
AFCLIBDLL_API EAFCErrors CAFC::OpenArchive(std::string const &_sFileName);
I try to debug pass-by-step through calling the function and look at _sFileName value inside function.
_sFileName in function sets any value(for example, t4gs..\n\t).
I try to detect any heap corruption, but compiler says, that there is no error.
DLL has been compiled in debug settings. .exe programm compiled in debug too.
What's wrong?? Help..!
P.S. I used Visual Studio 2013. WinApp.
EDIT
I have change header of func to this code:
AFCLIBDLL_API EAFCErrors CAFC::CreateArchive(char const *const _pArchiveName)
{
std::string _sArchiveName(_pArchiveName);
...
I really don't know, how to fix this bug...
About heap: it is allocated in virtual memory of our process, right? In this case, shared virtual memory is common.
The issue has little to do with STL, and everything to do with passing objects across application boundaries.
1) The DLL and the EXE must be compiled with the same project settings. You must do this so that the struct alignment and packing are the same, the members and member functions do not have different behavior, and even more subtle, the low-level implementation of a reference and reference parameters is exactly the same.
2) The DLL and the EXE must use the same runtime heap. To do this, you must use the DLL version of the runtime library.
You would have encountered the same problem if you created a class that does similar things (in terms of memory management) as std::string.
Probably the reason for the memory corruption is that the object in question (std::string in this case) allocates and manages dynamically allocated memory. If the application uses one heap, and the DLL uses another heap, how is that going to work if you instantiated the std::string in say, the DLL, but the application is resizing the string (meaning a memory allocation could occur)?
C++ classes like std::string can be used across module boundaries, but doing so places significant constraints on the modules. Simply put, both modules must use the same instance of the runtime.
So, for instance, if you compile one module with VS2013, then you must do so for the other module. What's more, you must link to the dynamic runtime rather than statically linking the runtime. The latter results in distinct runtime instances in each module.
And it looks like you are exporting member functions. That also requires a common shared runtime. And you should use __declspec(dllexport) on the entire class rather than individual members.
If you control both modules, then it is easy enough to meet these requirements. If you wish to let other parties produce one or other of the modules, then you are imposing a significant constraint on those other parties. If that is a problem, then consider using more portable interop. For example, instead of std::string use const char*.
Now, it's possible that you are already using a single shared instance of the dynamic runtime. In which case the error will be more prosaic. Perhaps the calling conventions do not match. Given the sparse level of detail in your question, it's hard to say anything with certainty.
I encountered similar problem.
I resolved it synchronizing Configuration Properties -> C / C++ settings.
If you want debug mode:
Set _DEBUG definition in Preprocessor Definitions in both projects.
Set /MDd in Code Generation -> Runtime Library in both projects.
If you want release mode:
Remove _DEBUG definition in Preprocessor Definitions in both projects.
Set /MD in Code Generation -> Runtime Library in both projects.
Both projects I mean exe and dll project.
It works for me especially if I don't want to change any settings of dll but only adjust to them.
C++ class constructor can be inlined or not be inlined. However, I found a strange situation where only inline class constructor can avoid Visual Studio memory crash. The example is as follows:
dll.h
class _declspec(dllexport) Image
{
public:
Image();
virtual ~Image();
};
class _declspec(dllexport) Testimage:public Image
{
public:
Testimage();
virtual ~Testimage();
};
typedef std::auto_ptr<Testimage> TestimagePtr;
dll.cpp
#include "dll.h"
#include <assert.h>
Image::~Image()
{
std::cout<<"Image is being deleted."<<std::endl;
}
Image::Image()
{
}
Testimage::Testimage()
{
}
Testimage::~Testimage()
{
std::cout<<"Geoimage is being deleted."<<std::endl;
}
The dll library is compiled as a dynamic library, and it is statically linked to the C++ runtime library (Multi-threaded Debug (/MTd)). The executable program that runs the library is as follows:
int main()
{
TestimagePtr my_img(new Testimage());
return 0;
}
The executable program will invoke the dll library and it also statically links the runtime library. The problem I have is that when running the executable program the following error message appears:
However, when the class constructor in dll is inlined as the following codes show:
class _declspec(dllexport) Image
{
public:
Image();
virtual ~Image();
};
class _declspec(dllexport) Testimage:public Image
{
public:
Testimage()
{
}
virtual ~Testimage();
};
The crash will disappear. Could someone explain the reason behind? Thanks! By the way, I am using VC2010.
EDIT: The following situation also trigger the same crash
.
Situation 1
int main()
{
//TestimagePtr my_img(new Testimage());
Testimage *p_img;
p_img = new Testimage();
delete p_img;
return 0;
}
it is statically linked to the C++ runtime library (Multi-threaded Debug (/MTd)
This is a very problematic scenario in versions of Visual Studio prior to VS2012. The issue is that you have more than one version of the CRT loaded in your process. One used by your EXE, another used by the DLL. This can cause many subtle problems, and not so subtle problems like this crash.
The CRT has global state, stuff like errno and strtok() cannot work properly when that global state is updated by one copy of the CRT and read back by another copy. Relevant to your crash, a hidden global state variable is the heap that the CRT uses to allocate memory from. Functions like malloc() and ::operator new use that heap.
This goes wrong when objects are allocated by one copy of the CRT and released by another. The pointer that's passed to free() or ::operator delete belongs to the wrong heap. What happens next depends on your operating system. A silent memory leak in XP. In Vista and up, you program runs with the debug version of the memory manager enabled. Which triggers a breakpoint when you have a debugger attached to your process to tell you that there's a problem with the pointer. The dialog in your screenshot is the result. It isn't otherwise very clear to me how inlining the constructor could make a difference, the fundamental issue however is that your code invokes undefined behavior. Which has a knack for producing random outcomes.
There are two approaches available to solve this problem. The first one is the simple one, just build both your EXE and your DLL project with the /MD compile option instead. This selects the DLL version of the CRT. It is now shared by both modules and you'll only have a single copy of the CRT in your process. So there is no longer a problem with having one module allocating and another module releasing memory, the same heap is used.
This will work fine to solve your problem but can still become an issue later. A DLL tends to live a life of its own and may some day be used by another EXE that was built with a different version of the CRT. The CRT will now again not be shared since they'll use different versions of the DLL, invoking the exact same failure mode you are seeing today.
The only way to guarantee that this cannot happen is to design your DLL interface carefully. And ensure that there will never be a case where the DLL allocates memory that the client code needs to release. That requires giving up on a lot of C++ goodies. You for example can never write a function that returns a C++ object, like std::string. And you can never allow an exception to cross the module boundary. You are basically down to a C-style interface. Note how COM addresses this problem by using interface-based programming techniques and a class factory plus reference counting to solve the memory management problem.
VS2012 has a counter-measure against this problem, it has a CRT version that allocates from the default process heap. Which solves this particular problem, not otherwise a workaround for the global state issue for other runtime functions. And adds some new problems, a DLL compiled with /MT that gets unloaded that doesn't release all of its allocations now causes an unpluggable leak for example.
This is an ugly problem in C++, the language fundamentally misses an ABI specification that addresses problems like this. The notion of modules is entirely missing from the language specification. Being worked on today but not yet completed. Not simple to do, it is solved in other languages like Java and the .NET languages by specifying a virtual machine, providing a runtime environment where memory management is centralized. Not the kind of runtime environment that excites C++ programmers.
I tried to reproduce your problem in VC2010 and it doesn't crash. It works with a constructor inline or not. Your problem is probably not in what you write here.
Your project is too hard to open as it seams to have its file pathes set in absolute, probably because generated with CMake. (So the files are not found by the compiler).
The problem I see in your code is that you declare the exported classes with _declspec(dllexport) directly written.
You should have a #Define to do this, and the value should be _declspec(dllimport) when read from the exe compilation. Maybe the problem comes from that.
I have a dynamic library of c++ code that is cross platform and mostly just native c++. I then use this dynamic library from my main exe. Up until now all has been good on OSX using gcc. Now I'm on windows I am confused as to what method I should use to enter the dll. I don't have a DllMain function at present as this wasn't required in gcc (to my knowledge). My initial tests worked but on inspection revealed that strangely one of my class constructors was being called on dll load, so I figured I needed to do something more on windows. So do I:
add a DllMain function?
am I safe to just use the noentry compiler option?
When I do either of the above I start getting compiler complaints in the vein of ".CRT section exists there may be unhandled static initializers or terminators"
I have read up on this using this article, but any advice and clarity on the best way forward would be greatly appreciated. Its all a bit blurry in my head as to what I need to do.
Based on the .CRT error, you definitely need a DllMain function. For most Windows compilers, a DllMain will be provided for you automatically, so that you don't need to write one yourself. Based on other parts of your question it seems most likely that you are using Visual C++, whose CRT does provide a DllMain for you. So while you do need a DllMain, you don't need to write the code for it.
The default VC CRT DllMain is used to initialize the CRT for the DLL in question, and to initialise all the static/global variables that the dll provides.
The model for DLLs on Unix and Windows is significantly different, and you should think of each DLL as having a more 'private' set of state. Although, if all Dlls opt into the same version of the CRT dll, some of that state will then be shared.
Because the CRT is providing a DllMain for you, you should not throw /noentry on the linker.
The .CRT section exists error (which you must have seen by throwing /noentry) is telling you that you need a DllMain because you've got one or more objects in your DLL that require static initialisation.
Martyn
If it is just a library, then NOENTRY should suffice. DllMain is there to control events that happen with the DLL (i.e. attach, detach etc).
You can change the code (slightly) to avoid all entry points but main. Essentially, if you have any variable defined outside of the functions (globally but not statically linked), wrap them in a function call. Use the often-forgotten static function variables.
Ie, change global declaration
SomeType var_name;
to this:
SomeType & var_name(){static SomeType var; return var;}
Similarly, you can change static class instance variables by changing this:
struct Container{
Container();
static Container instance;
};
Container Container::instance;
to this:
struct Container{
Container();
static Container & instance();
};
Container & Container::instance(){
static Container var;
return var;
}
This is essentially a singleton, but there might be some concurrency issues if the 1st time you access the instance will be from a multi-threaded environment. In fact, the thing to keep in mind is that unlike globally-defined variables, static locally-defined variables will be initialized the first time the function is called.