LoadLibrary() : How to handle invalid DLLs? - c++

Question

I have an application that relies heavily on plugins.
On startup it scans a directory for DLLs and loads them one by one, looking for ones that implement a certain exported function. However - if someone were to rename a different type of file to *.dll and put it in the directory, that file would then also be loaded by LoadLibrary(). LoadLibrary() doesn't like that and produces an error [dialog].
Is there a way to simply ignore invalid / incompatible .dll files (either detecting them prior to the call or have LoadLibrary() return NULL rather than throwing a fit)?

Answer 1

You need to set the error mode for your process. Do this once and for all at startup:
UINT oldMode = SetErrorMode(0);
SetErrorMode(oldMode | SEM_FAILCRITICALERRORS | SEM_NOOPENFILEERRORBOX);
After you've set the process error mode, when LoadLibrary fails no dialog box will be displayed and LoadLibrary will return NULL.
The documentation says:
Best practice is that all applications call the process-wide SetErrorMode function with a parameter of SEM_FAILCRITICALERRORS at startup. This is to prevent error mode dialogs from hanging the application.
I also recommend adding SEM_NOOPENFILEERRORBOX for reasons that I guess should be obvious.

Answer 2

http://msdn.microsoft.com/en-us/library/windows/desktop/ms684175%28v=vs.85%29.aspx
Remarks
To enable or disable error messages displayed by the loader during DLL loads, use the SetErrorMode function.
Don't forget to restore the error mode after you are finished.
If you wanted to do this yourself, in a primitive manner you could parse the PE header to identify obvious cases of bad DLL files, however it is not really possible to determine if a DLL is valid and loadable without a full PE loader which requires a lot of work and is already built into the operating system.

Answer 3

Updated answer after feedback in comments from djgandy and Remy Lebeau. Now a complete
function, better preservation of error mode at entry:
// This function will load the DLL named by pszPath if it is a valid library.
// If the function succeeds, it will return a valid HMODULE for the DLL. This
// handle should be passed to FreeLibrary when it is no longer needed.
// If the function fails, it will return NULL and no annoying dialog boxes will
// be displayed. It is therefore up to the caller to notify the user about what
// happened or take any other appropriate action. The reason for failure can
// be obtained from GetLastError(). Common problems:
// ERROR_BAD_EXE_FORMAT - Bad DLL (tested function with text file)
// ERROR_MOD_NOT_FOUND - Missing DLL (tested with file that did not exist)
//
// Module-loading functions can return several other errors, look at winerror.h
// list starting at ERROR_INVALID_MODULETYPE
//
// Obviously, since it's just a wrapper around LoadLibrary this function is not
// safe to call from DllMain.
//
// NB: GetErrorMode() is only available on Vista / Server 2003 or later.
HMODULE LoadLibraryIfValid(LPCTSTR pszPath)
{
HMODULE hModule = NULL;
UINT prevErrorMode = GetErrorMode();
SetErrorMode(prevErrorMode | SEM_FAILCRITICALERRORS);
hModule = LoadLibrary(pszPath);
SetErrorMode(prevErrorMode);
return hModule;
}
If targeting Windows 7 / Server 2008 R2 or later, the Get/SetThreadErrorMode()
functions are available, but might not be worth it or even a good alternative
(discussion in comments, below)
If anyone cared enough to put the time into it (I sure don't), a version of
this function could easily be written using GetModuleHandle for kernel32 and
GetProcAddress to be compatible with earlier versions of Windows as well as
provide a global/per-thread error mode option for platforms that support it
(truly pointless because it's only changed for the duration of one call anyway).
This is the largest commentary-to-code ratio in my life.