C++ WriteProcessMemory error INVALID_HANDLE_VALUE - c++

I'm using to the "CreateRemoteThread & WriteProcessMemory" Technique to inject my dll into another process. My code work fine on windows 7,8, but WriteProcessMemory function always return FALSE (GetLastError = 6 - INVALID_HANDLE_VALUE) when run on windows XP (VirtualBox machine). Can't u help me?
Here is the main code:
BOOL CHookDLL::DoHook(const DWORD dwProcessId, const CHAR* szDLLHookName)
{
CHAR szDllHookPath[1024] = "";
HANDLE hRemoteThread = NULL;
HMODULE hLib = 0;
LPVOID RemoteString = NULL;
LPVOID LoadLibAddy = NULL;
if (dwProcessId == NULL){
__OutputDebug("CHookDLL::DoHook\tpProcessId NULL");
return FALSE;
}
::GetFullPathNameA(szDLLHookName, MAX_PATH, szDllHookPath, NULL);
if (::PathFileExists((CString)szDllHookPath) == FALSE){
__OutputDebug("CHookDLL::DoHook\tPathFileExists FALSE");
return FALSE;
}
// enable SeDebugPrivilege
if (!SetPrivilege(m_hTokenSetPrivilege, SE_DEBUG_NAME, TRUE))
{
__OutputDebug("CHookDLL::DoHook\tSetPrivilege FAILED");
// close token handle
CloseHandle(m_hTokenSetPrivilege);
return FALSE;
}
m_hProcess = ::OpenProcess(PROCESS_ALL_ACCESS, FALSE, dwProcessId);
if (m_hProcess == NULL){
__OutputDebug("CHookDLL::DoHook\tOpenProcess FALSE: %d", GetLastError());
return FALSE;
}
LoadLibAddy = (LPVOID)::GetProcAddress(::GetModuleHandleA("kernel32.dll"), "LoadLibraryA");
if (LoadLibAddy == NULL){
__OutputDebug("CHookDLL::DoHook\tGetProcAddress NULL");
return FALSE;
}
// Allocate space in the process for our DLL
RemoteString = (LPVOID)VirtualAllocEx(m_hProcess, NULL, strlen(szDllHookPath) + 1,
MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
if (RemoteString == NULL){
__OutputDebug("CHookDLL::DoHook\tVirtualAllocEx NULL");
return FALSE;
}
// this line is return FALSE
if (WriteProcessMemory(m_hProcess, RemoteString, szDllHookPath, strlen(szDllHookPath) + 1, NULL) == FALSE)
{
__OutputDebug("CHookDLL::DoHook\tWriteProcessMemory FALSE: %d", GetLastError());
return FALSE;
}
hRemoteThread = ::CreateRemoteThread(m_hProcess, NULL, NULL,
(LPTHREAD_START_ROUTINE)LoadLibAddy,
(LPVOID)RemoteString, NULL, NULL);
::WaitForSingleObject(hRemoteThread, INFINITE);
// Get handle of the loaded module
::GetExitCodeThread(hRemoteThread, &m_hLibModule);
if (m_hLibModule == NULL){
__OutputDebug("CHookDLL::DoHook\tCreateRemoteThread NULL");
return FALSE;
}
// Clean up
::CloseHandle(hRemoteThread);
::VirtualFreeEx(m_hProcess, RemoteString,
strlen(szDllHookPath) + 1, MEM_RELEASE);
__OutputDebug("Hook OK");
return TRUE;
}
// Common function Output Debug String
static INT __OutputDebug(const CHAR* format, ...)
{
#ifndef DEBUG
return -1;
#endif // DEBUG
if (format[0] == 0) return -1;
CHAR szDebug[1024] = "";
va_list arglist;
va_start(arglist, format);
vsprintf_s(szDebug,format, arglist);
va_end(arglist);
strcat_s(szDebug, "\n");
OutputDebugStringA(szDebug);
return 1;
}

The problem lies in your OpenProcess call. From here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms684880(v=vs.85).aspx, listed under PROCESS_ALL_ACCESS access right:
Windows Server 2003 and Windows XP: The size of the PROCESS_ALL_ACCESS flag increased on Windows Server 2008 and Windows Vista. If an application compiled for Windows Server 2008 and Windows Vista is run on Windows Server 2003 or Windows XP, the PROCESS_ALL_ACCESS flag is too large and the function specifying this flag fails with ERROR_ACCESS_DENIED. To avoid this problem, specify the minimum set of access rights required for the operation. If PROCESS_ALL_ACCESS must be used, set _WIN32_WINNT to the minimum operating system targeted by your application (for example, #define _WIN32_WINNT _WIN32_WINNT_WINXP). For more information, see Using the Windows Headers.
Therefore, it may be that PROCESS_VM_READ and PROCESS_VM_OPERATION aren't getting set, hence the invalid handle error later on. I know that that OpenProcess should really be returning an error code if it fails - and it's not - but if this flag is genuinely overflowing, I can see how a silent failure might occur.

Related

C++ | Windows - Is there a way to find out which process has ownership of the locked file?

What I want to know is it possible to try an open a file (and when it fails because it's opened with another process with sharing off) to figure out which process is using said file?
The reason I am wanting to know this information is because I am making a little application that will "fix" malicious files.
For example, some malicious/adware etc set the file security descriptor so the user can't delete the file, etc. My application just resets the security descriptor allowing the user to regain control.
I have also seen a file open up its child process with for example (CreateFile) and have Shared Mode turned off so the file can't be touched, then the application would execute the childprocess from memory.
Yes, you can in general just use the openfiles command, after having enabled collection of this information via, it appears, openfiles /local on.
In Windows NT up to and including (it seems) Windows XP there was a similar Resource Kit command named oh, short for open handles.
An alternative to both is to use SysInternal's Process Explorer.
Note: In some cases openfiles will fail to list some handle. This happens for me when Windows refuses to unmount an USB disk, claiming that some process is using a file on that disk. No such process ever shows up.
I have developed a function to locate such process, kill it and delete the locked file.
bool ForceDeleteFile(LPWSTR FileName);
Here is the full source code:
bool KillFileProcess(LPWSTR FileName)
{
HANDLE hProcessSnap;
HANDLE hProcess;
PROCESSENTRY32 pe32;
DWORD dwPriorityClass;
bool result = false;
// Take a snapshot of all processes in the system.
hProcessSnap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
if (hProcessSnap == INVALID_HANDLE_VALUE)
{
//printError(TEXT("CreateToolhelp32Snapshot (of processes)"));
return(FALSE);
}
// Set the size of the structure before using it.
pe32.dwSize = sizeof(PROCESSENTRY32);
// Retrieve information about the first process,
// and exit if unsuccessful
if (!Process32First(hProcessSnap, &pe32))
{
//printError(TEXT("Process32First")); // show cause of failure
CloseHandle(hProcessSnap); // clean the snapshot object
return(FALSE);
}
// Now walk the snapshot of processes, and
// display information about each process in turn
do
{
// Retrieve the priority class.
dwPriorityClass = 0;
hProcess = OpenProcess(PROCESS_ALL_ACCESS, FALSE, pe32.th32ProcessID);
if (hProcess == NULL)
{
//printError(TEXT("OpenProcess"));
}
else
{
dwPriorityClass = GetPriorityClass(hProcess);
if (!dwPriorityClass)
{
//printError(TEXT("GetPriorityClass"));
}
CloseHandle(hProcess);
if (HANDLE hProcess = ::OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, pe32.th32ProcessID))
{
WCHAR filename[MAX_PATH] = {};
if (GetModuleFileNameEx(hProcess, NULL, filename, MAX_PATH))
{
if (_wcsicmp((const wchar_t *)FileName, (const wchar_t *)filename) == NULL)
{
if (TerminateProcess(pe32.th32ProcessID, 0))
{
_tprintf(L"Found: Process full killed\nKILLED!\n");
result = true;
}
else
{
_tprintf(L"Found: Process full \nFailed to terminate\n");
DoRun(((CString)L"taskkill /F /IM " + (CString)pe32.szExeFile).GetBuffer());
result = false;
}
}
}
else
{
// handle error
}
CloseHandle(hProcess);
}
}
} while (Process32Next(hProcessSnap, &pe32));
CloseHandle(hProcessSnap);
return(result);
}
bool ForceDeleteFile(LPWSTR FileName)
{
bool result = DeleteFile(FileName);
if (!result)
{
_tprintf(L"Can't delete file. using DeleteFile(). Trying to locate process and kill it\n");
result = KillFileProcess(FileName);
if (!result)
_tprintf(L"Couldn't find the process\n");
else
{
Sleep(1000);
result = DeleteFile(FileName);
if (result)
_tprintf(L"DeleteFile success");
else
_tprintf(L"DeleteFile ============== failed ===============");
}
}
return result;
}
BOOL TerminateProcess(DWORD dwProcessId, UINT uExitCode)
{
DWORD dwDesiredAccess = PROCESS_TERMINATE;
BOOL bInheritHandle = FALSE;
HANDLE hProcess = OpenProcess(dwDesiredAccess, bInheritHandle, dwProcessId);
if (hProcess == NULL)
return FALSE;
BOOL result = TerminateProcess(hProcess, uExitCode);
CloseHandle(hProcess);
return result;
}

OpenProcess Failure (SeDebugPrivilege?)

I'm trying to use OpenProcess, for some reason it keeps failing.
HANDLE GetProcessPid()
{
DWORD pid = 0;
wchar_t ProcessName[] = L"notepad.exe";
// Create toolhelp snapshot.
HANDLE snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
PROCESSENTRY32 process;
ZeroMemory(&process, sizeof(process));
process.dwSize = sizeof(process);
// Walkthrough all processes.
if (Process32First(snapshot, &process))
{
do
{
if (wcscmp(process.szExeFile, ProcessName) == 0)
{
pid = process.th32ProcessID;
break;
}
} while (Process32Next(snapshot, &process));
}
CloseHandle(snapshot);
if (pid != 0)
{
return OpenProcess(PROCESS_ALL_ACCESS, FALSE, pid);
}
return NULL;
}
This always returns 0xcccccccccccccccc or 0x0000000000000020 even when I've set the pid manually when calling OpenProcess.
I've tried running as admin outside of debugging inside visual studio and get the same results, after searching, I think I need to enable debug priviledges (SeDebugPrivilege), how do I do this in visual studio 2013?
Your code is fine, you just need to run your executable as administrator and it works, I just tested it.

ReadProcess Memory 299 on Windows 8

I have this program that works perfectly in windows 7 but on windows 8 the readprocessmemory seems to be blank when I output it.Get Last error code 299. I Did not create this part of of program for read process but I use it because it was working for windows 7. The game handles and aria location are same on windows 8 machine, I double checked them. and The game handle is found. The address works fine in windows 7.
hGameWindow = FindWindow(L"WFElementClient Window",NULL);
if(hGameWindow) {
GetWindowThreadProcessId( hGameWindow, &dwProcId );
if( dwProcId != 0 ) {
hProcHandle = OpenProcess( PROCESS_ALL_ACCESS, FALSE, dwProcId );
if( hProcHandle == INVALID_HANDLE_VALUE || hProcHandle == NULL ) {
GameStatus = "Failed to open process for valid handle";
}else{
GameStatus = "Game Found";
myaddr = FindPointerAddr(hProcHandle, ariaBase, aOffset);
// IsGameAvail = true;
}
}
else GameStatus = "Failed to obtain process id";
}
else GameStatus = "game handle not found";
ReadProcessMemory(hProcHandle, (LPCVOID)myaddr, &buffer, sizeof(buffer), NULL);
int FindPointerAddr(HANDLE pHandle,int baseaddr, DWORD offsets[])
{
int Address = baseaddr;
int offset = 0;
int offsetCount = 5;
for (int i = 0; i < offsetCount; i++)
{
ReadProcessMemory(pHandle, (LPCVOID)Address, &Address , 4, NULL);
Address+=offsets[i];
}
return Address;
}
Security permissions have changed from Windows 7 to Windows 8.
You may need to run as administrator and set SeDebugPrivelage now, when you were not required to on previous versions of Windows. Such as when calling OpenProcess() with PROCESS_ALL_ACCESS because PROCESS_VM_READ requires SeDebugPrivelage
Here is how you set SeDebugPrivelage:
bool SetDebugPrivilege(bool Enable)
{
HANDLE hToken{ nullptr };
if (!OpenProcessToken(GetCurrentProcess(), TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))
return false;
TOKEN_PRIVILEGES TokenPrivileges{};
TokenPrivileges.PrivilegeCount = 1;
TokenPrivileges.Privileges[0].Attributes = Enable ? SE_PRIVILEGE_ENABLED : 0;
if (!LookupPrivilegeValueA(nullptr, "SeDebugPrivilege", &TokenPrivileges.Privileges[0].Luid))
{
CloseHandle(hToken);
return false;
}
if (!AdjustTokenPrivileges(hToken, FALSE, &TokenPrivileges, sizeof(TOKEN_PRIVILEGES), nullptr, nullptr))
{
CloseHandle(hToken);
return false;
}
CloseHandle(hToken);
return true;
}

Failing dll injection

I'm in the process of making a security program for my network. One of it's instances is to check and monitor what api's and libraries are called. The dll to do that and the program that go along with it are already finished. But there is a problem that I cant seem to fix.
When trying to inject my dll into system processes (such as explorer.exe, my main test system process) with NtCreateThreadEx I get the return value: C0000022, it means something along the lines of: Status_Access_Denied (it returns in NTSTATUS, but DWORD will do)
I have no idea what to do, I'm running as Administrator, I raised my privileges, and used the proper functions, still I get c0000022
Here's the code I'm using to inject
#include "main.h"
typedef DWORD NTSTATUS;
struct NtCreateThreadExBuffer{
ULONG Size;
ULONG Unknown1;
ULONG Unknown2;
PULONG Unknown3;
ULONG Unknown4;
ULONG Unknown5;
ULONG Unknown6;
PULONG Unknown7;
ULONG Unknown8;
};
typedef NTSTATUS (WINAPI *LPFUN_NtCreateThreadEx)
(
OUT PHANDLE hThread,
IN ACCESS_MASK DesiredAccess,
IN LPVOID ObjectAttributes,
IN HANDLE ProcessHandle,
IN LPTHREAD_START_ROUTINE lpStartAddress,
IN LPVOID lpParameter,
IN BOOL CreateSuspended,
IN ULONG StackZeroBits,
IN ULONG SizeOfStackCommit,
IN ULONG SizeOfStackReserve,
OUT LPVOID lpBytesBuffer
);
using namespace std;
//#define CREATE_THREAD_ACCESS (PROCESS_CREATE_THREAD | PROCESS_QUERY_INFORMATION | PROCESS_VM_OPERATION | PROCESS_VM_WRITE | PROCESS_VM_READ)
#define CREATE_THREAD_ACCESS ( PROCESS_ALL_ACCESS )
BOOL LoadDll(char *procName, char *dllName);
BOOL InjectDLL(DWORD dwProcessID, char *dllName);
BOOL LoadDll(char *dllName, DWORD dwProcID){
printf("Process Id to Inject: %d",dwProcID);
if(!dwProcID){
printf("No vailid PID\n");
return false;
}
FILE* FileCheck = fopen(dllName, "r");
if(FileCheck==NULL){
printf("\nUnable to inject %s", dllName);
return false;
}
fclose(FileCheck);
if(!InjectDLL(dwProcID, dllName)){
printf("injection failed\n");
return false;
} else {
return true;
}
}
BOOL InjectDLL(DWORD dwProcessID, char *dllName){
HANDLE hProc;
HANDLE hToken;
char buf[50]={0};
LPVOID RemoteString, LoadLibAddy;
if(!dwProcessID)return false;
HANDLE hCurrentProc = GetCurrentProcess();
if (!OpenProcessToken(hCurrentProc,TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES,&hToken)){
printf("OpenProcessToken Error:%d\n", GetLastError());
} else {
if (!RaisePrivleges(hToken, (char*)SE_DEBUG_NAME)){
printf("SetPrivleges SE_DEBUG_NAME Error:%d\n", GetLastError());
}
}
if (hToken)CloseHandle(hToken);
hProc = OpenProcess(CREATE_THREAD_ACCESS, FALSE, dwProcessID);
printf("\nHandle to process: %x\n", hProc);
if(!hProc){
printf("OpenProcess() failed: %d", GetLastError());
return false;
}
LoadLibAddy = (LPVOID)GetProcAddress(GetModuleHandle("kernel32.dll"), "LoadLibraryA");
if(!LoadLibAddy){
printf("GetProcAddress() failed: %d", GetLastError());
return false;
}
RemoteString = (LPVOID)VirtualAllocEx(hProc, NULL, strlen(dllName), MEM_RESERVE|MEM_COMMIT, PAGE_READWRITE);
if(RemoteString == NULL){
printf("VirtualAllocEx() failed: %d", GetLastError());
return false;
}
printf("\nRemote address: %x\n", RemoteString);
if(WriteProcessMemory(hProc, (LPVOID)RemoteString, dllName, strlen(dllName), NULL) == NULL){
printf("WriteProcessMemory() failed: %d", GetLastError());
return false;
}
/*
if(!CreateRemoteThread(hProc, NULL, NULL, (LPTHREAD_START_ROUTINE)LoadLibAddy, (LPVOID)RemoteString, NULL, NULL)){
printf("CreateRemoteThread() failed: %d", GetLastError());
return false;
}
*/
HMODULE modNtDll = GetModuleHandle("ntdll.dll");
if( !modNtDll )
{
printf("n failed to get module handle for ntdll.dll, Error=0x%.8x", GetLastError());
return 0;
}
LPFUN_NtCreateThreadEx funNtCreateThreadEx =
(LPFUN_NtCreateThreadEx) GetProcAddress(modNtDll, "NtCreateThreadEx");
if( !funNtCreateThreadEx )
{
printf("n failed to get function (NTCreateThreadEx) address from ntdll.dll, Error=0x%.8x\nTrying CreateRemoteThread api\n", GetLastError());
if(!CreateRemoteThread(hProc, NULL, NULL, (LPTHREAD_START_ROUTINE)LoadLibAddy, (LPVOID)RemoteString, NULL, NULL)){
printf("CreateRemoteThread() failed: %d", GetLastError());
return false;
} else {
printf("CreateRemoteThread success!\n");
return true;
}
return 0;
}
NtCreateThreadExBuffer ntbuffer;
memset (&ntbuffer,0,sizeof(NtCreateThreadExBuffer));
DWORD temp1 = 0;
DWORD temp2 = 0;
HANDLE pRemoteThread = NULL;
ntbuffer.Size = sizeof(NtCreateThreadExBuffer);
ntbuffer.Unknown1 = 0x10003;
ntbuffer.Unknown2 = 0x8;
ntbuffer.Unknown3 = &temp2;
ntbuffer.Unknown4 = 0;
ntbuffer.Unknown5 = 0x10004;
ntbuffer.Unknown6 = 4;
ntbuffer.Unknown7 = &temp1;
ntbuffer.Unknown8 = 0;
NTSTATUS status = funNtCreateThreadEx(
&pRemoteThread,
0x1FFFFF,
NULL,
hProc,
(LPTHREAD_START_ROUTINE) LoadLibAddy,
(LPVOID)RemoteString,
FALSE, //start instantly
NULL,
NULL,
NULL,
&ntbuffer
);
printf("NTCreateThreadEx return: %x\n", status);
// Resume the thread execution
WaitForSingleObject(pRemoteThread, INFINITE);
//Check the return code from remote thread function
DWORD dwExitCode;
if( GetExitCodeThread(pRemoteThread, (DWORD*) &dwExitCode) )
{
printf("\n Remote thread returned with status = %d\n", dwExitCode);
}
CloseHandle(pRemoteThread);
CloseHandle(hProc);
return true;
}
BOOL RaisePrivleges( HANDLE hToken, char *pPriv ){
TOKEN_PRIVILEGES tkp;
tkp.PrivilegeCount = 1;
tkp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
tkp.Privileges[0].Luid.HighPart = 0;
tkp.Privileges[0].Luid.LowPart = 0;
if (!LookupPrivilegeValue(NULL, pPriv, &tkp.Privileges[0].Luid)){
printf("LookupPrivilegeValue Error:%d\n", GetLastError());
return FALSE;
}
int iRet = AdjustTokenPrivileges(hToken, FALSE, &tkp, 0x10, (PTOKEN_PRIVILEGES)NULL, 0);
if (iRet == NULL){
printf( "AdjustTokenPrivileges Error:%d\n", GetLastError());
return TRUE;
} else {
iRet = GetLastError();
switch (iRet){
case ERROR_NOT_ALL_ASSIGNED:
printf("AdjustTokenPrivileges ERROR_NOT_ALL_ASSIGNED\n" );
return FALSE;
case ERROR_SUCCESS:
return TRUE;
default:
printf("AdjustTokenPrivileges Unknow Error:%d\n", iRet);
return FALSE;
}
}
}
1) If you're running on VISTA or later then you're possibly trying to inject into a 'protected process' from a 'non protected process'. See Process Security and Access Rights in MSDN. Non protected processes can't create threads in protected processes; though I must admit I'd expect the call to open process to fail when you request the inappropriate access rights rather than the subsequent create thread call to fail.
2) Why are you using NtCreateThreadEx() rather than simply calling CreateRemoteThread()?
3) This probably isn't the cause of your problem, but... You're failing to allocate memory for the null terminator in the string, you should be allocating strlen(dllName) + 1.
4) I assume that the process that is doing the injecting and the process that you're injecting into are both the same architecture, you're not running an x86 exe on x64 and expecting to inject into an x64 exe?
Since it's hard to find the right answer to this problem, I am posting even though the thread is old.
I was trying to inject into x64 service on Win7 x64 and kept running into same problems. My solution was:
Compile both the injector and injection dll as x64.
Instead of CreateRemoteThread & NtCreateThreadEx (both failing) use RtlCreateUserThread.
You must specify the full path to the injected DLL, otherwise it will not be found.

Get file type from windows registry in c++

I am trying to display file type of given filename (based on extension) using AssocQueryKey() API function.
The problem is thar return wrong HKEY value sometimes. For example the following function works correctly on win 7 ultimate x64, but fails for some extensions like ".mp3" on my win xp x86 machine (other extensions works though).
Even when "succeeded" and returns S_OK, GetLastError() is 1008 ALWAYS after AssocQueryKey() call:
// Return STL string representation of file type from windows registry
stlstring GetFileTypeFromRegistry(const stlstring& m_filename)
{
CRegKey reg;
HKEY key = {0};
stlstring s;
//Get file extension
LPCTSTR fExt = PathFindExtension(m_filename.c_str());
if(AssocQueryKey(NULL, ASSOCKEY_CLASS, fExt, TEXT(""), &key) != S_OK)
DisplayError(_T("AssocQueryKey != S_OK"), GetLastError());
else
DisplayError(_T("AssocQueryKey == S_OK"), GetLastError());
if(reg.Open ( key, NULL, KEY_QUERY_VALUE) != ERROR_SUCCESS){
reg.Close();
DisplayError((LPTSTR)fExt);
return s;
}
//DWORD out = 0;
/*WCHAR *h = new WCHAR[1024];
ZeroMemory(h, sizeof(h));
AssocQueryStringByKey(0, ASSOCSTR_EXECUTABLE, HKEY_CLASSES_ROOT, NULL, h, &out);
//MessageBox(0,_T("gbtbb"),h,MB_OK);
delete[] h;*/
ULONG m_sz = 256;
//if( reg.QueryStringValue(NULL, NULL, &m_sz) == ERROR_SUCCESS){
TCHAR *m_regstring = new TCHAR[m_sz + 1];
if(reg.QueryStringValue(NULL, m_regstring, &m_sz) == ERROR_SUCCESS){
//DisplayError(_T(""));
s += m_regstring;
/*delete[] m_regstring; m_regstring = NULL;
reg.Close();
return s;*/
} else {
DisplayError(_T("CRegKey::QueryStringValue()"), GetLastError());
}
s += m_regstring;
delete[] m_regstring; m_regstring = NULL;
reg.Close();
return s;
/*}
reg.Close();
return s;*/
}
Any ideas on this ?? This function is from a DLL which is loaded by windows explorer, implementing IQueryInfo::GetInfoTip() if that matters.
You shouldn't use GetLastError for functions that return the error code directly. The MSDN page for AssocQueryKey says "Returns S_OK if successful, or a COM error value otherwise.", which means you already get the error code in the return value.
If you just want to get the file type information, there's a much simpler solution: SHGetFileInfo. It's really simple to use, like this:
SHFILEINFO shfi;
SHGetFileInfo(filename, 0, &shfi, sizeof(shfi), SHGFI_TYPENAME | SHGFI_USEFILEATTRIBUTES);
// shfi.szTypeName now contains the file type string of the given filename