Debugging shell extension in Windows 7 - c++

I'm trying to debug shell extension (IContextMenu) in Windows 7 with Visual C++ 2008. I have set DesktopProcess=1 in the registry and set host app to explorer.exe. But when I start the debugger, it launches explorer.exe and then detaches from the process. DllMain of the shell extension isn't called.
The same code with exactly the same settings launched in debugger without any problems in Windows XP + Visual C++ 2008.
Any thoughts how to debug the shell extension in Win7?

I've found a nice workflow that I think is the fastest way to rapidly iterate the code-build-test cycle when developing shell extensions. The following should work on any Windows version.
First prepare - set the start program of your shell extension project to be c:\windows\explorer.exe and also set it to be the start-up project.
Then, whenever you want to debug your shell extension perform the following steps:
Click on the task bar and press Alt-F4 - this will bring up the shut down dialog
Press Ctrl-Alt-Shift-Escape - this combination will close explorer.
Use Alt-Tab to go back to Visual Studio and press F5 - explorer will now launch with the VS debugger attached to it from the very beginning.
When done, just stop the debugger session. This will kill the debugged instance of explorer and will also automatically start a normal instance of it. This will also unlock the shell extension DLL so that you can build it again.
Caveat on Vista and 7 - be sure to run the Visual Studio that you use for debugging in non-Administrator mode (non-elevated), so that the explorer is started in its usual non-elevated mode.

Try launching explorer and THEN attaching the debugger to it.

You could try putting a DebugBreak() call in your code. This should launch the just-in-time debugger at the call and give you an idea of what is going on.

You should take a look at gflags.exe, part of the standard debugging tools sdk. It's got all the options you need to configure (global)flags for any process startup/services/heap/pool-tagging/stacktrace's-on-allocation etc...

Debugging Explorer.exe is usually overkill for extensions that operate in a shell view.
I personally use a little app I made that hosts an instance of IExplorerBrowser similar to this example. If your IContextMenu item is not the default item then you can just use Notepad.exe and its open file dialog...

Related

When are memory dump files exactly created?

I have configured my windows 7 to create mini dump files on crashes but when my application crashed, no dump file was created. The search for answer left me rather confused as to when are dump files created, when windows crashes or my application crashes?
In my case, I am looking for dump file when my application crashes. I receive a typical crash dialog that states:
TheApp Application has stopped working
Windows can check online for a solution to the problem
-> Check online for a solution and close the program
-> Close the program
-> Debug the program
So can I generate dump file for my application when it crashes? I can't produce this bug on development machine so I want to walk back from dump file. Is there any other option to trace the source of bug (to source code)?
First of all, there are different places to configure a "create a minidump on crash" setting, which are totally different.
You can configure Windows to create a kernel dump file when Windows crashes, i.e. when a Bluescreen of death (BSOD) occurs. This is done in the following screen on Windows 7:
You can configure Windows to create a user mode dump file when an application crashes, i.e. instead of the "Windows Error Reporting" dialog which would normally appear. To do so, and you know that in advance, then configure a Registry key called LocalDumps (MSDN). By default, dumps will be created below %LOCALAPPDATA%\CrashDumps and they will have the naming scheme app.exe.<PID>.dmp.
For the sake of completeness, there might be other triggers. The only sure way to tell is: when the method MiniDumpWriteDump (MSDN) is called.
I'm quite sure that you want option 2 of the above. If you have trouble with it, see whether all the conditions for LocalDump are fulfilled.
The answer given by #antlersoft does not work, for the reasons I have posted in my blog: at the time the dialog is shown, Windows has triggered a breakpoint to stop the application and it has injected a callstack of Windows Error Reporting. All in all, not a good starting point for debugging.
What would work is:
attach a debugger of your choice
press "Go" in the debugger
press the "Debug" button of the WER dialog
confirm the warning about the debugger which is already attached
click "No" when asked to start debugging using the selected debugger
Using Task Manager to create a crash dump is not recommended, since it will not consider the bitness of the application, which may cause trouble later. See ways to create good and useful crash dumps.
Minidump is created when Windows crashes. It's not intended to application crash.
If you want to debug crashes of your application, you may attach it to a debugger after it is started. Clicking on the "Debug" button when application crashes do the same. You can use the debugger of MS Visual Studio to do that, for example.
See this page for help on attaching a process to MS Visual Studio debugger:
https://msdn.microsoft.com/en-us/library/3s68z0b3.aspx
EDIT: following text removed, as this may not work as expected (comment from Thomas)
You can also create a dump file from task manager, however you will still need a debugger to analyze it and, actually I am not sure you will be able to get the dump file at the point application crashes. The best way, if you can, is to debug the process on the target machine by attaching it to debugger either after it is started or when crash occurs.
When you get the crash dialog, go to Task Manager, find the process, right click on the process, and select "Create dump file". The dump file is created in the AppData/Local/Temp folder for the user; it will be named %AppData%\Local\Temp\.DMP; if you create multiple it will be -1.DMP, etc. You can move the dump file to your development machine and open it within Visual Studio. Visual Studio will then act as if you had hit "Break all" at the point of the crash while running the process in the debugger.

Can the visual studio debugger still be used to debug ppapi trusted plugins when using the --ppapi-out-of-process flag?

I am using the new MessageLoop class introduced in pepper-25 so I can run a background thread with blocking ppapi calls for File IO. Up until now I have been running chrome from visual studio with the flags --single-process and --register-pepper-plugins so I can debug my plugin from within visual studio.
Using these flags I found that the call to PPB_GetInterface get_browser for PPB_MESSAGING_INTERFACE was returning null and after some searching I found this issue which states you must run with the flag --ppapi-out-of-process to get MessageLoop support.
With that flag added get_browser does return a valid interface pointer but I can no longer debug my plugin with the visual studio ide as it cannot attach to the child process that my plugin is run from. Is there anyway to tell it to attach to my plug-in process or a way of running from a single process with support for MessageLoop?
Thanks,
James
To get Visual Studio to attach to child processes automatically, you can use the workarounds described at Can Visual Studio be made to debug child processes like WinDBG?.
A different approach is to use the flags --no-sandbox and --ppapi-startup-dialog when you start Chrome. This will cause a message box to appear with the process id when a plugin process is created. You can then connect the Visual Studio debugger to that process using Debug -> Attach to process... -> pick process id. Of course, you could have attached to the process without the Chrome flags and it's not automatic, so this really just gives you certainty that you're attaching to the right process if you have many running at the same time.
A third approach is to use the free Microsoft-provided debugger WinDbg. It is complex and much less user-friendly than the built-in debugger in Visual Studio, but it does have the ability to attach to child processes automatically. Download is available from http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx.
As for MessageLoop it will not be supported for in-process plugins. As the issue you link to mentions, developers should move to out-of-process, even if it adds the inconvenience of not being able to attach the Visual Studio debugger automatically without workarounds.

Is there a known bug in Visual Studio 2008 for using drop target in C++ when running Visual Studio as administrator?

We have an application that allows a user to drag a picture to a window and that window is a drop target (using OLE). When this code is run in the debugger and the Visual Studio instance is run as administrator (right click from desktop) then the target drop is not allowed.
If the exe is run on its own it works fine.
If the visual studio instance is run NOT as an admin the functionality works fine in the debugger. (Same solution/project files/etc)
Win7 OS. Visual Studio 2008. Unmanaged C++
I find it very odd. Not sure why it is happening. In fact I would have guessed the OPPOSITE regarding running VS as an admin.
Has anyone seen this or does anyone have links to workarounds or explanations?
This is most likely happening due to UIPI (User Interface Privilage Isolation).
In the case where you've launched your processes as Admin (due to the parent process (Visual Studio) being run as admin), UIPI isn't going to let non-elevated (admin) processes send any messages to your app. Drag and Drop between applications is implemented using Windows messages.
To work around this, you can use the ChangeWindowMessageFilterEx() API to opt into the appropriate drag and drop messages.
If you want to really and truly fix it, then you would need to
Detect when your process is running elevated
Spawn a non-elevated helper process which registers as the drop target
Pass the dropped data through IPC to the elevated process, using a method that's safe to use across privilege boundaries (i.e. no active objects which carry code)
This is a LOT of extra work when the workaround could be as simple as dropping from another elevated app (to get an elevated Explorer, just call up the File->Open dialog of any elevated app), but has the advantage that drag-and-drop will work properly if any of your customers ever run the app elevated.
Unfortunately this looks like a flaw in the OS:
http://blogs.msdn.com/b/patricka/archive/2010/01/28/q-why-doesn-t-drag-and-drop-work-when-my-application-is-running-elevated-a-mandatory-integrity-control-and-uipi.aspx
Oh well.
I find this whole issue appalling. MS has screwed up on this IMO. Essentially we can't debug an app correctly if I want to run MSVC in elevated mode (for example when I build the solution it registers COM servers).
http://social.msdn.microsoft.com/forums/en-US/windowsuidevelopment/thread/2fa935cf-be57-4bcc-9b96-7ee5a6b2b7a5/

Run .exe outside IDE but use break points inside IDE

Using VS .NET 2003. Would like to run the .exe from outside the IDE (i.e. command prompt or double clicking .exe icon in windows) However, still want break points to hit in the IDE.
How do I set this up?
(Running from outside IDE but IDE seeing it as run from "Debug" -> "Start")
Thanks.
On the Debug menu, choose the "Attach to process" option to attach a debugger to your externally-running application.
Visual Studio enables Just In Time Debugging by default. If you haven't turned it off you can call DebugBreak() and you will get a popup allowing you to attach a debugger. If you don't attach a debugger then the program will exit, so you could try wrapping the DebugBreak call in a MessageBox or some other conditional code based on an environment variable or config item.
Since it is C the call to DebugBreak() is correct - this will give you a nasty error dialog (different look depending on the OS), which should have a 'Debug' option. If you click this you should get a dialog to select one of the installed debuggers (VS.NET shoud be among them). Selecting it should bring you to the DebugBreak() line. However this can fail if the debugger can not find the pdb files for your app - in that case you will just get the disassembly view and no source code view.
You can also use WinDBG and the 'Open executable option' - again it will need the pdb files to yield anything useful.

DebugBreak not breaking

I'm writing a class in C++ that I cannot debug by using F5. The code will run from another "service" that will invoke it.
In the past I've used __debugbreak() and when I got a window telling me that an exception was thrown selected to debug it.
Recently I've updated to windows 7 and it kept working for a while.
Today when I've tried to debug a piece of my code instead of shown the regular dialog that tells me that VSTestHost has stopped working and enable me to to debug the application I got a different dialog suggesting I send the data to microsoft for analysis.
Does anyone knows how can I fix this issue so I'll be able to debug my code?
Finally I found the cause of the issue.
It's a Vista/Win7 cause:
Open The Action center control
Goto Action Center settings
Goto Problem Reporting Settings
Choose "Each time a problem occurs, ask me before checking for solution"
Although this is more of IT solution/question I've been plagued with this problem all day and wanted to share the solution with other developers who encounter this problem.
I finally found the solution for Windows 10/11 here:
https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/enabling-postmortem-debugging
And also: https://learn.microsoft.com/en-us/windows/desktop/Debug/configuring-automatic-debugging
To enable automatic debugger launch, you should add a registry value:
key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug, value Auto = 1 (of type REG_DWORD)
The configured debugger is set the by the value Debugger (type REG_SZ); a Visual Studio installation sets this to:
"C:\WINDOWS\system32\vsjitdebugger.exe" -p %ld -e %ld
Note that on 64 bit OS this only works for 64 bit executables. To enable the same behaviour in 32 bit executables set the same values in this key:
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug
In that case...
http://community.codesmithtools.com/blogs/blake/archive/2009/06/03/tips-amp-tricks-debugging-codesmith-on-microsoft-windows-7.aspx
Here is the quick overview of what you need to-do to enable debugging on a Microsoft Windows 7 machine:
Update the Just-In-Time debugger setting DbgJITDebugLaunchSetting. The setting is found in the registry at [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework].
Set the value of DbgJITDebugLaunchSetting to 2.
If you are using a 64bit operating system then you must also set the same key (DbgJITDebugLaunchSetting) in this folder [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework] to 2.
If you run into any issues try running CodeSmith Studio and Visual Studio as an administrator.
Now when CodeSmith enters a break point you will see something like this:
You could try debug > attach to process.