How to write a shell extension in C++? - c++

Question

This seemed like a common question but after doing some searching, I wasn't really able to find my answers. There is an article on this here:
http://www.codeproject.com/KB/shell/shellextguide1.aspx
But it's for a very old version of Visual Studio. I'm using VS 2008, so the instructions and interfaces don't seem to match what I'm seeing.
I want to create a simple shell extension using C++ that creates a context menu for files with extension .GZ. When right clicking on these files, I should be able to click my context menu item and have a callback in code to do some sort of operation on that file.
Other context menu items would do things like spawn modless dialogs to accept user input before executing some action.
From what I've seen, ATL is used for this but I have never used ATL, so all of the object types and interfaces are very confusing to me. It wouldn't be so bad if I had a proper tutorial or documentation to read.
Can anyone help me out? Isn't there some sort of tutorial out there that isn't 10 years old?

Answer 1

I can't tell you exactly how to write a shell extension, but I will provide a number of tips. Writing a Shell Extension offers some significant advantages over the much simpler “registry-only” method:
With a Shell Extension, you can dynamically create a context menu item (or submenu) that is more relevant to the selected file(s). For example, if you are writing a Shell Extension for zip files, it is possible to create a submenu within the context menu that shows the entire contents of the zip.
You can handle multiple files simultaneously, which may be more beneficial not just for performance purposes but also so that you can work out what to do based on the selection as a whole rather than just for each file.
Some of the downfalls to Shell Extensions are:
Substantially increased complexity. Be prepared to spend a lot of effort on this to get it working. Have a home-espresso machine installed next to your computer and/or hire someone to make you coffee.
Substantially increased difficulty in debugging. Ditto about coffee.
It's difficult to write a Shell Extension because they can be very hard to debug.
Shell Extensions are loaded by the explorer.exe process, and without specific configuration of Explorer, you need to force-quit the explorer.exe process so that you can install a newer version of your Shell Extension. There is a way to get Explorer to unload DLLs that it is no longer using, but you should only do this on a development machine and not on a deployment target:
In RegEdit, browse to the following key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer
Add a new DWORD key called “AlwaysUnloadDLL” and set its value to 1.
Restart explorer.
This works most of the time, but there may still be times where you need to close Explorer because the Shell Extension was not unloaded.
Keep in mind that your Shell Extension may be loaded by other applications, for example, if you right-click on a file with an applications “open file” dialog, then your Shell Extension will be loaded into that application, and not Explorer.
If your Shell Extension causes a runtime error, quite often the result will simply be that your context menu item does not show, very rarely will you be told that your Shell Extension failed to load or that it caused a runtime error.
Configuration can be hard, even with an installation, registry data needs to be created in several places, and depending where you want your context menu to show, the places in the registry may differ between different versions of Windows.
What you'll need to do:
Visual Studio offers some shortcuts to creating Shell Extensions, but basically you'll need to create a COM DLL. A Shell Extension for context menu items must implement both the IContextMenu interface and the IShellExtInit interface.
In the IShellExtInit::Initialize() method, you can obtain the selected files from the IDataObject parameter. From memory, the data is in “Drag-n-Drop” format, so you need to get an HDROP handle from the IDataObject and query the files from there (this is from memory, it may actually be different than as I described here, so proceed with caution).
Once your DLL is ready to be “installed”, you must copy it somewhere, and then run regsvr32 to make sure it is registered.
Follow this guide to know where to put registry keys.
There may be issues with 64-bit Windows, if you build a 32-bit DLL it may not load in 64-bit Explorer… so keep this in mind if you are having trouble with 64-bit Windows.
Your DLL will actually have two GUIDs associated with it. I can't remember exactly how it works, but one GUID refers to the DLL itself and the other refers to the actual Shell Extension. Make sure you use the GUID of the actual Shell Extension when creating keys in the registry where a GUID is required.
All things considered… (tl;dr)
Weigh up the costs of whether a Shell Extension is worth it. If you want to create menu items dynamically based on the selected files, then a Shell Extension may be the only way. If you want to handle all files simultaneously then you'll probably need a Shell Extension as well.
An alternative to the context menu method, could be to have a drag-n-drop target on the user's desktop or something. Explore other ways that you could have the user submit your files to your application, because a Shell Extension is often far more effort than it is worth. I found this out the hard way and I think everyone else has too.