Extending functionalitty (aka Modifying) visual studio C/C++ compiler - c++

I want to extend the features of C/C++ compiler used in Visual studio. Basically, I want to write a tool which parses the c/c++ code and prints out where all branching (if check, break statement, for/while loops, etc) happens in the code. Then I would like to use this information while executing code to grey out areas of code that have not been executed, for a given testcase.
Is it possible? Does Microsoft provide any way to add features to its compiler/debugger?
--Thanks

Microsoft's compilers always were a black box. (Taken from their own site from the upcoming link). So you have a chance that it is not possible right now. But with project Roslyn that's about to be changed.
Anyway, it seems to me you shouldn't look into the compiler but the debugging part of Visual Studio. There are APIs that allow you to interact with the debugger and that's probably the road you want to take (and others did take).

You would not normally do that by modifying the compiler, and with Microsoft's compiler you cannot in any case. Rather you would write a preproccessor that instruments the code (inserts additional code at the conditional nodes to trace the control flow), and then a tool that process the trace data to determine what ran.
Visual Studio itself has an add-in architecture that would allow you to render this data in the editor in the manner you describe.
The instrumentation itself is not trivial - it will need to be able to parse all valid C and C++ code and be able to retain the original line number information so that the uninstrumented code can be presented. For completeness it would have to be able to re-factor code using the ternary ?: operator so that its flow could be instrumented. The instrumented code would also need to be the code output by the standard pre-processor rather than the original source code - making line number tracking a little more difficult (although the preprocesor already manages that for use with the debugger).

Use preprocessor to instrument code. Code should spit out a file while running. At breakpoints, process this information and "grey out" code using debugger APIs + visual studio addin mechanism.

Related

Visual Studio C++ Optimizing compiled disassembly

Recently, I was creating some tasks for reverse engineering on C++. I understood I know nothing about the compiler workflow and how to use it properly. Usually, I was using Visual Studio for C++ and never tried to reverse it and read my own functions. However, I need to create a program that is quiet easy to read after disassembly and I wanted to teach other to reverse engineer based on my own code, that would be safe and etc.
How can I compile the code without lots of "junk"? Like security cookies, debugging info, lots of functions (that I didn't actually used). I was reversing many "crackme's" in the past and they were quite readable, so I hope i can create something similar. Is there any way to control the program's flow manually?

Tool for verifying effect of Visual Studio conformance switches on generated code

A recent minor release version of Visual Studio (15.7) implements the compiler switch /permissive-, which among other effects, enables two-phase name lookup for templates (https://learn.microsoft.com/en-us/cpp/build/reference/permissive-standards-conformance). I'd like to enable this switch on our codebase of about 4M lines of code. The problem is, the instantiated templates might change if this switch is enabled (see the first example at https://blogs.msdn.microsoft.com/vcblog/2017/09/11/two-phase-name-lookup-support-comes-to-msvc/), which lead to silent changes in the run-time behavior.
Is there a way to check whether the code generated with this conformance switch enabled is identical to the old code?
I'm aware that the correct answer is "you run your unit tests, duh!". Sadly with the amount of legacy code lying around, this is out of reach for the next couple of years, so I'm looking for an alternative.
Doing a compare on the binaries does not help, since there are differences present due to metadata changes.
New compilation errors aren't really of any help either: they only expose non-conforming syntax; changes in generated code can still be hidden.
Actually seeing the generated code is not important. A tool showing "this line will compile differently" would be sufficient. Or something similar to what "Preprocess to a File" does for the preprocessor.
The best I can think of is checking generated symbols via the dumpbin tool on every .obj file to see whether the same ones are being generated. This can expose only a subset of issues: the set of template instances in a file might be identical, but their locations in the code might be change.
You may analyze the compiler output (at assembly level) to check for differences generated with the 2 compiler settings you mentioned.
Please look at this related SO question about Visual C++ compilers.
Also, better suited for relatively small portions of code,
you may want to use the marvelous GodBolt's Compiler Explorer,
that does the same task but with an handy web interface and extended to various compilers (not only Microsoft Visual C++).
I'm sure it will become one of the most valuable tools in your developer's toolset.

Visual Studio 2015 Debug: How to stay within my own program only

I'm a beginner at c++, and whenever I write a trivial program in VS2015, and try to debug it, the debugger takes me through every single detail of every standard library function that I used, with 20-30 break points between the statements that I actually wrote.
I can't debug at all this way, and I'd really like to know if there's a way to turn this feature off.
I want to debug MY code, not the library code.
I suggest you to read the manual. Consider options like the breakpoints or the "step over".
One workaround is that you could think about using the "Step Into Specific" or not step into Functions using the registry key like this case:
Is there a way to automatically avoiding stepping into certain functions in Visual Studio?

C++ Video Game Programming IDE

I am looking for a C++ IDE in which I can actively play the game and test the updates live instead of testing it, redoing th code, compiling it and running it again. I'm running Windows 7 x86 professional.
This isn't really an answer, and so probably shouldn't get upvotes, but has information.
I don't know of any C++ IDE that can do runtime updates of code, but it's definitely not impossible. There's lots of C++ assemblers which already JIT code, live updates is merely the next step that no IDE has taken quite yet that I know of.
asmjit can JIT C++
Visual Studio can JIT C++/CLI (which isn't quite C++) (RMartinho corrects that VIsual Studio compiles C++/CLI to IL, and then JITs the IL. Tehcnically different.)
cling uses the clang fruntend and LLVM backend, which has a JIT code generation system.
R.Martinho has also reminded me that Microsoft Visual Studio already has this feature. http://msdn.microsoft.com/en-us/library/esaeyddf(v=vs.100).aspx If you "stop" the code, you can make changes, and it will apply those changes and resume execution.
There's an interesting project at http://runtimecompiledcplusplus.blogspot.co.uk/ that is working on this problem and looks like it might work for you; I haven't used it myself but it looks active if still a little raw. It uses the Visual Studio 2010 compiler.
You can't run C++ code without compiling. Minor syntactic differences between languages shouldn't be an issue so you shouldn't limit yourself to just one language.
I suggest you give Unity a chance; there's a fairly robust free version available. You can write scripts in C# (a language similar to C++), or UnityScript (somehting similar to JavaScript) or Boo (similar to Python) and you can test the results right away, without having to compile.
What about Edit and Continue in Visual Studio? In order to use it, you have to pause execution (either by breakpoint or Pause button), recompile and resume. Note, that you can edit the code while the program is running. I know you can't test the game live, but you don't have to reload resources etc. It's IDE integration makes it really easy and straightforward to use.
If you want changes to be visible live, though, consider using script language such as Lua. One of their purposes is what you want to achieve.
I've listed the options for runtime compilation of C++ code on the alternatives wiki page of Runtime Compiled C++.
From the sounds you may interested in either Recode or Alcanterea.
Organize your C++ game to use plugins, and add a feature to load a new (binary version of) plugin during the game.
Then, you could play your game, recompile a plugin, reload it (so invoke your dynamic linker at runtime), and continue playing.
It is not failproof, but it might usually work.
Of course you cannot unload a plugin which has some active call frame on the (or some thread's) call stack. I would suggest to avoid unloading old plugins...

Tell visual studios to skip over?

While debugging i press F11 often to step into a function. For this project i use properties everwhere which is simply a RAII wrapper that checks if i have set the variable and gives me an assert if i have not. Its been useful.
However now debugging is annoying since F11 will step into the property. Can i skip it somehow? by writing attributes, keywords or anything?
I am using VS11beta
This is quite easy to set up when you're dealing with managed code. You can manually mark the function with the DebuggerHiddenAttribute class, and even turn on built-in debugger settings like "Step over properties and operators".
Unfortunately, automatically stepping over a particular function is not supported by Visual Studio for native C++ code. (At least, it wasn't supported up until VS 2010—I haven't had enough time to play with VS 11 to see if this is something they gave us to make up for the fact that they stole all our colors.)
There is a workaround, though, documented a long while ago on Andy Pennell's blog:
How to Not Step Into Functions using the Visual C++ Debugger
Essentially, you edit the following registry key (for VS 2010):
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\10.0\NativeDE\StepOver
or for 64-bit applications:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\NativeDE\StepOver
to specify a regular expression that will be matched against functions by the debugger.
For example, if you don't want the debugger to step into overloaded operators, you can use the following expression:
\scope:operator\oper:=NoStepInto
As the disclaimer in the blog post says:
This is not a documented feature. Well obviously you are reading this “documentation” right here, but what I mean is that is not guaranteed to work as it was never officially tested, not is it supported by Microsoft. Its existence in future versions or update to current versions is not guaranteed.