Roslyn C# CodeFix fails for 'Project not Equal' - unit-testing

I've written a CodeAnalyzer/CodeFix using VS Community 17.4.3, and I'm trying to test them with the Visual Studio generated unit test code. The analyzer text runs successfully, and the CodeFix produces what appears to be a valid modified document. However,
the CodeFix test fails because CodeFixTest<TVerifier>.FixEachAnalyzerDiagnosticAsync() returns an error. Specifically, the failure is caused by
var fixedProject = await ApplyCodeActionAsync(project, actionToApply, verifier, cancellationToken).ConfigureAwait(false);
if (fixedProject != project)
{
done = false;
project = fixedProject;
break;
}
because the fixedProject and project are not equal.
I'm a bit lost trying to resolve this because no real projects are involved. The test is using hardcoded strings as both input and expected output.
I've validated the output from the CodeFix equals the expected modification fed to the unit test. I was expecting the unit test to succeed.

There was a problem with my altered document after all. This is perhaps a 'newbie' issue writing CodeFix.
My CodeFix imposes a custom sort order on the UsingDirectives in the code under analysis. I extracted the Usings from the CompilationUnitSyntax, cloned each UsingDirective, then sorted them into the desired order. I then replaced the UsingDirectives in the CompilationUnitSyntax in a 'for' loop using ReplaceNode as shown in Microsoft documentation samples.
My replacement loop I believe ran afoul of a classic List (in this case SyntaxList) update problem. After the first replacement, I transiently had two Directives in the CompilationSyntax that had the same 'NameSyntax' due to the sequential replacement algorithm I used. I got no errors, the replacement ran to completion, but the resulting SyntaxTree was not my desired state. Whether this is a bug or feature of ReplaceNode I don't know.
I removed my iterative replacement loop and used CompilationUnitSyntax.WithUsing(SyntaxList) to replace all of the usings at once, using my sorted List to create the SyntaxList. Viola!

Related

How do syntax highlighting tools implement automated testing?

How do syntax highlighting tools such as pygments and textmate bundle do automated testing?
Tools like this often simply resort to a large collection of snippets of text representing a chosen input and the expected output. For instance if you look at the Pygments Github, you can see they have giant lists of text files divided into an input section and a tokens section like so:
---input---
f'{"quoted string"}'
---tokens---
'f' Literal.String.Affix
"'" Literal.String.Single
'{' Literal.String.Interpol
'"' Literal.String.Double
'quoted string' Literal.String.Double
'"' Literal.String.Double
'}' Literal.String.Interpol
"'" Literal.String.Single
'\n' Text
Since a highlighting tool reads a piece of code and then has to identify which bits of text are parts of which bits of code (is this the start of a function? is this a comment? is it a variable name?), they usually perform various processing steps that will result in a list of tokens as above, which they can then feed into the next step (insert highlights from the first Literal.String.Interpol to the next, bold any Literal.String.Single, etc. by generating the appropriate HTML or CSS or other markup relevant to the system). Checking that these tokens are generated properly from the input text is key.
Then, depending on the language the tool is built in you might use an existing testing suite or build your own (pygments seems to use a Python-based tool called pyTest), which essentially consists of running each of the inputs through your tool in a loop, reading the output, and comparing it to the expected values. If the output doesn't match, you can display a message showing what test failed, what the input/output/expected/error values were. If an output passes, you could simply signal with a happy green checkmark. Then when the test finishes, the developer can hopefully reason out what they broke by looking over the results.
It is often a good idea to randomize the order that these inputs so that you can be sure that each step in the test doesn't have side effects that are getting passed along to the next test and cause it to pass or fail incorrectly. It might also be a good idea to time the length of the complete test. If the whole thing was taking 12 seconds yesterday, but now it takes two minutes, we may have broken something even if all the test technically "pass".
In tools like a code highlighter, you often have a good idea of what many of the inputs and outputs will look like before you can code everything up, for instance if some spec document already exists. In that case, it may be a good idea to include tests that you know won't pass right away, but mark them with some tag (perhaps some text marker within the file that says "NOT PASSING", or naming the file in a certain way), and telling your testing suite to expect those tests to fail. Then, as you fix bugs and add features, say you fixed Bug X in your attempt to make test #144 pass. Now when you run the text, it also alerts you that 10 other tests that should be failing are now passing. Congrats! You just saved yourself a lot of work trying to fix several separate problems that were actually caused by the same root issue.
As the codebase is updated, a developer would run and rerun the test to ensure that any changes he makes doesn't break tests that were working before, and then would add new tests to the collection to verify that his new feature, fixed edge case, etc., now has a known expected output that you can be sure someone won't accidentally break in the future.

How Powershell affect test execution in TFS2015

I have ancestral project and my Nightly build fails. I can`t find out how to fix it. Problem is about test cases sequence dependent execution.
Environment:
There is used TFS2015 build definitions. Basically, I have a definition with a Visual Studio Build task and then a Visual Studio Test task. Visual Studio Test task is also overwritten with Powershell file and I see that in TFS Nightly build process my predefined VS Ordered Test statements is ignored.
Important
Test cases are sequence dependent (as I said, this project is ancestral).
Problem
Interesting is that build log files always shows test execution in sequence what I define in VS Ordered Test, but at TFS2015 Detailed report Test results sequence always is different. So I can`t find out what affects test case execution procedures in TFS. Also I am not sure how tests are executed - parallel or sequentially (As I see, both TFS and PowerShell has no indications to run test cases in Parallel).
I have 2 questions:
Powershell brake down all in VS defined conditions?
Which is the best way to define test execution order, so that it takes into account?
Actually, those Test method run in the order that you defined in Ordered Test file during TFS build process. The build log already shows the correct sequence.
Just like you mentioned above, in the test result page, the order is the same as what you defined in the Ordered Test, but you can see that in front of each test method, it has an order number. You could download the test result file to check again and you will find those test methods are run in the correct order.
In the higher version, like TFS 2017, you could click the Column title 'Test' to make it sort by order().
You could also add a Date started column to know which test method run the first.

Debugging and difference search after refactoring in Visual Stduio

I am involved in a c++ refactoring project and sometimes there are differences resulting, when there should be none. Currently, what I do is basically setting a breakpoint at some place, and then go through the program by F10/F11. The first problem is the size of the projects, traversing it takes a lot of time. Second, sometimes I have differences only in the end of a very big test sentences (say, 600 words), thus just getting to the different word is painfully slow.
1. Is it possible to write some kind of macro for Visual Studio, which will start from the breakpoint, then go step-by-step through the program until end while printing some fields?
2. Are there any neat tricks or tools to simplify the task?
Thanks!
You can create Macros by using Tools>Macros>Macro IDE
If prefer the following method because it's faster for me.
You can record macros using Tools>Macros>Record temporary macro
Everything you type will then be recorded into a macro.
After you recorded what you want to be automated, you can edit the generated code by using View>Other windows>Macro Explorer. Your macro will be recorded in MyMacros>RecordingModule>TemporaryMacro in Macro Explorer window. If you right click that and select edit.
One way to test if the program is terminated:
While Not DTE.Debugger.CurrentProgram Is Nothing

VS2010 and Create Unit Tests... no tests generated

I'm trying to add some unit tests to an existing code base using Visual Studio 2010's unit test generator. However, in some cases when I open a class, right click --> Create Unit Tests..., after I select the methods to generate tests for it will create what is essentially a blank test. Are there situations where this can happen? In every case I select at least one public method to gen tests for, and all it generates is this:
using TxRP.Controllers; //The location of the code to be tested
using Microsoft.VisualStudio.TestTools.UnitTesting;
That's it. Nothing else. Strange, right?
I should note that this is all MVC 2 controller code, and I have been able to gen tests for other controllers with no problem, and all my controllers follow pretty much the same format. No error seems to be thrown, as it gens the empty page happily and adds it to the project as if everything is just fine.
Has anyone had experience with the same type of thing happening, and was there any answer found as to why?
UPDATE:
There is in fact an error during generation:
While trying to generate your tests, the following errors occurred:
Value cannot be null.
Parameter name: key
After some research, the only possible solution I found is that this error occurrs if you're trying to generate tests to a test file that already exists. However, this solution is not working for me...
If you try to generate tests for a class which already has existing tests in another file in the project, it will just generate an empty file as described above. Changing the filename is not sufficient, nor is using a different location within the project. Basically it seems to enforce the one-testfile-per-class convention across the entire project.
This problem is caused by the previously generated test file having been moved to a folder other than the root folder in the test project.
Resolution
Move the test file into the test project root folder.
Generate the new tests
Move the test file back to the folder location you want in the test project.
I have no clue why they dont call it a BUG! in a typical enterprise level software development it is more than a coincidence where multiple people generate unit tests for different methods of the same class # different points of time.
We always end up with this error and it is not helping us any way! Feels as if the Context Menu "Create Unit Tests" has lil use!
Error description:
"While trying to generate your tests, the following errors occurred:
Value cannot be null.
Parameter name: key
"

CPP unit setup for C++

In CPP unit we run unit test as part of build as part of post build setup. We will be running multiple tests as part of this. In case if any test case fails post build should not stop, it should go ahead and run all the test cases and should report summary how many test cases passed and failed. how can we achieve this.
Thanks!
His question is specific enough. You need a test runner. Encapsulate each test in its own behavior and class. The test project is contained separately from the tested code. Afterwards just configure your XMLOutputter. You can find an excellent example of how to do this in the linux website. http://www.yolinux.com/TUTORIALS/CppUnit.html
We use this way to compile our test projects for our main projects and observe if everything is ok. Now it all becomes the work of maintaining your test code.
Your question is too vague for a precise answer. Usually, a unit test engine return a code to tell it has failed (like a non zero return code in the shell on linux) or generate some output file with results. The calling system handle this. If you have written it (some home made scripts) you have to give the option to go on tests execution even if an error occurred. If you are using some tools like continuous integration server, then you have to go through the doc and find the option that allows you to go on when tests fails.
A workaround is to write a script that return a "OK" result even if the unit test fails, but there you lose some automatic verification ...
Be more specific if you want more clues.
my2c
I would just write your tests this way. Instead of using the CPPUNIT_ASSERT macros or whatever you would write them in regular C++ with some way of logging errors.
You could use a macro for this too of course. Something like:
LOGASSERT( some_expression )
could be defined to execute some_expression and to log the expression together with FILE and LINE if it fails, and you can also log exceptions of course, as well as ones that are not thrown, simply by writing them in your tests (with macros if you want to log the expression that caused them with FILE and LINE).
If you are writing macros I would advise you to limit the content of your macro to calling an inline function with extra parameters.