How to find and replace SkippedTokensTrivia using Roslyn - roslyn

Question

I'm trying to fix the following VBA statement (converting some old code just for fun and to learn Roslyn, not at all looking for anything perfect) to remove the Set keyword so it's a valid VB.NET statement:
Set f = New Foo()
When I look at it through the Syntax Visualizer, I see it turns into trailing trivia.
I'm trying to figure out how to find it using a query. I tried several approaches but all of the following came up empty:
var attempt1 = root.DescendantTokens().Where(t=>t.IsKind(SyntaxKind.SkippedTokensTrivia));
var attempt2 = root.DescendantTokens().Where(t => t.IsKind(SyntaxKind.SetKeyword));
var attempt3 = root.DescendantTrivia().Where(t => t.IsKind(SyntaxKind.SetKeyword));
var attempt4 = root.DescendantNodes()
.OfType<EmptyStatementSyntax>()
.Where(e => e.DescendantTokens().Any(t => t.IsKeyword()));
(Yes, I'm using C# to work with a VisualBasicSyntaxTree)
I can't seem to just find the SetKeyword token that appears in the visualizer, so I thought maybe it's doing some more heavy lifting to piece together what it really is (is that what's meant by structured trivia?). I read something in the documentation that mentioned the compiler can choose to represent it a couple of different ways, so I thought that may be what's going on here.
The query was just the first thing I tried, but in reality I have a SyntaxRewriter I'm using to visit the code to find and fix all such problems (I'm already able to fix missing parentheses around ArgumentLists, for example) but in this case I can't seem to figure out which Visit method to override.
So again, 1) how to query for these from the root and 2) the best override to select from a rewriter. I've been beating my face on the keyboard for two days on this which exponentially increases the likelihood that I'm having a cranio/recto-insertion moment and I need one of you kind souls to pull me out of it.
Cheers!
Brian
Edit: Fixed typo in query attempt1

Answer 1

So it appears that when the compiler reaches an error condition, it will skip all tokens up to the next point where it can recover and continue parsing (the end of the line in this case). The node representing this error condition is an EmptyStatement with trailing syntax trivia containing the rest of the text as parsed tokens.
So if you're going to rewrite a node, you'll want to rewrite EmptyStatements. But you don't want to write just any empty statement, just the ones with the "BC30807" diagnostic code.
public override SyntaxNode VisitEmptyStatement(EmptyStatementSyntax node)
{
var diagnostic = GetLetSetDiagnostic(node);
if (diagnostic == null)
return base.VisitEmptyStatement(node);
return RewriteLetSetStatement(node);
}
private Diagnostic GetLetSetDiagnostic(EmptyStatementSyntax node)
{
//'Let' and 'Set' assignment statements are no longer supported.
const string code = "BC30807";
return node.GetDiagnostics().SingleOrDefault(n => n.Id == code);
}
The implementation of the RewriteLetSetStatement() method is a bit of a mystery to me, I'm not sure how it can be implemented utilizing the compiler services effectively, I don't think that this is a use case that it covers well. The trivia retains the parsed tokens, but there's not much you can do with those tokens AFAIK.
Ideally, we'd just want to ignore the Set token from the tokens and throw it back into the parser to be reparsed. And as far as I can tell, that's not possible, we can only parse from text.
So, I guess the next best thing to do would be to take the text, rewrite it to remove the Set and parse the text again.
private SyntaxNode RewriteLetSetStatement(EmptyStatementSyntax node)
{
var letSetTokens = node.GetTrailingTrivia()
.Where(triv => triv.IsKind(SyntaxKind.SkippedTokensTrivia))
.SelectMany(triv => triv.GetStructure().ChildTokens())
.TakeWhile(tok => new[] {SyntaxKind.LetKeyword, SyntaxKind.SetKeyword}
.Contains(tok.VisualBasicKind()));
var span = new RelativeTextSpan(node.FullSpan);
var newText = node.GetText().WithChanges(
// replacement spans must be relative to the text
letSetTokens.Select(tok => new TextChange(span.GetSpan(tok.Span), ""))
);
return SyntaxFactory.ParseExecutableStatement(newText.ToString());
}
private class RelativeTextSpan(private TextSpan span)
{
public TextSpan GetSpan(TextSpan token)
{
return new TextSpan(token.Start - span.Start, token.Length);
}
}