Difficulty parsing comments that come after ; in a line - c++

Question

I'm parsing a file that has definitions of functions. Since functions may be written in multiple lines, I'm parsing until encountering a ;:
#include <iostream>
#include <string>
void removeLineBreaks(std::string &str)
{
auto pos = str.find('\n');
while (pos != std::string::npos)
{
str.replace(pos, 1, "");
pos = str.find('\n', pos);
}
}
int main()
{
std::ifstream ifStream("a.pr");
std::string sLine;
const char sDelim(';');
while (std::getline(ifStream, sLine, sDelim))
{
sLine += sDelim;
removeLineBreaks(sLine);
// process further
}
}
The text can be something like this:
a=f(b,c); // comment
d=f(e,f);
Since I'm reading until ;, here I get two pieces:
a=f(b,c); and
// comment
\n d=f(e,f);.
If I call removeLineBreaks on the second piece, it'll become // comment d=f(e,f); so it'd be treated a comment by my parser.
What options do I have to make this work correctly? I could think of this - before calling removeLineBreaks on the line, get the string until \n, and if it starts with //, cut that part from the line, and only then call removeLineBreaks.
Any other ideas?

Answer 1

You first need to remove the //comments from your input and only then you can split at the semicolons.
Consider the following input:
a=f(b,c); // Functions comments are not functions; a=F(b,c);
If you first split on semicolons and then remove the comments, then you would end up with two functions:
a=f(b,c);
a=F(b,c);
But you only want to have the first one.
The solution is to:
Read the file line by line (lines delimited by LF).
While doing that remove the line based // comments and also all line breaks.
Combine all input back into a single string.
Split the string on semicolons to extract all functions which are outside comments.
The steps do not have to be done sequentially. You can do all these steps simultaneously on the input stream of characters, end emit a stream of functions. In fact, this is what real parsers would do.
You are essentially writing a simple parser. As your language gets more and more complicated you will find it more and more difficult to parse a file in such a way. For example with the approach above it will be difficult to emit error messages with line number information.
If you want to write a proper parser I would recommend a recursive descent parser together with a PEG (parser expression grammar). This approach is easy to learn, has less pitfalls than other approaches and is yet very powerful for computer languages. See here: https://en.wikipedia.org/wiki/Parsing_expression_grammar
Warning: If you hear people suggesting flex and bison (or lex and yacc), I strongly recommend against using them. They are complicated to use and are very limited in what they can parse and how it needs to be specified. I would rather suggest to use a light-weight and modern parsing framework like PEGTL: https://github.com/taocpp/PEGTL.