Parsing tokens issue during interpreter development - c++

Question

I'm building a code interpreter in C++ and while I have the whole token logic working, I ran into an unexpected issue.
The user inputs a string into the console, the program parses said string into different objects type Token, the problem is that the way I do this is the following:
void splitLine(string aLine) {
stringstream ss(aLine);
string stringToken, outp;
char delim = ' ';
// Break input string aLine into tokens and store them in rTokenBag
while (getline(ss, stringToken, delim)) {
// assing value of stringToken parsed to t, this labes invalid tokens
Token t (readToken(stringToken));
R_Tokens.push_back(t);
}
}
The issue here is that if the parse receives a string, say Hello World! it will split this into 2 tokens Hello and World!
The main goal is for the code to recognize double quotes as the start of a string Token and store it whole (from " to ") as a single Token.
So if I type in x = "hello world" it will store x as a token, then next run = as a token, and then hello world as a token and not split it

Answer 1

You can use C++14 quoted manipulator.
#include <string>
#include <sstream>
#include <iomanip>
#include <iostream>
void splitLine(std::string aLine) {
std::istringstream iss(aLine);
std::string stringToken;
// Break input string aLine into tokens and store them in rTokenBag
while(iss >> std::quoted(stringToken)) {
std::cout << stringToken << "\n";
}
}
int main() {
splitLine("Heloo world \"single token\" new tokens");
}

Answer 2

You really don't want to tokenize a programming language by splitting at a delimiter.
A proper tokenizer will switch on the first character to decide what kind of token to read and then keep reading as long as it finds characters that fit that token type and then emit that token when it finds the first non-matching character (which will then be used as the starting point for the next token).
That could look something like this (let's say it is an istreambuf_iterator or some other iterator that iterates over the input character-by-character):
Token Tokenizer::next_token() {
if (isalpha(*it)) {
return read_identifier();
} else if(isdigit(*it)) {
return read_number();
} else if(*it == '"') {
return read_string();
} /* ... */
}
Token Tokenizer::read_string() {
// This should only be called when the current character is a "
assert(*it == '"');
it++;
string contents;
while(*it != '"') {
contents.push_back(*it);
it++;
}
return Token(TokenKind::StringToken, contents);
}
What this doesn't handle are escape sequences or the case where we reach the end of file without seeing a second ", but it should give you the basic idea.
Something like std::quoted might solve your immediate problem with string literals, but it won't help you if you want x="hello world" to be tokenized the same way as x = "hello world" (which you almost certainly do).
PS: You can also read the whole source into memory first and then let your tokens contain indices or pointers into the source rather than strings (so instead of the contents variable, you'd just save the start index before the loop and then return Token(TokenKind::StringToken, start_index, current_index)). Which one's better depends partly on what you do in the parser. If your parser directly produces results and you don't need to keep the tokens around after processing them, the first one is more memory-efficient because you never need to hold the whole source in memory. If you create an AST, the memory consumption will be about the same either way, but the second version will allow you to have one big string instead of many small ones.

Answer 3

So I finally figured it out, and I CAN use getline() to achieve my goals.
This new code runs and parses the way I need it to be:
void splitLine(string aLine) {
stringstream ss(aLine);
string stringToken, outp;
char delim = ' ';
while (getline(ss, stringToken, delim)) { // Break line into tokens and store them in rTokenBag
//new code starts here
// if the current parse sub string starts with double quotes
if (stringToken[0] == '"' ) {
string torzen;
// parse me the rest of ss until you find another double quotes
getline(ss, torzen, '"' );
// Give back the space cut form the initial getline(), add the parsed sub string from the second getline(), and add a double quote at the end that was cut by the second getline()
stringToken += ' ' + torzen + '"';
}
// And we can all continue with our lives
Token t (readToken(stringToken)); // assing value of stringToken parsed to t, this labes invalid tokens
R_Tokens.push_back(t);
}
}
Thanks to everyone who answered and commented, you were all of great help!