Perhaps someone could tell me what is happening here?
My intention is to split an input string on braces: ie: either '(' or ')'.
For an input string of "(well)hello(there)world" I expect 4 tokens to be returned: well; hello; there; world.
As you can see from my exemplar app below I am getting 5 tokens back (The 1st is an empty string).
Is there any way to get this to return me only the non-empty strings?
#include <iostream>
#include <boost/algorithm/string.hpp>
#include <vector>
int main()
{
std::string in = "(well)hello(there)world";
std::vector<std::string> tokens;
boost::split(tokens, in, boost::is_any_of("()"));
for (auto s : tokens)
std::cout << "\"" << s << "\"" << std::endl;
return 0;
}
Output:
$ a.out
"" <-- where is this token coming from?
"well"
"hello"
"there"
"world"
I have tried using boost::algorithm::token_compress_on but I get the same result.
Yes, the first result returned is the empty set {} immediately preceding the first open parenthesis. The behavior is as expected.
If you don't want to use that result, simply test for an empty returned variable and discard it.
To test that this is the expected behavior, put a parenthesis at the end of the line and you will have another empty result at the end. :)
this thread is kinda old but this is better solution boost::token_compress_on, add this after the delimeter in boost::split
Related
I use this function to remove duplicate words in a file
But I need it to remove duplicate expressions instead
for example What the function is currently doing
If I have the expression
"Hello World"
"beautiful world"
The function will remove the word "world" from both expressions
And I need this function to replace the entire expression only if it is found more than once in the file
for example
If I have the expressions
"Hello World"
"Hello World"
"beautiful world"
"beautiful world"
The function will remove the expression "Hello world" and "beautiful world" and leave only one from each of them but it will not touch the word "world" because the function will treat everything that is within the quotes as one word
This is the code I use now
#include <string>
#include <sstream>
#include <iostream>
#include <unordered_set>
void Remove_Duplicate_Words(string str)
{
ofstream Write_to_file{ "test.txt" };
// Used to split string around spaces.
istringstream ss(str);
// To store individual visited words
unordered_set<string> hsh;
// Traverse through all words
do
{
string word;
ss >> word;
// If current word is not seen before.
while (hsh.find(word) == hsh.end()) {
cout << word << '\n';
Write_to_file << word << endl; // write to outfile
hsh.insert(word);
}
} while (ss);
}
int main()
{
ifstream Read_from_file{ "test.txt" };
string file_content{ ist {Read_from_file}, ist{} };
Remove_Duplicate_Words(file_content);
return 0;
}
How do I remove duplicate expressions instead of duplicate words?
Unfortunately my knowledge on this subject is very basic and usually what I do is try all kinds of things until I succeed. I tried to do it here too and I just can not figure out how to do it
Any help would be greatly appreciated
Requires a little bit of String parsing.
Your example works by reading tokens, which are similar to words (but not exactly). For your problem, the token becomes word OR quoted string. The more complex your definition of tokens, the harder the problem becomes. Try starting by thinking of tokens as either words or quoted strings on the same line. A quoted string across lines might be a little more complex.
Here's a similar SO question to get you started: Reading quoted string in c++. You need to do something similar, but instead of having set positions, your quoted string can occur anywhere in the line. So you read tokens something like this:
Read next word token (as you're doing now)
If last read token is quote character ("), read till next (") as a single token
Check on the set and output token only if it isn't already there (if token is quoted, don't forget to output the quotes)
Insert token into set.
Repeat till EOF
Hope that helps
I'm building a code interpreter in C++ and while I have the whole token logic working, I ran into an unexpected issue.
The user inputs a string into the console, the program parses said string into different objects type Token, the problem is that the way I do this is the following:
void splitLine(string aLine) {
stringstream ss(aLine);
string stringToken, outp;
char delim = ' ';
// Break input string aLine into tokens and store them in rTokenBag
while (getline(ss, stringToken, delim)) {
// assing value of stringToken parsed to t, this labes invalid tokens
Token t (readToken(stringToken));
R_Tokens.push_back(t);
}
}
The issue here is that if the parse receives a string, say Hello World! it will split this into 2 tokens Hello and World!
The main goal is for the code to recognize double quotes as the start of a string Token and store it whole (from " to ") as a single Token.
So if I type in x = "hello world" it will store x as a token, then next run = as a token, and then hello world as a token and not split it
You can use C++14 quoted manipulator.
#include <string>
#include <sstream>
#include <iomanip>
#include <iostream>
void splitLine(std::string aLine) {
std::istringstream iss(aLine);
std::string stringToken;
// Break input string aLine into tokens and store them in rTokenBag
while(iss >> std::quoted(stringToken)) {
std::cout << stringToken << "\n";
}
}
int main() {
splitLine("Heloo world \"single token\" new tokens");
}
You really don't want to tokenize a programming language by splitting at a delimiter.
A proper tokenizer will switch on the first character to decide what kind of token to read and then keep reading as long as it finds characters that fit that token type and then emit that token when it finds the first non-matching character (which will then be used as the starting point for the next token).
That could look something like this (let's say it is an istreambuf_iterator or some other iterator that iterates over the input character-by-character):
Token Tokenizer::next_token() {
if (isalpha(*it)) {
return read_identifier();
} else if(isdigit(*it)) {
return read_number();
} else if(*it == '"') {
return read_string();
} /* ... */
}
Token Tokenizer::read_string() {
// This should only be called when the current character is a "
assert(*it == '"');
it++;
string contents;
while(*it != '"') {
contents.push_back(*it);
it++;
}
return Token(TokenKind::StringToken, contents);
}
What this doesn't handle are escape sequences or the case where we reach the end of file without seeing a second ", but it should give you the basic idea.
Something like std::quoted might solve your immediate problem with string literals, but it won't help you if you want x="hello world" to be tokenized the same way as x = "hello world" (which you almost certainly do).
PS: You can also read the whole source into memory first and then let your tokens contain indices or pointers into the source rather than strings (so instead of the contents variable, you'd just save the start index before the loop and then return Token(TokenKind::StringToken, start_index, current_index)). Which one's better depends partly on what you do in the parser. If your parser directly produces results and you don't need to keep the tokens around after processing them, the first one is more memory-efficient because you never need to hold the whole source in memory. If you create an AST, the memory consumption will be about the same either way, but the second version will allow you to have one big string instead of many small ones.
So I finally figured it out, and I CAN use getline() to achieve my goals.
This new code runs and parses the way I need it to be:
void splitLine(string aLine) {
stringstream ss(aLine);
string stringToken, outp;
char delim = ' ';
while (getline(ss, stringToken, delim)) { // Break line into tokens and store them in rTokenBag
//new code starts here
// if the current parse sub string starts with double quotes
if (stringToken[0] == '"' ) {
string torzen;
// parse me the rest of ss until you find another double quotes
getline(ss, torzen, '"' );
// Give back the space cut form the initial getline(), add the parsed sub string from the second getline(), and add a double quote at the end that was cut by the second getline()
stringToken += ' ' + torzen + '"';
}
// And we can all continue with our lives
Token t (readToken(stringToken)); // assing value of stringToken parsed to t, this labes invalid tokens
R_Tokens.push_back(t);
}
}
Thanks to everyone who answered and commented, you were all of great help!
I am trying to write some code that changes a string like "/path/file.extension" to another specified extension. I am trying to use boost::xpressive to do so. But, I am having problems. It appears that a garbage character appears in the output:
#include <iostream>
#include <boost/xpressive/xpressive.hpp>
using namespace boost::xpressive;
using namespace std;
int main()
{
std::string str( "xml.xml.xml.xml");
sregex date = sregex::compile( "(\\.*)(\\.xml)$");
std::string format( "\1.zipxml");
std::string str2 = regex_replace( str, date, format );
std::cout << "str = " << str << "\n";
std::cout << "str2 = " << str2 << "\n";
return 0;
}
Now compile and run it:
[bitdiot#kantpute foodir]$ g++ badregex.cpp
[bitdiot#kantpute foodir]$ ./a.out > output
[bitdiot#kantpute foodir]$ less output
[bitdiot#kantpute foodir]$ cat -vte output
str = xml.xml.xml.xml$
str2 = xml.xml.xml^A.zipxml$
In the above example, I redirect output to a file, and use cat to print out the non-printable character. Notice the ctrl-A in the str2.
Anyways, am I using boost libraries incorrectly? Is this a boost bug? Is there another regular expression I can use that can allow me to string replace the ".tail" with some other string? (It's fix in my example.)
thanks.
At least as I'm reading things, the culprit is right here: std::string format( "\1.zipxml");.
You forgot to escape the backslash, so \1 is giving you a control-A. You almost certainly want \\1.
Alternatively (if your compiler is new enough) you could use a raw string instead, so it would be something like: R"(\1.zipxml)", and you wouldn't have to escape your backslashes. I probably wouldn't bother to mention this, except for the fact that if you're writing REs in C++ strings, raw strings are pretty much your new best friend (IMO, anyway).
As Jerry Coffin pointed out to me. It was a stupid mistake on my part.
The errant code is the following:
std::string format( "\1.zipxml");
This should be replaced with:
std::string format( "$1.zipxml");
Thanks for your help everyone.
I am having problem formatting a string which contains quotationmarks.
For example, I got this std::string: server/register?json={"id"="monkey"}
This string needs to have the four quotation marks replaced by \", because it will be used as a c_str() for another function.
How does one do this the best way on this string?
{"id"="monkey"}
EDIT: I need a solution which uses STL libraries only, preferably only with String.h. I have confirmed I need to replace " with \".
EDIT2: Nvm, found the bug in the framework
it is perfectly legal to have the '"' char in a C-string. So the short answer is that you need to do nothing. Escaping the quotes is only required when typing in the source code
std::string str("server/register?json={\"id\"=\"monkey\"}")
my_c_function(str.c_str());// Nothing to do here
However, in general if you want to replace a substring by an other, use boost string algorithms.
#include <boost/algorithm/string/replace.hpp>
#include <iostream>
int main(int, char**)
{
std::string str = "Hello world";
boost::algorithm::replace_all(str, "o", "a"); //modifies str
std::string str2 = boost::algorithm::replace_all_copy(str, "ll", "xy"); //doesn't modify str
std::cout << str << " - " << str2 << std::endl;
}
// Displays : Hella warld - Hexya warld
If you std::string contains server/register?json={"id"="monkey"}, there's no need to replace anything, as it will already be correctly formatted.
The only place you would need this is if you hard-coded the string and assigned it manually. But then, you can just replace the quotes manually.
I have a std::wstring variable that contains a text and I need to split it by separator. How could I do this? I wouldn't use boost that generate some warnings. Thank you
EDIT 1
this is an example text:
hi how are you?
and this is the code:
typedef boost::tokenizer<boost::char_separator<wchar_t>, std::wstring::const_iterator, std::wstring> Tok;
boost::char_separator<wchar_t> sep;
Tok tok(this->m_inputText, sep);
for(Tok::iterator tok_iter = tok.begin(); tok_iter != tok.end(); ++tok_iter)
{
cout << *tok_iter;
}
the results are:
hi
how
are
you
?
I don't understand why the last character is always splitted in another token...
In your code, question mark appears on a separate line because that's how boost::tokenizer works by default.
If your desired output is four tokens ("hi", "how", "are", and "you?"), you could
a) change char_separator you're using to
boost::char_separator<wchar_t> sep(L" ", L"");
b) use boost::split which, I think, is the most direct answer to "split a wstring by specified character"
#include <string>
#include <iostream>
#include <vector>
#include <boost/algorithm/string.hpp>
int main()
{
std::wstring m_inputText = L"hi how are you?";
std::vector<std::wstring> tok;
split(tok, m_inputText, boost::is_any_of(L" "));
for(std::vector<std::wstring>::iterator tok_iter = tok.begin();
tok_iter != tok.end(); ++tok_iter)
{
std::wcout << *tok_iter << '\n';
}
}
test run: https://ideone.com/jOeH9
You're default constructing boost::char_separator. The documentation says:
The function std::isspace() is used to identify dropped delimiters and std::ispunct() is used to identify kept delimiters. In addition, empty tokens are dropped.
Since std::ispunct(L'?') is true, it is treated as a "kept" delimiter, and reported as a separate token.
Hi you can use wcstok function
You said you don't want boost so...
This is maybe a wierd approach to use in C++ but I used it one in a MUD where i needed a lot of tokenization in C.
take this block of memory assigned to the char * chars:
char chars[] = "I like to fiddle with memory";
If you need to tokenize on a space character:
create array of char* called splitvalues big enough to store all tokens
while not increment pointer chars and compare value to '\0'
if not already set set address of splitvalues[counter] to current memory address - 1
if value is ' ' write 0 there
increment counter
when you finish you have the original string destroyed so do not use it, instead you have the array of strings pointing to the tokens. the count of tokens is the counter variable (upperbound of the array).
the approach is this:
iterate the string and on first occurence update token start pointer
convert the char you need to split on to zeroes that mean string termination in C
count how many times you did this
PS. Not sure if you can use a similar approach in a unicode environment tough.