I wondering about an efficient way to split text into sentences.
Sentences are split by a dot + space
Example text
The quick brown fox jumps
over the lazy dog. I love eating toasted cheese and tuna sandwiches.
My algorithm works like this
Read first line from text file to string
Find what is needed
Write to file
However sometimes half of a sentence can be on a upcoming line.
So I was wondering what is the best way to confront this problem
Yes a tried googling "search across multiple lines" and I don't want to use regex
Initially my idea is to check if the first line ends with a .+ space and if not grab another line and search through it. But I have a feeling I am missing out on something.
EDIT: Sorry forgot to mention that I am doing this in C++
You can use something like accumulator.
1. Read line
2. Check the last symbols in this line.
3. If last symbols are dot or dot+space
3.1 Split it and write all strings to output
3.2 GOTO 1
ELSE
3.3 split the line, write length-1 strings to output
3.4 Keep last piece in some variable and append next readed line to it.
Hope my idea is clear.
Here is my approach for this problem
void to_sentences()
{
// Do not skip whitespaces
std::cin >> std::noskipws;
char c;
// Loop until there is no input
while (std::cin >> c) {
// Skip new lines
if (c == '\n')
continue;
// Output the character
std::cout << c;
// check if there is a dot folowed by space
// if there add new line
if (c == '.') {
std::cin >> c;
if (c == ' ')
std::cout << endl;
}
}
// Reset skip whitespaces
std::cin >> std::skipws;
}
You can read the comments and ask if there is something unclear.
You can use std::getline(), with custom delimeter '.'
#include <sstream>
#include <string>
#include <vector>
auto split_to_sentences(std::string inp)
{
std::istringstream ss(inp); // make a stream using the string
std::vector< std::string > sentences; // return value
while(true) {
std::string this_sentence;
std::getline(ss, this_sentence, '.');
if (this_sentence != "")
sentences.push_back(std::move(this_sentence));
else
return sentences;
}
}
Note that if you have the input text as a stream, then you can skip the std::stringstream step, and give the stream directly to std::getline, in the place of ss.
The use of std::move is not necessary, but might increase performance, by preventing a copy and a deletion of the dynamic parts (on heap) of std::string.
Related
I am very new to C++, so I apologize if this isn't a good question but I really need help in understanding how to use istream.
There is a project I have to create where it takes several amounts of input that can be on one line or multiple and then pass it to a vector (this is only part of the project and I would like to try the rest on my own), for example if I were to input this...
>> aaa bb
>> ccccc
>> ddd fff eeeee
Makes a vector of strings with "aaa", "bb", "ccccc", "ddd", "fff", "eeeee"
The input can be a char or string and the program stops asking for input when the return key is hit.
I know getline() gets a line of input and I could probably use a while loop to try and get the input such as...(correct me if I'm wrong)
while(!string.empty())
getline(cin, string);
However, I don't truly understand istream and it doesn't help that my class has not gone over pointers so I don't know how to use istream& or string& and pass it into a vector. On the project description, it said to NOT use stringstream but use functionality from getline(istream&, string&). Can anyone give somewhat of a detailed explanation as to how to make a function using getline(istream&, string&) and then how to use it in the main function?
Any little bit helps!
You're on the right way already; solely, you'd have to pre-fill the string with some dummy to enter the while loop at all. More elegant:
std::string line;
do
{
std::getline(std::cin, line);
}
while(!line.empty());
This should already do the trick reading line by line (but possibly multiple words on one line!) and exiting, if the user enters an empty line (be aware that whitespace followed by newline won't be recognised as such!).
However, if anything on the stream goes wrong, you'll be trapped in an endless loop processing previous input again and again. So best check the stream state as well:
if(!std::getline(std::cin, line))
{
// this is some sample error handling - do whatever you consider appropriate...
std::cerr << "error reading from console" << std::endl;
return -1;
}
As there might be multiple words on a single line, you'd yet have to split them. There are several ways to do so, quite an easy one is using an std::istringstream – you'll discover that it ressembles to what you likely are used to using std::cin:
std::istringstream s(line);
std::string word;
while(s >> word)
{
// append to vector...
}
Be aware that using operator>> ignores leading whitespace and stops after first trailing one (or end of stream, if reached), so you don't have to deal with explicitly.
OK, you're not allowed to use std::stringstream (well, I used std::istringstream, but I suppose this little difference doesn't count, does it?). Changes matter a little, it gets more complex, on the other hand, we can decide ourselves what counts as words an what as separators... We might consider punctuation marks as separators just like whitespace, but allow digits to be part of words, so we'd accept e. g. ab.7c d as "ab", "7c", "d":
auto begin = line.begin();
auto end = begin;
while(end != line.end()) // iterate over each character
{
if(std::isalnum(static_cast<unsigned char>(*end)))
{
// we are inside a word; don't touch begin to remember where
// the word started
++end;
}
else
{
// non-alpha-numeric character!
if(end != begin)
{
// we discovered a word already
// (i. e. we did not move begin together with end)
words.emplace_back(begin, end);
// ('words' being your std::vector<std::string> to place the input into)
}
++end;
begin = end; // skip whatever we had already
}
}
// corner case: a line might end with a word NOT followed by whitespace
// this isn't covered within the loop, so we need to add another check:
if(end != begin)
{
words.emplace_back(begin, end);
}
It shouldn't be too difficult to adjust to different interpretations of what is a separator and what counts as word (e. g. std::isalpha(...) || *end == '_' to detect underscore as part of words, but digits not). There are quite a few helper functions you might find useful...
You could input the value of the first column, then call functions based on the value:
void Process_Value_1(std::istream& input, std::string& value);
void Process_Value_2(std::istream& input, std::string& value);
int main()
{
// ...
std::string first_value;
while (input_file >> first_value)
{
if (first_value == "aaa")
{
Process_Value_1(input_file, first_value);
}
else if (first_value = "ccc")
{
Process_Value_2(input_file, first_value);
}
//...
}
return 0;
}
A sample function could be:
void Process_Value_1(std::istream& input, std::string& value)
{
std::string b;
input >> b;
std::cout << value << "\t" << b << endl;
input.ignore(1000, '\n'); // Ignore until newline.
}
There are other methods to perform the process, such as using tables of function pointers and std::map.
If I had to read in a word from a document (one word at a time), and then pass that word into a function until I reach the end of the file, how would I do this?
What also must be kept in mind is that a word is any consecutive string of letters and the apostrophe ( so can't or rojas' is one word). Something like bad-day should be two separate words, and something like to-be-husband should be 3 separate words. I also need to ignore periods ., semi-colons ;, and pretty much anything that isn't part of a word. I have been reading it in using file >> s; and then removing stuff from the string but it has gotten very complicated. Is there a way to store into s only alphabet characters+apostrophes and stop at the end of a word (when a space occurs)?
while (!file.eof()) {
string s;
file >> s; //this is how I am currently reading it it
passToFunction(s);
}
Yes, there is a way: simply write the code to do it. Read one character at a time, and collect the characters in the string, until you gets a non-alphabetic, non-apostrophe character. You've now read one word. Wait until you read the next character that's a letter or an apostrophe, and then you take it from the top.
One other thing:
while (!file.eof())
This is always a bug, and a wrong thing to do. Just thought I'd mention this. I suppose that fixing this is going to be your first order of business, before writing the rest of your code.
OnlyLetterNumAndApp facet for a stream
#include <locale>
#include <string>
#include <fstream>
#include <iostream>
// This facet treats letters/numbers and apostrophe as alpha
// Everything else is treated like a space.
//
// This makes reading words with operator>> very easy to sue
// when you want to ignore all the other characters.
class OnlyLetterNumAndApp: public std::ctype<char>
{
public:
typedef std::ctype<char> base;
typedef base::char_type char_type;
OnlyLetterNumAndApp(std::locale const& l)
: base(table)
{
std::ctype<char> const& defaultCType = std::use_facet<std::ctype<char> >(l);
for(int loop = 0;loop < 256;++loop) {
table[loop] = (defaultCType.is(base::alnum, loop) || loop == '\'')
? base::alpha
: base::space;
}
}
private:
base::mask table[256];
};
Usage
int main()
{
std::ifstream file;
file.imbue(std::locale(std::locale(), new OnlyLetterNumAndApp(std::locale())));
file.open("test.txt");
std::string word;
while(file >> word) {
std::cout << word << "\n";
}
}
Test File
> cat test.txt
This is %%% a test djkhfdkjfd
try another $gh line's
bad-people.Do bad things
Result
> ./a.out
This
is
a
test
djkhfdkjfd
try
another
gh
line's
bad
people
Do
bad
things
Long story short I need my vector to cout as a single line without creating its own new lines for my program to work correctly. the text file i read into the vector was
laptop#a small computer that fits on your lap#
helmet#protective gear for your head#
couch#what I am sitting on#
cigarette#smoke these for nicotine#
binary#ones and zeros#
motorcycle#two wheeled motorized bike#
oj#orange juice#
test#this is a test#
filled the vector using the loop:
if(myFile.is_open())
{
while(getline(myFile, line, '#'))
{
wordVec.push_back(line);
}
cout << "words added.\n";
}
and printed it using this:
for(int i = 0; i < wordVec.size(); i++)
{
cout << wordVec[i];
}
and it outputs as such:
laptopa small computer that fits on your lap
helmetprotective gear for your head
couchwhat I am sitting on
cigarettesmoke these for nicotine
binaryones and zeros
motorcycletwo wheeled motorized bike
ojorange juice
testthis is a test
my program works if I manually input the words and add them to my data structure but if added from the vector which is filled via text file, half of the program doesnt work. before anyone says asks for a better description of the problem, all I need to know is how to fill the vector so that it will output as a single line.
You code getline(myFile, line, '#') reads everything up to end-of-file or the next '#' into line - that includes any newlines. So, as you read text file content...
laptop#a small computer that fits on your lap#
helmet#protective gear for your head#
...which you could also think of as...
"laptop#a small computer that fits on your lap#\nhelmet#protective gear for your head#"
...line takes on successive values...
"laptop"
"a small computer that fits on your lap"
"\nhelmet"
...etc....
Note the newline in "\nhelmet".
There are many ways to avoid or correct this, such as...
while ((myFile >> std::skipws) and getline(myFile, line, '#'))
...
...or...
if (not line.empty() and line[0] == '\n')
line.erase(0, 1);
...or (as Barry suggests in comments)...
while (getline(myFile, line))
{
std::istringstream iss(line);
std::string field;
while (getline(iss, field, '#'))
...
}
while(getline(myFile, line, '#'))
Here, you told std::getline to use the '#' character instead of a newline, '\n', as a delimiter.
So, this simply means that std::getline will no longer think there's anything special about '\n'. It's just another character that std::getline() will keep reading, looking for the next #.
So, you end up reading newline characters into your individual strings, and then outputing them to std::cout, as part of the strings you've printed.
I have code where I am inputting stuff from a file. My txt file looks like this:
file.txt
hello world
...
1 2
The numbers at the bottom are supposed to be read into variables. As for "hello world", it should be picked up by getline. But I don't know how many lines there will be in the txt file so I don't know how to break out of it. Here is my code:
while (getline(file, line))
{
std::cout << line << std::endl;
// ...
}
file >> a >> b; // 1 2
If I was doing this with cin I could just do Ctrl+Z to stop getline loop from running. How do I break out of the while loop at the right time before I get to 1 2?
For each line string line, you can put it into an istringstream iss. And then try to stream it into a and b using iss >> a >> b, if it can be done successfully, it means you enter the right line. Otherwise, you go on checking the next line.
int a, b;
while (getline(file, line))
{
istringstream iss(line);
if (iss >> a >> b)
{
// you are in the right line, and a,b has the values e.g. 1 2
}
}
It should also work for other strings besides "hello world", like "aaa bbb cc" etc. as long as they are not the numbers you are looking for.
P.S.: you can also take use of regex if you use C++11 to check if given line has/matches the pattern you are looking for.
Use a condition, and a break; statement.
E.g.:
while (getline(file, line))
{
std::cout << line << std::endl;
// ...
if(line == "hello world"/){
break;//Exits the loop
}
}
A break statement makes your code exit the most inner loop it's used in. In this case, it exits the while loop.
EDIT:
If you don't want to break on a specific line, then you'll better use regular expression or another mechanism (like std::stringstream) to find a match of the string you're looking for, and capture the part you're interested in. I suggest you take a look at Boost.Regex for this.
The idea is to loop on the lines, i.e. just as you do. As soon as you have a match, you can break (the same way) and capture from the string you're currently reading (which in your code would be in the line variable).
It's not too clear how you determine that you want to break out
of the loop. What is the criterion? If you want to read all
lines but the last, the simplest solution is to simply read all
of the lines into an std::vector<std::string>, and then
process that; you can iterate over a vector until the next to
the last element (which you can't do on a stream). If it's some
pattern your looking to match (say "\\d+\\s+\\d+"), then you
can add this to the condition:
std::string line;
std::regex matchNumbers( "\\d+\\s+\\d+" );
while ( std::getline( file, line ) && ! regex_match( line, matchNumbers ) ) {
// ...
}
std::istringstream numbers( line );
numbers >> a >> b;
And so on.
I'm implementing a custom lexer in C++ and when attempting to read in whitespace, the ifstream won't read it out. I'm reading character by character using >>, and all the whitespace is gone. Is there any way to make the ifstream keep all the whitespace and read it out to me? I know that when reading whole strings, the read will stop at whitespace, but I was hoping that by reading character by character, I would avoid this behaviour.
Attempted: .get(), recommended by many answers, but it has the same effect as std::noskipws, that is, I get all the spaces now, but not the new-line character that I need to lex some constructs.
Here's the offending code (extended comments truncated)
while(input >> current) {
always_next_struct val = always_next_struct(next);
if (current == L' ' || current == L'\n' || current == L'\t' || current == L'\r') {
continue;
}
if (current == L'/') {
input >> current;
if (current == L'/') {
// explicitly empty while loop
while(input.get(current) && current != L'\n');
continue;
}
I'm breaking on the while line and looking at every value of current as it comes in, and \r or \n are definitely not among them- the input just skips to the next line in the input file.
There is a manipulator to disable the whitespace skipping behavior:
stream >> std::noskipws;
The operator>> eats whitespace (space, tab, newline). Use yourstream.get() to read each character.
Edit:
Beware: Platforms (Windows, Un*x, Mac) differ in coding of newline. It can be '\n', '\r' or both. It also depends on how you open the file stream (text or binary).
Edit (analyzing code):
After
while(input.get(current) && current != L'\n');
continue;
there will be an \n in current, if not end of file is reached. After that you continue with the outmost while loop. There the first character on the next line is read into current. Is that not what you wanted?
I tried to reproduce your problem (using char and cin instead of wchar_t and wifstream):
//: get.cpp : compile, then run: get < get.cpp
#include <iostream>
int main()
{
char c;
while (std::cin.get(c))
{
if (c == '/')
{
char last = c;
if (std::cin.get(c) && c == '/')
{
// std::cout << "Read to EOL\n";
while(std::cin.get(c) && c != '\n'); // this comment will be skipped
// std::cout << "go to next line\n";
std::cin.putback(c);
continue;
}
else { std::cin.putback(c); c = last; }
}
std::cout << c;
}
return 0;
}
This program, applied to itself, eliminates all C++ line comments in its output. The inner while loop doesn't eat up all text to the end of file. Please note the putback(c) statement. Without that the newline would not appear.
If it doesn't work the same for wifstream, it would be very strange except for one reason: when the opened text file is not saved as 16bit char and the \n char ends up in the wrong byte...
You could open the stream in binary mode:
std::wifstream stream(filename, std::ios::binary);
You'll lose any formatting operations provided my the stream if you do this.
The other option is to read the entire stream into a string and then process the string:
std::wostringstream ss;
ss << filestream.rdbuf();
OF course, getting the string from the ostringstream rquires an additional copy of the string, so you could consider changing this at some point to use a custom stream if you feel adventurous.
EDIT: someone else mention istreambuf_iterator, which is probably a better way of doing it than reading the whole stream into a string.
Wrap the stream (or its buffer, specifically) in a std::streambuf_iterator? That should ignore all formatting, and also give you a nice iterator interface.
Alternatively, a much more efficient, and fool-proof, approach might to just use the Win32 API (or Boost) to memory-map the file. Then you can traverse it using plain pointers, and you're guaranteed that nothing will be skipped or converted by the runtime.
You could just Wrap the stream in a std::streambuf_iterator to get data with all whitespaces and newlines like this .
/*Open the stream in default mode.*/
std::ifstream myfile("myfile.txt");
if(myfile.good()) {
/*Read data using streambuffer iterators.*/
vector<char> buf((std::istreambuf_iterator<char>(myfile)), (std::istreambuf_iterator<char>()));
/*str_buf holds all the data including whitespaces and newline .*/
string str_buf(buf.begin(),buf.end());
myfile.close();
}
By default, this skipws flag is already set on the ifstream object, so we must disable it. The ifstream object has these default flags because of std::basic_ios::init, called on every new ios_base object (more details). Any of the following would work:
in_stream.unsetf(std::ios_base::skipws);
in_stream >> std::noskipws; // Using the extraction operator, same as below
std::noskipws(in_stream); // Explicitly calling noskipws instead of using operator>>
Other flags are listed on cpp reference.
The stream extractors behave the same and skip whitespace.
If you want to read every byte, you can use the unformatted input functions, like stream.get(c).
Why not simply use getline ?
You will get all the whitespaces, and while you won't get the end of lines characters, you will still know where they lie :)
Just Use getline.
while (getline(input,current))
{
cout<<current<<"\n";
}
I ended up just cracking open the Windows API and using it to read the whole file into a buffer first, and then reading that buffer character by character. Thanks guys.