I want to write a function equivalent to the Linux tail -n command in C++. While, I parsed over the data of that file line-by-line thereby incrementing the line count, if the file size gets really big(~gigabytes), this method will take a lot of time! Is there a better approach or a data structure to implement this function?
Here are my 2 methods:
int File::countlines()
{
int lineCount = 0;
string str;
if (file)
{
while (getline(file, str))
{
lineCount += 1;
}
}
return lineCount;
}
void File::printlines()
{
int lineCount = 0;
string line;
if (file)
{
lineCount = countlines();
file.clear();
file.seekg(ios::beg);
if (lineCount <= 10)
{
while (getline(file, line))
{
cout << line << endl;
}
}
else
{
int position = lineCount - 10;
while (position--)
{
getline(file, line);
}
while (getline(file, line))
{
cout << line << endl;
}
}
}
}
This method is time consuming if the file size increases, so I want to either replace it with another data structure, or write a more efficient code.
One of the things that is slowing down your program is reading the file twice, so you could keep the last n EOL positions (n=10 in your program) and the most convenient data structure is a circular buffer but this isn't provided by the standard library as far as I know (boost has one). It can be implemented by an std::vector with size n, with an index where a modulo of n is done after incrementing.
With that circular buffer, you can jump immediately to the lowest offset (next one if buffer is full) in the file and print the needed lines.
When I've done this, I've done a generous estimate of the maximum length of a line (e.g., one kilobyte), seeked to that distance from the end, and started reading lines into a circular buffer until the end of the file.
In nearly every case, you get more than n lines, so you just print out the contents of the circular buffer, and you're done. Note, however, that you do need to assure that you read more than n lines, not just n lines. The first line you read will usually only be a partial line, so if you read exactly n lines, the first would probably be only a partial line.
On rare occasion, you haven't gotten the required number of lines, so you seek back twice as far (or other factor of your choice), and restart. If you want to get really fancy, you can extrapolate the number of lines you'll need based on the average length of the lines you did read (but honestly, this is such a rare situation it's not worth a lot of work to optimize it).
This normally works essentially instantly, regardless of file size. I suppose (in theory) for a file with incredibly long lines, it would get slower, but if that's the case, the user has probably made a mistake, and tried to tail something that isn't a text file (which is generally useless anyway).
Related
I'm trying to write my own vocabulary with a test for my little brother, but I have a problem when I want to read data from file into two arrays - first with English words, and second with Polish words. File looks alike
black - czarny
red - czerwony etc.
And my function:
void VOC::readout()
{
fstream file;
VOC *arr = new VOC;
string line;
file.open("slowka.txt");
if(file.good())
{
int i=0;
while(!file.eof())
{
getline(file, line);
size_t pos = line.find(" - ");
int position = static_cast<int>(pos);
file>>arr[i].en;
file>>arr[i].pl;
++i;
}
}
}
I thought it could be a good idea to insert a line into first array until the function finds " - ", and after that insert the rest of line into second array, but I have some problems with that. Could someone help me? I know I can solve it by using std::vector but I care to do that by using arrays.
If you insist on using plain arrays, you'll first have to count the number of lines in your file and then allocate enough memory. Arrays -- unlike std::vector objects -- won't grow automatically but have a fixed size.
That being said, note that using !file.eof() is not the best way to read a stream until the end is reached. You can use the simpler
std::string line;
while (std::getline(file, line)) {
// ...
}
idiom instead, which also takes care of error conditions. See this question (and corresponding answers) for more information on that.
I'm just trying to use the fstream library and I wanna read a given row.
I thought this, but I don't know if is the most efficient way.
#include <iostream>
#include <fstream>
using namespace std;
int main(){
int x;
fstream input2;
string line;
int countLine = 0;
input2.open("theinput.txt");
if(input2.is_open());
while(getline(input2,line)){
countLine++;
if (countLine==1){ //1 is the lane I want to read.
cout<<line<<endl;
}
}
}
}
Is there another way?
This does not appear to be the most efficient code, no.
In particular, you're currently reading the entire input file even though you only care about one line of the file. Unfortunately, doing a good job of skipping a line is somewhat difficult. Quite a few people recommend using code like:
your_stream.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
...for this job. This can work, but has a couple of shortcomings. First and foremost, if you try to use it on a non-text file (especially one that doesn't contain new-lines) it can waste inordinate amounts of time reading an entire huge file, long after you've read enough that you would normally realize that there must be a problem. For example, if you're reading a "line", that's a pretty good indication that you're expecting a text file, and you can pretty easily set a much lower limit on how long that first line could reasonably be, such as (say) a megabyte, and usually quite a lot less than that.
You also usually want to detect whether it stopped reading because it reached that maximum, or because it got to the end of the line. Skipping a line "succeeded" only if a new-line was encountered before reaching the specified maximum. To do that, you can use gcount() to compare against the maximum you specified. If you stopped reading because you reached the specified maximum, you typically want to stop processing that file (and log the error, print out an error message, etc.)
With that in mind, we might write code like this:
bool skip_line(std::istream &in) {
size_t max = 0xfffff;
in.ignore(max, '\n');
return in.gcount() < max;
}
Depending on the situation, you might prefer to pass the maximum line size as a parameter (probably with a default) instead:
bool skip_line(std::istream &in, size_t max = 0xfffff) {
// skip definition of `max`, remainder identical
With this, you can skip up to a megabyte by default, but if you want to specify a different maximum, you can do so quite easily.
Either way, with that defined, the remainder becomes fairly trivial, something like this:
int main(){
std::ifstream in("theinput.txt");
if (!skip_line(in)) {
std::cerr << "Error reading file\n";
return EXIT_FAILURE;
}
// copy the second line:
std::string line;
if (std::getline(in, line))
std::cout << line;
}
Of course, if you want to skip more than one line, you can do that pretty easily as well by putting the call to skip_line in a loop--but note that you still usually want to test the result from it, and break the loop (and log the error) if it fails. You don't usually want something like:
for (int i=0; i<lines_to_skip; i++)
skip_line(in);
With this, you'd lose one of the basic benefits of assuring that your input really is what you expected, and you're not producing garbage.
I think you can condense your code to this. if (input) is sufficient to check for failure.
#include <iostream>
#include <fstream>
#include <limits>
int main()
{
std::ifstream input("file.txt");
int row = 5;
int count = 0;
if (input)
{
while (count++ < row) input.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
std::string line;
std::getline(input, line);
std::cout << line;
}
}
I'm really stuck with this problem I'm having for reading rows and columns from a text file. We're using text files that our prof gave us. I have the functionality running so when the user in puts "numrows (file)" the number of rows in that file prints out.
However, every time I enter the text files, it's giving me 19 for both. The first text file only has 4 rows and the other one has 7. I know my logic is wrong, but I have no idea how to fix it.
Here's what I have for the numrows function:
int numrows(string line) {
ifstream ifs;
int i;
int row = 0;
int array [10] = {0};
while (ifs.good()) {
while (getline(ifs, line)) {
istringstream stream(line);
row = 0;
while(stream >>i) {
array[row] = i;
row++;
}
}
}
}
and here's the numcols:
int numcols(string line) {
int col = 0;
int i;
int arrayA[10] = {0};
ifstream ifs;
while (ifs.good()) {
istringstream streamA(line);
col = 0;
while (streamA >>i){
arrayA[col] = i;
col++;
}
}
}
edit: #chris yes, I wasn't sure what value to return as well. Here's my main:
int main() {
string fname, line;
ifstream ifs;
cout << "---- Enter a file name : ";
while (getline(cin, fname)) { // Ctrl-Z/D to quit!
// tries to open the file whose name is in string fname
ifs.open(fname.c_str());
if(fname.substr(0,8)=="numrows ") {
line.clear();
for (int i = 8; i<fname.length(); i++) {
line = line+fname[i];
}
cout << numrows (line) << endl;
ifs.close();
}
}
return 0;
}
This problem can be more easily solved by opening the text file as an ifstream, and then using std::get to process your input.
You can try for comparison against '\n' as the end of line character, and implement a pair of counters, one for columns on a line, the other for lines.
If you have variable length columns, you might want to store the values of (numColumns in a line) in a std::vector<int>, using myVector.push_back(numColumns) or similar.
Both links are to the cplusplus.com/reference section, which can provide a large amount of information about C++ and the STL.
Edited-in overview of possible workflow
You want one program, which will take a filename, and an 'operation', in this case "numrows" or "numcols". As such, your first steps are to find out the filename, and operation.
Your current implementation of this (in your question, after editing) won't work. Using cin should however be fine. Place this earlier in your main(), before opening a file.
Use substr like you have, or alternatively, search for a space character. Assume that the input after this is your filename, and the input in the first section is your operation. Store these values.
After this, try to open your file. If the file opens successfully, continue. If it won't open, then complain to the user for a bad input, and go back to the beginning, and ask again.
Once you have your file successfully open, check which type of calculation you want to run. Counting a number of rows is fairly easy - you can go through the file one character at a time, and count the number that are equal to '\n', the line-end character. Some files might use carriage-returns, line-feeds, etc - these have different characters, but are both a) unlikely to be what you have and b) easily looked up!
A number of columns is more complicated, because your rows might not all have the same number of columns. If your input is 1 25 21 abs 3k, do you want the value to be 5? If so, you can count the number of space characters on the line and add one. If instead, you want a value of 14 (each character and each space), then just count the characters based on the number of times you call get() before reaching a '\n' character. The use of a vector as explained below to store these values might be of interest.
Having calculated these two values (or value and set of values), you can output based on the value of your 'operation' variable. For example,
if (storedOperationName == "numcols") {
cout<< "The number of values in each column is " << numColsVal << endl;
}
If you have a vector of column values, you could output all of them, using
for (int pos = 0; pos < numColsVal.size(); pos++) {
cout<< numColsVal[pos] << " ";
}
Following all of this, you can return a value from your main() of 0, or you can just end the program (C++ now considers no return value from main to a be a return of 0), or you can ask for another filename, and repeat until some other method is used to end the program.
Further details
std::get() with no arguments will return the next character of an ifstream, using the example code format
std::ifstream myFileStream;
myFileStream.open("myFileName.txt");
nextCharacter = myFileStream.get(); // You should, before this, implement a loop.
// A possible loop condition might include something like `while myFileStream.good()`
// See the linked page on std::get()
if (nextCharacter == '\n')
{ // You have a line break here }
You could use this type of structure, along with a pair of counters as described earlier, to count the number of characters on a line, and the number of lines before the EOF (end of file).
If you want to store the number of characters on a line, for each line, you could use
std::vector<int> charPerLine;
int numberOfCharactersOnThisLine = 0;
while (...)
{
numberOfCharactersOnThisLine = 0
// Other parts of the loop here, including a numberOfCharactersOnThisLine++; statement
if (endOfLineCondition)
{
charPerLine.push_back(numberOfCharactersOnThisLine); // This stores the value in the vector
}
}
You should #include <vector> and either specific std:: before, or use a using namespace std; statement near the top. People will advise against using namespaces like this, but it can be convenient (which is also a good reason to avoid it, sort of!)
I'm working on a C++ program (C++ 98). It reads a text file with lots of lines (10000 lines). These are tab separated values and then I parse it into Vector of Vector objects. However It seems to work for some files (Smaller) but One of my files gives me the following error (this file has 10000 lines and it's 90MB). I'm guessing this is a memory related issue?
Can you please help me?
Error
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Abort
Code
void AppManager::go(string customerFile) {
vector<vector<string> > vals = fileReader(customerFile);
for (unsigned int i = 0; i < vals.size();i++){
cout << "New One\n\n";
for (unsigned int j = 0; j < vals[i].size(); j++){
cout << vals[i][j] << endl;
}
cout << "End New One\n\n";
}
}
vector<vector<string> > AppManager::fileReader(string fileName) {
string line;
vector<vector<string> > values;
ifstream inputFile(fileName.c_str());
if (inputFile.is_open()) {
while (getline(inputFile,line)) {
std::istringstream iss(line);
std::string val;
vector<string> tmp;
while(std::getline(iss, val, '\t')) {
tmp.push_back(val);
}
values.push_back(tmp);
}
inputFile.close();
}
else {
throw string("Error reading the file '" + fileName + "'");
}
return values;
}
There's nothing wrong with your code, you're simply running on a platform likely with small memory limits, likely an old compiler and likely an old C++ library. It all conspires against you. You'll have to micro-optimize :(
Here's what you can do, starting with lowest hanging fruit first:
Do a dry run through the file, just counting the lines. Then values.resize(numberOfLines) , seek to the beginning and only then read the values. Of course you won't be using values.push_back anymore, merely values[lineNumber] = tmp. Resizing the values vector as you add to it may more than double the amount of memory your process needs on a temporary basis.
At the end of the line, do tmp.resize(tmp.size() - it'll shrink the vector to just fit the data.
You can reduce overheads in the existing code by storing all the values in one vector.
If each line has a different number of elements, but you access them sequentially later on, you can store an empty string as an internal delimiter, it may have lower overhead than the vector.
If each line has same number of values, then splitting them by lines adds unnecessary overhead - you know the index of the first value in each line, it's simply lineNumber * valuesPerLine, where first line has number 0.
Memory-map the file. Store the beginning and end of each word in a structure element of a vector, perhaps with a line number as well if you need it split up in lines.
I have numerical text data lines ranging between 1mb - 150 mb in size, i need to write lines of numbers related to heights, for example: heights=4 , new text must include lines: 1,5,9,13,17,21.... consequentially.
i have been trying to find a way to do this for a while now, tried using a list instead of vector which ended up with compilation errors.
I have cleaned up the code as advised. It now writes all lines sample2 text, all done here. Thank you all
I am open to method change as long as it delivers what i need, Thank you for you time and help.
following is what i have so far:
#include <iostream>
#include <fstream>
#include <string>
#include <list>
#include <vector>
using namespace std;
int h,n,m;
int c=1;
int main () {
cout<< "Enter Number Of Heights: ";
cin>>h;
ifstream myfile_in ("C:\\sample.txt");
ofstream myfile_out ("C:\\sample2.txt");
string line;
std::string str;
vector <string> v;
if (myfile_in.is_open()) {
myfile_in >> noskipws;
int i=0;
int j=0;
while (std::getline(myfile_in, line)) {
v.push_back( line );
++n;
if (n-1==i) {
myfile_out<<v[i]<<endl;
i=i+h;
++j;
}
}
cout<<"Number of lines in text file: "<<n<<endl;
}
else cout << "Unable to open file(s) ";
cout<< "Reaching here, Writing one line"<<endl;
system("PAUSE");
return 0;
}
You need to use seekg to set the position at the beginning of the file, once you have read it (you have read it once, to count the lines (which I don't think you actually need, as this size is never used, at least in this piece of code)
And what is the point if the inner while? On each loop, you have
int i=1;
myfile_out<<v[i]; //Not writing to text
i=i+h;
So on each loop, i gets 1, so you output the element with index 1 all the time. Which is not the first element, as indices start from 0. So, once you put seekg or remove the first while, your program will start to crash.
So, make i start from 0. And get it out of the two while loops, right at the beginning of the if-statement.
Ah, the second while is also unnecessary. Leave just the first one.
EDIT:
Add
myfile_in.clear();
before seekg to clear the flags.
Also, your algorithm is wrong. You'll get seg fault, if h > 1, because you'll get out of range (of the vector). I'd advise to do it like this: read the file in the while, that counts the lines. And store each line in the vector. This way you'll be able to remove the second reading, seekg, clear, etc. Also, as you already store the content of the file into a vector, you'll NOT lose anything. Then just use for loop with step h.
Again edit, regarding your edit: no, it has nothing to do with any flags. The if, where you compare i==j is outside the while. Add it inside. Also, increment j outside the if. Or just remove j and use n-1 instead. Like
if ( n-1 == i )
Several things.
First you read the file completely, just to count the number of lines,
then you read it a second time to process it, building up an in memory
image in v. Why not just read it in the first time, and do everything
else on the in memory image? (v.size() will then give you the number
of lines, so you don't have to count them.)
And you never actually use the count anyway.
Second, once you've reached the end of file the first time, the
failbit is set; all further operations are no-ops, until it is reset.
If you have to read the file twice (say because you do away with v
completely), then you have to do myfile_in.clear() after the first
loop, but before seeking to the beginning.
You only test for is_open after having read the file once. This test
should be immediately after the open.
You also set noskipws, although you don't do any formatted input
which would be affected by it.
The final while is highly suspect. Because you haven't done the
clear, you probably never enter the loop, but if you did, you'd very
quickly start accessing out of bounds: after reading n lines, the size
of v will be n, but you read it with index i, which will be n * h.
Finally, you should explicitly close the output file and check for
errors after the close, just in case.
It's not clear to me what you're trying to do. If all you want to do is
insert h empty lines between each existing line, something like:
std::string separ( h + 1, '\n' );
std::string line;
while ( std::getline( myfile_in, line ) ) {
myfile_out << line << separ;
}
should do the trick. No need to store the complete input in memory.
(For that matter, you don't even have to write a program for this.
Something as simple a sed 's:$:\n\n\n\n:' < infile > outfile would do
the trick.)
EDIT:
Reading other responses, I gather that I may have misunderstood the
problem, and that he only wants to output every h-th line. If this is
the case:
std::string line;
while ( std::getline( myfile_in, line ) ) {
myfile_out << line << '\n';
for ( int count = h - 1; h > 0; -- h ) {
std::getline( myfile_in, line );
// or myfile_in.ignore( INT_MAX, '\n' );
}
}
But again, other tools seem more appropriate. (I'd follow thiton's
suggestion and use AWK.) Why write a program in a language you don't
know well when tools are already available to do the job.
If there is no absolutely compelling reason to do this in C++, you are using the wrong programming language for this. In awk, your whole program is:
{ if ( FNR % 4 == 1 ) print; }
Or, giving the whole command line e.g. in sh to filter lines 1,5,9,13,...:
awk '{ if ( FNR % 4 == 1 ) print; }' a.txt > b.txt