Related
I want to read an entire line of a file character by character using gzgetc and stop when the newline is encountered. I know there is a function to grab the entire line but I would like to try to do it this way first. I tried:
Int c;
do {
c = gzgetc((gzFile) fp);
cout << c;
} while (c != '\n');
The result was an infinite loop. I tried adding (char) before c, still the same result. What am I doing wrong? The data file I am trying to read is encoded in base64 and I want to read in each token separated by space. Some of the lines are variable length and have a mixture of encoded and not encoded data which I set up an algorithm for I just need to know how to stop at newline.
You need to also check for gzgetc() returning -1, which indicates an error or end of file, and exiting the loop in that case. Your infinite loop is likely due to one of those.
I've got a slight problem. It appears that for some reason my function, when counting the size of a .txt file, counts a newline as it was two chars instead of one. Here's the function:
#define IN_FILE "in_mat.txt"
#define IN_BUF
#ifdef IN_BUF
void inBuf(char *(&b)){
streampos size;
ifstream f(IN_FILE, ios::in);
f.seekg(0,ios::end);
size=f.tellg();
b=new char[size];
f.seekg(0, ios::beg);
f.read(b, size);
f.close();
}
#endif
And here's the read file:
2 2
1 0
0 1
2 2
i 0
0 -i
2 2
0 1
-1 0
2 2
0 i
i 0
Earlier, i've put some couts, and it appears, that size=60, while the actual size is 49 (checked it), and the count of newlines in the file is 11, so exactly 60-49. Could somebody help me with that?
To add to the other answers, if you want to read special characters such as newline characters, you should open your file in binary mode, not text mode.
ifstream f(IN_FILE, ios::in | ios::binary);
If you don't open the file in binary mode, the actual characters that make up the '\n' are translated by the runtime to a single character (namely '\n'). So in text mode, you don't get the "real" version of the file in terms of all of the actual characters that the file consists of.
In addition, functions such as seekg() and tellg() will not work as expected with a file opened in text mode, or at the very least, will give you "wrong results" (actually not wrong to the functions themselves, but wrong if you're writing a program that tries to "hone in" on a position within the file). Again, the newline (and EOF) translation that is done under the hood by the runtime gets in the way of these functions working as you would expect them to.
On the other hand, a file opened in binary mode allows these functions to work as expected -- no translation of newline, or EOF -- whatever the individual bytes that makes up the file contents are, that is what you get.
The next thing you need to determine is whether it is a Unix text file or a Windows text file. Depending on which one it is, the line endings will be different.
Windows uses "\r\n" to return to the beginning of the line ('\r') and begin a new one ('\n').
To remove them from your count you have to read the whole file and count the number of '\r's.
Windows stores newlines as two characters: '\r\n', known as carriage return and line feed. That's why it's counted twice: there are actually two characters to be counted.
I am assuming that you are running on Windows. If not, disregard my answer below.
Windows stores new line characters in text files as two characters (CR LF or '\r' '\n'). So, seeking to the end of the file and calling tellg() will return the binary size of the file (60), not the text size (49).
In order to get the correct text size (49), one solution would be to count each new line character (11) and subtract that number from the total byte size.
Specifically I'm interested in istream& getline ( istream& is, string& str );. Is there an option to the ifstream constructor to tell it to convert all newline encodings to '\n' under the hood? I want to be able to call getline and have it gracefully handle all line endings.
Update: To clarify, I want to be able to write code that compiles almost anywhere, and will take input from almost anywhere. Including the rare files that have '\r' without '\n'. Minimizing inconvenience for any users of the software.
It's easy to workaround the issue, but I'm still curious as to the right way, in the standard, to flexibly handle all text file formats.
getline reads in a full line, up to a '\n', into a string. The '\n' is consumed from the stream, but getline doesn't include it in the string. That's fine so far, but there might be a '\r' just before the '\n' that gets included into the string.
There are three types of line endings seen in text files:
'\n' is the conventional ending on Unix machines, '\r' was (I think) used on old Mac operating systems, and Windows uses a pair, '\r' following by '\n'.
The problem is that getline leaves the '\r' on the end of the string.
ifstream f("a_text_file_of_unknown_origin");
string line;
getline(f, line);
if(!f.fail()) { // a non-empty line was read
// BUT, there might be an '\r' at the end now.
}
Edit Thanks to Neil for pointing out that f.good() isn't what I wanted. !f.fail() is what I want.
I can remove it manually myself (see edit of this question), which is easy for the Windows text files. But I'm worried that somebody will feed in a file containing only '\r'. In that case, I presume getline will consume the whole file, thinking that it is a single line!
.. and that's not even considering Unicode :-)
.. maybe Boost has a nice way to consume one line at a time from any text-file type?
Edit I'm using this, to handle the Windows files, but I still feel I shouldn't have to! And this won't fork for the '\r'-only files.
if(!line.empty() && *line.rbegin() == '\r') {
line.erase( line.length()-1, 1);
}
As Neil pointed out, "the C++ runtime should deal correctly with whatever the line ending convention is for your particular platform."
However, people do move text files between different platforms, so that is not good enough. Here is a function that handles all three line endings ("\r", "\n" and "\r\n"):
std::istream& safeGetline(std::istream& is, std::string& t)
{
t.clear();
// The characters in the stream are read one-by-one using a std::streambuf.
// That is faster than reading them one-by-one using the std::istream.
// Code that uses streambuf this way must be guarded by a sentry object.
// The sentry object performs various tasks,
// such as thread synchronization and updating the stream state.
std::istream::sentry se(is, true);
std::streambuf* sb = is.rdbuf();
for(;;) {
int c = sb->sbumpc();
switch (c) {
case '\n':
return is;
case '\r':
if(sb->sgetc() == '\n')
sb->sbumpc();
return is;
case std::streambuf::traits_type::eof():
// Also handle the case when the last line has no line ending
if(t.empty())
is.setstate(std::ios::eofbit);
return is;
default:
t += (char)c;
}
}
}
And here is a test program:
int main()
{
std::string path = ... // insert path to test file here
std::ifstream ifs(path.c_str());
if(!ifs) {
std::cout << "Failed to open the file." << std::endl;
return EXIT_FAILURE;
}
int n = 0;
std::string t;
while(!safeGetline(ifs, t).eof())
++n;
std::cout << "The file contains " << n << " lines." << std::endl;
return EXIT_SUCCESS;
}
Are you reading the file in BINARY or in TEXT mode? In TEXT mode the pair carriage return/line feed, CRLF, is interpreted as TEXT end of line, or end of line character, but in BINARY you fetch only ONE byte at a time, which means that either character MUST be ignored and left in the buffer to be fetched as another byte! Carriage return means, in the typewriter, that the typewriter car, where the printing arm lies in, has reached the right edge of the paper and is returned to the left edge. This is a very mechanical model, that of the mechanical typewriter. Then the line feed means that the paper roll is rotated a little bit up so the paper is in position to begin another line of typing. As fas as I remember one of the low digits in ASCII means move to the right one character without typing, the dead char, and of course \b means backspace: move the car one character back. That way you can add special effects, like underlying (type underscore), strikethrough (type minus), approximate different accents, cancel out (type X), without needing an extended keyboard, just by adjusting the position of the car along the line before entering the line feed. So you can use byte sized ASCII voltages to automatically control a typewriter without a computer in between. When the automatic typewriter is introduced, AUTOMATIC means that once you reach the farthest edge of the paper, the car is returned to the left AND the line feed applied, that is, the car is assumed to be returned automatically as the roll moves up! So you do not need both control characters, only one, the \n, new line, or line feed.
This has nothing to do with programming but ASCII is older and HEY! looks like some people were not thinking when they begun doing text things! The UNIX platform assumes an electrical automatic typemachine; the Windows model is more complete and allows for control of mechanical machines, though some control characters become less and less useful in computers, like the bell character, 0x07 if I remember well... Some forgotten texts must have been originally captured with control characters for electrically controlled typewriters and it perpetuated the model...
Actually the correct variation would be to just include the \r, line feed, the carriage return being unnecessary, that is, automatic, hence:
char c;
ifstream is;
is.open("",ios::binary);
...
is.getline(buffer, bufsize, '\r');
//ignore following \n or restore the buffer data
if ((c=is.get())!='\n') is.rdbuf()->sputbackc(c);
...
would be the most correct way to handle all types of files. Note however that \n in TEXT mode is actually the byte pair 0x0d 0x0a, but 0x0d IS just \r: \n includes \r in TEXT mode but not in BINARY, so \n and \r\n are equivalent... or should be. This is a very basic industry confusion actually, typical industry inertia, as the convention is to speak of CRLF, in ALL platforms, then fall into different binary interpretations. Strictly speaking, files including ONLY 0x0d (carriage return) as being \n (CRLF or line feed), are malformed in TEXT mode (typewritter machine: just return the car and strikethrough everything...), and are a non-line oriented binary format (either \r or \r\n meaning line oriented) so you are not supposed to read as text! The code ought to fail maybe with some user message. This does not depend on the OS only, but also on the C library implementation, adding to the confusion and possible variations... (particularly for transparent UNICODE translation layers adding another point of articulation for confusing variations).
The problem with the previous code snippet (mechanical typewriter) is that it is very inefficient if there are no \n characters after \r (automatic typewriter text). Then it also assumes BINARY mode where the C library is forced to ignore text interpretations (locale) and give away the sheer bytes. There should be no difference in the actual text characters between both modes, only in the control characters, so generally speaking reading BINARY is better than TEXT mode. This solution is efficient for BINARY mode typical Windows OS text files independently of C library variations, and inefficient for other platform text formats (including web translations into text). If you care about efficiency, the way to go is to use a function pointer, make a test for \r vs \r\n line controls however way you like, then select the best getline user-code into the pointer and invoke it from it.
Incidentally I remember I found some \r\r\n text files too... which translates into double line text just as is still required by some printed text consumers.
The C++ runtime should deal correctly with whatever the endline convention is for your particular platform. Specifically, this code should work on all platforms:
#include <string>
#include <iostream>
using namespace std;
int main() {
string line;
while( getline( cin, line ) ) {
cout << line << endl;
}
}
Of course, if you are dealing with files from another platform, all bets are off.
As the two most common platforms (Linux and Windows) both terminate lines with a newline character, with Windows preceding it with a carriage return,, you can examine the last character of the line string in the above code to see if it is \r and if so remove it before doing your application-specific processing.
For example, you could provide yourself with a getline style function that looks something like this (not tested, use of indexes, substr etc for pedagogical purposes only):
ostream & safegetline( ostream & os, string & line ) {
string myline;
if ( getline( os, myline ) ) {
if ( myline.size() && myline[myline.size()-1] == '\r' ) {
line = myline.substr( 0, myline.size() - 1 );
}
else {
line = myline;
}
}
return os;
}
One solution would be to first search and replace all line endings to '\n' - just like e.g. Git does by default.
Other than writing your own custom handler or using an external library, you are out of luck. The easiest thing to do is to check to make sure line[line.length() - 1] is not '\r'. On Linux, this is superfluous as most lines will end up with '\n', meaning you'll lose a fair bit of time if this is in a loop. On Windows, this is also superfluous. However, what about classic Mac files which end in '\r'? std::getline would not work for those files on Linux or Windows because '\n' and '\r' '\n' both end with '\n', eliminating the need to check for '\r'. Obviously such a task that works with those files would not work well. Of course, then there exist the numerous EBCDIC systems, something that most libraries won't dare tackle.
Checking for '\r' is probably the best solution to your problem. Reading in binary mode would allow you to check for all three common line endings ('\r', '\r\n' and '\n'). If you only care about Linux and Windows as old-style Mac line endings shouldn't be around for much longer, check for '\n' only and remove the trailing '\r' character.
Unfortunately the accepted solution does not behave exactly like std::getline(). To obtain that behavior (to my tests), the following change is necessary:
std::istream& safeGetline(std::istream& is, std::string& t)
{
t.clear();
// The characters in the stream are read one-by-one using a std::streambuf.
// That is faster than reading them one-by-one using the std::istream.
// Code that uses streambuf this way must be guarded by a sentry object.
// The sentry object performs various tasks,
// such as thread synchronization and updating the stream state.
std::istream::sentry se(is, true);
std::streambuf* sb = is.rdbuf();
for(;;) {
int c = sb->sbumpc();
switch (c) {
case '\n':
return is;
case '\r':
if(sb->sgetc() == '\n')
sb->sbumpc();
return is;
case std::streambuf::traits_type::eof():
is.setstate(std::ios::eofbit); //
if(t.empty()) // <== change here
is.setstate(std::ios::failbit); //
return is;
default:
t += (char)c;
}
}
}
According to https://en.cppreference.com/w/cpp/string/basic_string/getline:
Extracts characters from input and appends them to str until one of the following occurs (checked in the order listed)
end-of-file condition on input, in which case, getline sets eofbit.
the next available input character is delim, as tested by Traits::eq(c, delim), in which case the delimiter character is extracted from input, but is not appended to str.
str.max_size() characters have been stored, in which case getline sets failbit and returns.
If no characters were extracted for whatever reason (not even the discarded delimiter), getline sets failbit and returns.
If it is known how many items/numbers each line has, one could read one line with e.g. 4 numbers as
string num;
is >> num >> num >> num >> num;
This also works with other line endings.
I get a text file from aix and try to process it on windows
but some lines contains strange character like , in ultraedit , it is displayed like
when fgets function encounter the line, it raise a ferror and stop at drawing. Then it refuse to continue even if I force to run fgets again on after meets the line.
The hexa code of the character is 1A
The explanation of this character in ASCII table is substitute character, which is used to replace the character that cannot be represented on the device.
Does that means
I have a specific character of AIX and there is no way to process it on windows.
Does this happens only in case of a cross plateform file ?
Thanks!
There are several issues.
If you use fopen with the "r" mode, the file will be opened in text mode and then the ASCII character 0x1b will be interpreted as end of file character. Furthermore if your file comes from aix, the line endings are certainly "\n" (0x10) instead of "\r\n" (0x13 0x10) on Windows, and fgets regognizes only "\r\n" as line endings.
You need to implement your own fgets like function by reading the file character by character with the fgetc function, and you must fopen the aix file with "rb" mode instead of the "r" mode.
Your new fgets like function should be no more than 5 or 6 lines long.
I understand EOF and EOL but when I was reading this question (second part of answer) and i got my concepts broken :
Specially the para :
It won't stop taking input until it finds the end of file(cin uses
stdin, which is treated very much like a file)
so i want to know when we do some thing like in c++ under windows :
std::cin>>int_var; , and we press enter , this end the input but according to reference link it should only stop taking input after hitting ctrl+z.
So i would love to know how std::*stream deal with EOF and EOL.
Second part:
please have a look at this example :
std::cin.getline(char_array_of_size_256 ,256);
cin.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
cout << "artist is " << artist << endl;
If i remove std::cin.ignore() it simply stops taking input (which is known case) but when i keep it , it waits for a new input which is ended by '\n' . But it should simply clear up stream rather then waiting for any new input ending-up with '\n'.
Thanks for giving you time)
End-of-line and end-of-file are very different concepts.
End-of-line is really just another input character (or character sequence) that can appear anywhere in an input stream. If you're reading input one character at a time from a text stream, end-of-line simply means that you'll see a new-line ('\n') character. Some input routines treat this character specially; for example, it tells getline to stop reading. (Other routines treat ' ' specially; there's no fundamental difference.)
Different operating systems use different conventions for marking the end of a line. On Linux and other Unix-like systems, the end of a line in a file is marked with a single ASCII linefeed (LF, '\n') character. When reading from a keyboard, both LF and CR are typically mapped to '\n' (try typing either Enter, Control-J, or Control-M). On Windows, the end of a line in a file is marked with a CR-LF pair (\r\n). The C and C++ I/O systems (or the lower-level software they operate on top of) map all these markers to a single '\n' character, so your program doesn't have to worry about all the possible variations.
End-of-file is not a character, it's a condition that says there are no more characters available to be read. Different things can trigger this condition. When you're reading from a disk file, it's just the physical end of the file. When you're reading from a keyboard on Windows, control-Z denotes end-of-file; on Unix/Linux, it's typically control-D (though it can be configured differently).
(You'll usually have an end-of-line (character sequence) just before end-of-file, but not always; input can sometimes end in an unterminated line, on some systems.)
Different input routines have different ways of indicating that they've seen an end-of-file condition. Read the documentation for each one for the details.
As for EOF, that's a macro defined in <stdio.h> or <cstdio>. It expands to a negative integer constant (typically -1) that's returned by some functions to indicate that they've reached an end-of-file condition.
EDIT: For example, suppose you're reading from a text file containing two lines:
one
two
Let's say you're using C's getchar(), getc(), or fgetc() function to read one character at a time. The values returned on successive calls will be:
'o', 'n', 'e', '\n', 't', 'w', 'o', '\n', EOF
Or, in numeric form (on a typical system):
111, 110, 101, 10, 116, 119, 111, 10, -1
Each '\n', or 10 (0x0a) is a new-line character read from the file. The final -1 is the value of EOF; this isn't a character, but an indication that there are no more characters to be read.
Higher-level input routines, like C's fgets() and C++'s std::cin >> s or std::getline(std::cin, s), are built on top of this mechanism.
First "part"
so i want to know when we do some thing like in c++ under windows : std::cin>>int_var; , and we press enter , this end the input but according to reference link it should only stop taking input after hitting ctrl+z.
No, you're confusing formatted input operations with stream iterators. The following will use the formatted input operation (operator>>) repeatedly until the end of file is reached because the "end iterator" represents the end of the stream.
std::vector<int> integers;
std::copy(
std::istream_iterator<int>(std::cin),
std::istream_iterator<int>(),
std::back_inserter(integers));
If you use the following:
int i = 0;
std::cin >> i;
in an interactive shell (e.g. in console mode), std::cin will block on user input which is acquired line by line. So, if no data (or only white space) is available, this operation will actually force the user to type a line of input and press the enter key.
However,
int i = 0;
int j = 0;
std::cin >> i >> j;
may block on one or two lines of input, depending on what the user types. In particular, if the user types
1<space>2<enter>
then the two input operations will be applies using the same line of input.
Second "part"
Considering the code snippet:
std::cin.getline(char_array_of_size_256 ,256);
cin.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
cout << "artist is " << artist << endl;
If the line contains 255 or less lines of character data, std::cin.getline() will consume the end-of-line character. Thus, the second line will consume all characters until the next line is completed. If you want to capture only the current line and ignore all characters past 256, I suggest you use something like:
std::cin.getline(char_array_of_size_256 ,256);
if (std::cin.gcount() == 256) {
cin.ignore(std::numeric_limits<std::streamsize>::max(), '\n');
}
cout << "artist is " << artist << endl;
On the second part:
When the linked answer said "read into a string", I guess they meant
std::string s;
std::getline(std::cin, s);
which always reads the entire line into the string s (while setting s to the proper size).
That way there is nothing left over from the input line to clean up.