I understand that QString::split should be used to get a QStringList from a multiline QString. But if I have a file and I don't know if it comes from Mac, Windows or Unix, I'm not sure if QString.split("\n") would work well in all the cases. What is the best way to handle this situation?
If it's acceptable to remove blank lines, you can try:
QString.split(QRegExp("[\r\n]"),QString::SkipEmptyParts);
This splits the string whenever any of the newline character (either line feed or carriage return) is found. Any consecutive line breaks (e.g. \r\n\r\n or \n\n) will be considered multiple delimiters with empty parts between them, which will be skipped.
Emanuele Bezzi's answer misses a couple of points.
In most cases, a string read from a text file will have been read using a text stream, which automatically translates the OS's end-of-line representation to a single '\n' character. So if you're dealing with native text files, '\n' should be the only delimiter you need to worry about. For example, if your program is running on a Windows system, reading input in text mode, line endings will be marked in memory with single \n characters; you'll never see the "\r\n" pairs that exist in the file.
But sometimes you do need to deal with "foreign" text files.
Ideally, you should probably translate any such files to the local format before reading them, which avoids the issue. Only the translation utility needs to be aware of variant line endings; everything else just deals with text.
But that's not always possible; sometimes you might want your program to handle Windows text files when running on a POSIX system (Linux, UNIX, etc.), or vice versa.
A Windows-format text file on a POSIX system will appear to have an extra '\r' character at the end of each line.
A POSIX-format text file on a Windows system will appear to consist of one very long line with embedded '\n' characters.
The most general approach is to read the file in binary mode and deal with the line endings explicitly.
I'm not familiar with QString.split, but I suspect that this:
QString.split(QRegExp("[\r\n]"),QString::SkipEmptyParts);
will ignore empty lines, which will appear either as "\n\n" or as "\r\n\r\n", depending on the format. Empty lines are perfectly valid text data; you shouldn't ignore them unless you're certain that it makes sense to do so.
If you need to deal with text input delimited either by "\n", "\r\n", or "\r", then I think something like this:
QString.split(QRegExp("\n|\r\n|\r"));
would do the job. (Thanks to parsley72's comment for helping me with the regular expression syntax.)
Another point: you're probably not likely to encounter text files that use just '\r' to delimit lines. That's the format used by MacOS up to version 9. MaxOS X is based on UNIX, and it uses standard UNIX-style '\n' line endings (though it probably tolerates '\r' line endings as well).
Related
I need to create a program that reads strings from two different files and write these strings on a new file. The thing is, it must alternate both files, meaning that it should write a line from one file, and then one line from the other, and so on.
I'm having a problem with my code, it writes the first line of the first file, and then it writes all lines from the second file.
Anyone knows how to solve this problem?
do {
getline(archivo1, sLinea);
archivoS << sLinea << endl;
getline(archivo2, sLinea2);
archivoS << sLinea2 << endl;
} while (!archivo1.eof() && !archivo2.eof());
The code looks correct and should work under normal circumstances. This might be a problem with the encoding of the second file, where the newline characters are not being recognised as such on your platform, which could result in the entire second file being interpreted as a single line by the C++ standard library.
Windows (CR+LF), Unix/Linux (LF), and Mac (CR) each have different conventions for newlines. Search about the carriage return and line feed characters across platforms to learn more about this topic.
To identify if this is the issue, try running the code on two separate copies of the first file to see if it produces the expected output?
If newline encoding is your issue, you will either need to convert the second file to use your platform's newline encoding (you can use a tool like Notepad++ to easily do this) or incorporate logic which controls for this into your program.
Check your second file. In all likelihood it does not contain the line delimiter "\n" , per line. There may be only one at the end
Now I am quite confused about the end of line character I am working with c++ and I know that text files have a end of line marker which sets the limit for reading a line which a single shifing operator(>>).Data is read continously untill eol character does not apprears and while opening a file in text mode carriage return(CR) is converted into CRLF which is eol marker so if i add white spaces in my text then would it act as eol maker cause it does.
Now i created a normal file i.e. a file without .txt
eg
ifstream("test"); // No .txt
Now what is eol marker in this case
The ".txt" at the end of the filename is just a convention. It's just part of the filename.
It does not signify any magical property of the file, and it certainly doesn't change how the file is handled by your operating system kernel or file system driver.
So, in short, what difference is there? None.
I know that text files have a end of line marker which sets the limit for reading a line which a single shifing operator(>>)
That is incorrect.
Data is read continously untill eol character does not apprears
Also incorrect. Some operating systems (e.g. Windows IIRC) inject an EOF (not EOL!) character into the stream to signify to calling applications that there is no more data to read. Other operating systems don't even do that. But in neither case is there an actual EOF character at the end of the actual file.
while opening a file in text mode carriage return(CR) is converted into CRLF which is eol marker
That conversion may or may not happen and, either way, EOL is not EOF.
if i add white spaces in my text then would it act as eol maker cause it does.
That's a negative, star command.
I'm not sure where you're getting all this stuff from, but you've been heavily mistaught. I suggest a good, peer-reviewed, well-recommended book from Amazon about how computer operating systems work.
When reading strings in C++ using the extraction operator >>, the default is to skip spaces.
If you want the entire line verbatim, use std::getline.
A typical input loop is:
int main(void)
{
std::string text_from_file;
std::ifstream input_file("My_data.txt");
if (!input_file)
{
cerr << "Error opening My_data.txt for reading.\n";
return EXIT_FAILURE;
}
while (input_file >> text_from_file)
{
// Process the variable text_from_file.
}
return EXIT_SUCCESS;
}
A lot of old and mainframe operating systems required a record structure of all data files which, for text files, originated with a Hollerith (punch) card of 80 columns and was faithfully preserved through disk file records, magnetic tapes, output punch card decks, and line printer lines. No line ending was used because the record structure required that every record have 80 columns (and were typically filled with spaces). In later years (1960s+), having variable length records with an 80 column maximum became popular. Today, even OpenVMS still requires the file creator to specify a file format (sequential, indexed, or "stream") and record size (fixed, variable) where the maximum record size must be specified in advance.
In the modern era of computing (which effectively began with Unix) it is widely considered a bad idea to force a structure on data files. Any programmer is free to do that to themselves and there are plenty of record-oriented data formats like compiler/linker object files (.obj, .so, .o, .lib, .exe, etc.), and most media formats (.gif, .tiff, .flv, .mov, mp3, etc.)
For communicating text lines, the paradigm is to target a terminal or printer and for that, line endings should be indicated. Most operating systems environments (except MSDOS and Windows) use the \n character which is encoded in ASCII as a linefeed (ASCII 10) code. MSDOS and ilk use '\r\n' which are encoded as carriage return then linefeed (ASCII 13, 10). There are advantages and disadvantages to both schemes. But text files may also contain other controls, most commonly the ANSI escape sequences which control devices in specific ways:
clear the screen, either in part or all of it
eject a printer page, skip some lines, reverse feed, and other little-used features
establish a scrolling region
change the text color
selecting a font, text weight, page size, etc.
For these operations, line endings are not a concern.
Also, data files encoded in ASCII such as JSON and XML (especially HTML with embedded Javascript), might not have any line endings, especially when the data is obfuscated or compressed.
To answer your questions:
I am quite confused about the end of line character I am working with c++ and I know that text files have a end of line marker
Maybe. Maybe not. From a C or C++ program's viewpoint, writing \n indicates to the runtime environment the end of a line. What the system does with that varies by runtime operating environment. For Unix and Linux, no translation occurs (though writing to a terminal-like device converts to \r\n). In MSDOS, '\n' is translated to \r\n. In OpenVMS, '\n' is removed and that record's size is set. Reading does the inverse translation.
which sets the limit for reading a line which a single shifing operator(>>).
There is no such limit: A program can choose to read data byte-by-byte if it wants as well as ignore the line boundaries.
The "shifting operators" are overloaded for filestreams to input or output data but are not related to bit twiddling shifts. These operators were chosen for visual approximation of input/output and due to their low operator precedence.
Data is read continously untill eol character does not apprears
This bit is confusing: I think you meant until eol character appears, which is indeed how the line-oriented functions gets() and fgets() work.
and while opening a file in text mode carriage return(CR) is converted into CRLF which is eol marker so if i add white spaces in my text then would it act as eol maker cause it does.
Opening the file does not convert anything, but reading from a file might. However, no environment (that I know of) converts input to CR LF. MSDOS converts CR LF on input to \n.
Adding spaces has no effect on end of lines, end of file, or anything. Spaces are just data. However, the C++ streaming operations reading/writing numbers and some other datatypes use whitespace (a sequence of spaces, horizontal tabs, vertical tabs, form feed, and maybe some others) as a delimiter. This convenience feature may cause some confusion.
Now i created a normal file i.e. a file without .txt eg
ifstream("test"); \No .txt
Now what is eol marker in this case
The filename does not determine the file type. In fact, file.txt may not be a text file at all. Using a particular file extension is convenient for humans to communicate a file's purpose, but it is not obligatory.
What are difference between:
\r\n - Line feed followed by carriage return.
\n - Line feed.
\r - Carriage Return.
They are the line terminator used by different systems:
\r\n = Windows
\n = UNIX and Mac OS X
\r = Old Mac
You should use std::endl to abstract it, if you want to write one out to a file:
std::cout << "Hello World" << std::endl;
In general, an \r character moves the cursor to the beginning of the line, while an \n moves the cursor down one line. However, different platforms interpret this in different ways, leading to annoying compatibility issues, especially between Windows and UNIX. This is because Windows requires an \r\n to move down one line and move the cursor to the start of the line, whereas on UNIX a single \n suffices.
Also, obligatory Jeff Atwood link: http://www.codinghorror.com/blog/2010/01/the-great-newline-schism.html
Historical info
The terminology comes from typewriters. Back in the day, when people used typewriters to write, when you got to the end of a line you'd press a key on the typewriter that would mechanically return the carriage to the left side of the page and feed the page up a line. The terminology was adopted by computers and represented as the ascii control codes 0xa, for the linefeed, and 0xd for the carriage return. Different operating systems use them differently, which leads to problems when editing a text file written on a Unix machine on a Windows machine and vice-versa.
Pragmatic info
On Unix based machines in text files a newline is represented by the linefeed character, 0xa. On Windows both a linefeed and carriage return are used. For example when you write some code on Linux that has the following in it where the file was opened in text mode:
fprintf(f, "\n");
the underlying runtime will insert only a linefeed character 0xa to the file. On Windows it will translate the \n and insert 0xd0xa. Same code but different results depending on the operating system used. However this changes if the file is opened in binary mode on Windows. In this case the insertion is done literally and only a linefeed character is inserted. This means that the sequence you asked about \r\n will have a different representation on Windows if output to a binary or text stream. In the case of it being a text stream you'll see the following in the file: 0xd0xd0xa. If the file was in binary mode then you'll see: 0xd0xa.
Because of the differences in how operating systems represent newlines in text files text editors have had to evolve to deal with them, although some, like Notepad, still don't know what to do. So in general if you're working on Windows and you're given a text file that was originally written on a Unix machine it's not a good idea to edit it in Notepad because it will insert the Windows style carriage return linefeed (0xd0xa) into the file when you really want a 0xa. This can cause problems for programs running on old Unix machines that read text files as input.
See http://en.wikipedia.org/wiki/Newline
Different operating systems have different conventions; Windows uses \r\n, Mac uses \r, and UNIX uses \n.
\r
This sends the cursor to the beginning column of the display
\n
This moves the cursor to the new line of the display, but the cursor stays in the same column as the previous line.
\r\n
Combine 1 and 2. The cursor is moved to the new line, and it is also moved to the first column of the display.
Some compilers prints both a new line and carriage return when you specify only \n.
I have a multi-line ASCII string coming from some (Windows/UNIX/...) system. Now, I know about differences in newline character in Windows and UNIX (CR-LF / LF) and I want to parse this string on both (CR and LF) characters to detect which newline character(s) is used in this string, so I need to know what "\n" in VS6 C++ means.
My question is if I write a peace of code in Visual Studio 6 for Windows:
bool FindNewline (string & inputString) {
size_t found;
found = inputString.find ("\n");
return (found != string::npos ? true : false);
}
does this searches for CR+LF or only LF? Should I put "\r\n" or compiler interprets "\n" like CR+LF?
inputString.find ("\n");
will search for the LF character (alone).
Library routines may 'translate' between CR/LF and '\n' when I/O is performed on a text stream, but inside the realm of your program code, '\n' is just a line-feed.
"\n" means "\n". Nothing else. So you search for LF only. However Microsoft CRT does some conversions for you when you read a file in text mode, so you can write simpler code, sometimes.
All translation between "\n" and "\r\n" happens during I/O. At all other times, "\n" is just that and nothing more.
Somehow: return (found != string::npos ? true : false); reminds me of another answer I wrote a while back.
Apart from the VS6 part (you really, really want to upgrade this, the compiler is way out of date and Microsoft doesn't really support it anymore), the answer to the question depends on how you are getting the string.
For example, if you read it from a file in text mode, the runtime library will translate \r\n into \n. So if all your text strings are read in text mode via the usual file-based APIs, your search for\n` (ie, newline only) would be sufficient.
If the strings originate in files that are read in binary mode on Windows and are known to contain the DOS/Windows line separator \r\n, the you're better off searching for that character sequence.
EDIT: If you do get it in binary form, yes, ideally you'd have to check for both \r\n and \n. However I would expect that they aren't mixed within one string and still carry the same meaning unless it's a really messed up data format. I would probably check for \r\n first and then \n second if the strings are short enough and scanning them twice doesn't make that much of a difference. If it does, I'd write some code that checks for both \r\n and single \n in a single pass.
I am writing code that runs in Windows and outputs a text file that later becomes the input to a program in Linux. This program behaves incorrectly when given files that have newlines that are CR+LF rather than just LF.
I know that I can use tools like dos2unix, but I'd like to skip the extra step. Is it possible to get a C++ program in Windows to use the Linux newline instead of the Windows one?
Yes, you have to open the file in "binary" mode to stop the newline translation.
How you do it depends on how you are opening the file.
Using fopen:
FILE* outfile = fopen( "filename", "wb" );
Using ofstream:
std::ofstream outfile( "filename", std::ios_base::binary | std::ios_base::out );
OK, so this is probably not what you want to hear, but here's my $0.02 based on my experience with this:
If you need to pass data between different platforms, in the long run you're probably better off using a format that doesn't care what line breaks look like. If it's text files, users will sometimes mess with them. If by messing the line endings up they cause your application to fail, this is going to be a support intensive application.
Been there, done that, switched to XML. Made the support guys a lot happier.
A much cleaner solution is to use the ASCII escape sequence for the LF character (decimal 10): '\012' or '\x0A' represents an explicit single line feed regardless of platform.
Note that this at least on some compilers does not work; for example, on MSVC 2019 16.11.6, both '\012' and '\x0A' get translated to carriage return and line feed. It also does not matter there whether a string literal ("\012") or a char literal ('\012') is used.
This method also avoids string length surprises, as '\n' can expand to two characters. But so can multibyte unicode characters, in UTF8, when written directly into a string literal in the source code.
Note also that '\r' is the platform-independent code for a single carriage return (decimal 13). The '\f' character is not the line feed, but rather the form feed (decimal 12), which is not a newline on any platform I am aware of. C does not offer a single-character backslash escape for the line feed, thus the need for the longer octal or hexadecimal escapes.