How to properly convert USC-2 little endian into UTF-8? - c++

Question

I have a file and the line endings are in the windows style \r\n; it is encoded in USC-2 little endian.
Say this is my file fruit.txt (USC-2 little endian):
So I open it in a std::wifstream and try to parse the contents:
// open the file
std::wifstream file("fruit.txt");
if( ! file.is_open() ) throw std::runtime_error(std::strerror(errno));
// create container for the lines
std::forward_list<std::string> lines;
// Add each line to the container
std::wstring line;
while(std::getline(file,line)) lines.emplace_front(wstring_to_string(line));
If I try to print to cout...
// Printing to cout
for( auto it = lines.cbegin(); it != lines.cend(); ++it )
std::cout << *it << std::endl;
...This is what it outputs:
Cherry
Banana
ÿþApple
Worse yet, if I open it in Notepad++, this is what it looks like
I can sort-of rectify this by forcibly converting the encoding back to USC-2 which results in this:
My wstring_to_string function is defined as this:
std::string wstring_to_string( const std::wstring& wstr ) {
std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> convert;
return convert.to_bytes(wstr);
}
What in the world is going on here? How can I get a normal UTF-8 string? I have tried this method too: How to read utf-16 file into utf-8 std::string line by line, but imbuing the std::wifstream first results in no outputs altogether. Can someone please help direct me in the best way to go about converting USC-2 LE data to readable UTF-8 data?
Edit I think there may be a bug with mingw64/mingw-w64-x86_64-gcc 6.3.0-2 which is provided by MSYS2. I have tried everyone's suggestions and imbuing the locale into the streams is just rendering no output at all. I do know there are only two native locales provided, "C" and "POSIX". I was going to try Visual Studio but don't have sufficient internet speed for the 4GB download. I have used ICU like #Andrei R. suggested and it is working great.
I would have loved to use standard libraries but I am ok with this. Please take a look at my code if you need this solution: https://pastebin.com/qudy7yva

Answer 1

The code itself is fine as-is.
The real problem is that your input file is NOT valid UTF-16LE to begin with (your use of std::codecvt_utf8_utf16 requires UTF-16, not UCS-2). This is clearly shown in your Notepad++ screenshots.
Offhand, the file data looks like a UTF-16LE file with a BOM (ÿþ is the UTF-16LE BOM when viewed as 8bit ANSI) was appended as-is to the end of a UCS-2BE (or UTF-16BE) file that did not have a BOM.
You need to fix the input file so the entire file is valid UTF-16LE from beginning to end (with or without a BOM in front, not in the middle).
Then the code you already have will work.

Answer 2

converting to/from unicode is in general not so trivial. Have a look at ICU libraries, I believe, this is by far most complete encoding conversion library for c/c++.
There are also platform-dependent ways like WideCharToMultibyte (Win) or iconv (Linux). Or, with Qt, you can use QString::fromUtf16. Probably you will have to reverse endianness yourself.

Answer 3

For your case, the main issue is that you made the wifstream read the file in a wrong way. If you print the size of wstr in wstring_to_string, you will find that it's not what you expect.
https://stackoverflow.com/a/19698449/4005852
Set proper locale will fix this issue.
std::string wstring_to_string( const std::wstring& wstr ) {
std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> convert;
return convert.to_bytes(wstr);
}
int main()
{
// open the file
std::wifstream file("fruit.txt", std::ios::binary);
file.imbue(std::locale(file.getloc(),
new std::codecvt_utf16<wchar_t, 0x10ffff, std::little_endian>));
if( ! file.is_open() ) throw std::runtime_error(std::strerror(errno));
// create container for the lines
std::forward_list<std::string> lines;
// Add each line to the container
std::wstring line;
file.get(); // remove BOM
while(std::getline(file,line)) lines.emplace_front(wstring_to_string(line));
// Printing to cout
for( auto it = lines.cbegin(); it != lines.cend(); ++it )
std::cout << *it << std::endl;
return 0;
}