C++ u8 literal - unexpected encoding on Windows - c++

Question

I'm sure there's something I am missing here but I am comparing the contents of a regular string literal (in a utf8 encoded document) with a u8 string literal and on Windows the u8 encoded literal doesn't contain the expected utf8 encoded data while on Linux it does.
Details:
cpp file is utf8 encoded
C++17 is enabled
compiling using vs 2019 on Windows
compiling using gcc 9.2.1 on Linux
Here's the code:
#include <iostream>
#include <string>
struct HexCharStruct {
unsigned char c;
HexCharStruct(unsigned char _c) : c(_c) { }
};
inline std::ostream& operator<<(std::ostream& o, const HexCharStruct& hs) {
return (o << std::hex << (int)hs.c);
}
inline HexCharStruct hex(unsigned char _c) {
return HexCharStruct(_c);
}
int main( int argc, char** argv ) {
std::string s1 = "🎂";
std::string s2 = u8"🎂";
std::cout << "s1: ";
for (const char& c : s1)
std::cout << hex(c) << " ";
std::cout << "\ns2: ";
for (const char& c : s2)
std::cout << hex(c) << " ";
return 0;
}
Here are the hex values printed on Windows and Linux for s1 and s2 when I run this:
s1 (Windows): f0 9f 8e 82
s1 (Linux): f0 9f 8e 82
s2 (Windows): c3 b0 c5 b8 c5 bd e2 80 9a
s2 (Linux): f0 9f 8e 82
The utf8 hex values for 🎂 are f0 9f 8e 82 so everything is as expected except for s2 on Windows. Can anyone explain this?

Answer 1

The Microsoft compiler assumes source is ANSI-encoded, which depends on the localized version of Windows in use. On U.S. and Western European Windows the encoding is assumed to be Windows-1252.
When the compiler assumes Windows-1252, it decodes the UTF-8 bytes encoded in the source in the wrong encoding and thinks it is four Windows-1252 characters, then encodes those characters in UTF-8. A quick demo (Python):
>>> '🎂'.encode('utf8') # bytes in the file
b'\xf0\x9f\x8e\x82'
>>> b'\xf0\x9f\x8e\x82'.decode('Windows-1252') # What the compiler reads.
'🎂'
>>> '🎂'.encode('utf8') # What the compiler generates for u8 string.
b'\xc3\xb0\xc5\xb8\xc5\xbd\xe2\x80\x9a'
To use UTF-8 sources, two options are to encode the source in UTF-8 w/ BOM or add the /utf-8 compiler switch.