I am writing a program for school, and want to include an Easter Egg with the look of disapproval (ಠ_ಠ), however, I do not know what classes in C++ support this character. I would assume this character be included in unicode, but I am not sure how to use that to output into a console application.
If anyone could help me out with this, I would appreciate it. Thanks
Well you need to enable Unicode for your application.
For MSVC you can use _setmode(_fileno(stdout), _O_U16TEXT); at the start of your main.
But for mingw this is a bit more difficult, see here.
To actually print it you can use the character code and use wcout but be wary!
From: http://www.cplusplus.com/reference/iostream/cout/
A program should not mix output operations on cout with output operations on wcout (or with other wide-oriented output operations on stdout): Once an output operation has been performed on either, the standard output stream acquires an orientation (either narrow or wide) that can only be safely changed by calling freopen on stdout.
You can use \u0CA0, that's the Unicode code for this eye thing. Here's a live example:
int main() {
cout << "hi \u0CA0_\u0CA0";
return 0;
}
https://ideone.com/QBFtsM
Although IDEOne supports unicode (as in, I can just paste ಠ) so I'm not sure if that will work for you. Let me know.
Related
The standard recognizes int main() and int main(int, char* []) entry points, and there is no unicode version of the function, except of course if provided by implementation such as wmain on windows.
I have code which strictly uses 'wide' strings ie. wchar_t and std::wstring not even a single LOC makes any use of char type.
Also all the 'program output' is printed with std::wcout.
Now I do understand when the program starts it's stream will be set to 'wide' stream, and even if somewhere in the code std::cout appears instead of std::wcout then output may be wrong.
Now my first misunderstanding is, if a user runs several command line tools in same terminal (ie. cmd.exe) where one program uses wide string and other uses ASCII strings then what is the behavior of the terminal?
Does it handle both streams just fine? is wide stream and non wide stream related to program and not to terminal?
Secondly, my main issue is that my 'unicode' program expects user input, which will most likely be char because int main(int, char* []) accepts char string, but my code uses wide strings.
What are my options here? do I need to convert input non unicode strings to wide strings?
And what happens if I use int wmain(int, wchar_t* []) on windows? how do I know if terminal will interpret user input as wide characters and forward them to my entry point?
what if user copy/pastes ASCII string into the console as input to my entry point?
So my questions are, console program input and output char and wchar_t in both portable and non portable code, missing wide main function by standard is what confuses me most in combination with different terminals.
edit
To be clear,
I want to use standard main and wide strings.
my code uses wide string.
my program expects user input, and writes output. so how to best handle this?
Is wmain the only solution to get unicode input and how do terminals work in regard to unicode vs non unicode with different programs in same console session.
I've been using StackOverflow since the beginning, and have on occasion been tempted to post questions, but I've always either figured them out myself or found answers posted eventually... until now. This feels like it should be fairly simple, but I've been wandering around the internet for hours with no success, so I turn here:
I have a pretty standard utf-16 text file, with a mixture of English and Chinese characters. I would like those characters to end up in a string (technically, a wstring). I've seen a lot of related questions answered (here and elsewhere), but they're either looking to solve the much harder problem of reading arbitrary files without knowing the encoding, or converting between encodings, or are just generally confused about "Unicode" being a range of encodings. I know the source of the text file I'm trying to read, it will always be UTF16, it has a BOM and everything, and it can stay that way.
I had been using the solution described here, which worked for text files that were all English, but after encountering certain characters, it stopped reading the file. The only other suggestion I found was to use ICU, which would probably work, but I'd really rather not include a whole large library in an application for distribution, just to read one text file in one place. I don't care about system independence, though - I only need it to compile and work in Windows. A solution that didn't rely on that fact would prettier, of course, but I would be just as happy for a solution that used the stl while relying on assumptions about Windows architecture, or even solutions that involved win32 functions, or ATL; I just don't want to have to include another large 3rd-party library like ICU. Am I still totally out of luck unless I want to reimplement it all myself?
edit: I'm stuck using VS2008 for this particular project, so C++11 code sadly won't help.
edit 2: I realized that the code I had been borrowing before didn't fail on non-English characters like I thought it was doing. Rather, it fails on specific characters in my test document, among them '：' (FULLWIDTH COLON, U+FF1A) and '）' (FULLWIDTH RIGHT PARENTHESIS, U+FF09). bames53's posted solution also mostly works, but is stumped by those same characters?
edit 3 (and the answer!): the original code I had been using -did- mostly work - as bames53 helped me discover, the ifstream just needed to be opened in binary mode for it to work.
The C++11 solution (supported, on your platform, by Visual Studio since 2010, as far as I know), would be:
#include <fstream>
#include <iostream>
#include <locale>
#include <codecvt>
int main()
{
// open as a byte stream
std::wifstream fin("text.txt", std::ios::binary);
// apply BOM-sensitive UTF-16 facet
fin.imbue(std::locale(fin.getloc(),
new std::codecvt_utf16<wchar_t, 0x10ffff, std::consume_header>));
// read
for(wchar_t c; fin.get(c); )
std::cout << std::showbase << std::hex << c << '\n';
}
When you open a file for UTF-16, you must open it in binary mode. This is because in text mode, certain characters are interpreted specially - specifically, 0x0d is filtered out completely and 0x1a marks the end of the file. There are some UTF-16 characters that will have one of those bytes as half of the character code and will mess up the reading of the file. This is not a bug, it is intentional behavior and is the sole reason for having separate text and binary modes.
For the reason why 0x1a is considered the end of a file, see this blog post from Raymond Chen tracing the history of Ctrl-Z. It's basically backwards compatibility run amok.
Edit:
So it appears that the issue was that the Windows treats certain magic byte sequences as the end of the file in text mode. This is solved by using binary mode to read the file, std::ifstream fin("filename", std::ios::binary);, and then copying the data into a wstring as you already do.
The simplest, non-portable solution would be to just copy the file data into a wchar_t array. This relies on the fact that wchar_t on Windows is 2 bytes and uses UTF-16 as its encoding.
You'll have a bit of difficulty converting UTF-16 to the locale specific wchar_t encoding in a completely portable fashion.
Here's the unicode conversion functionality available in the standard C++ library (though VS 10 and 11 implement only items 3, 4, and 5)
codecvt<char32_t,char,mbstate_t>
codecvt<char16_t,char,mbstate_t>
codecvt_utf8
codecvt_utf16
codecvt_utf8_utf16
c32rtomb/mbrtoc32
c16rtomb/mbrtoc16
And what each one does
A codecvt facet that always converts between UTF-8 and UTF-32
converts between UTF-8 and UTF-16
converts between UTF-8 and UCS-2 or UCS-4 depending on the size of target element (characters outside BMP are probably truncated)
converts between a sequence of chars using a UTF-16 encoding scheme and UCS-2 or UCS-4
converts between UTF-8 and UTF-16
If the macro __STDC_UTF_32__ is defined these functions convert between the current locale's char encoding and UTF-32
If the macro __STDC_UTF_16__ is defined these functions convert between the current locale's char encoding and UTF-16
If __STDC_ISO_10646__ is defined then converting directly using codecvt_utf16<wchar_t> should be fine since that macro indicates that wchar_t values in all locales correspond to the short names of Unicode charters (and so implies that wchar_t is large enough to hold any such value).
Unfortunately there's nothing defined that goes directly from UTF-16 to wchar_t. It's possible to go UTF-16 -> UCS-4 -> mb (if __STDC_UTF_32__) -> wc, but you'll loose anything that's not representable in the locale's multi-byte encoding. And of course no matter what, converting from UTF-16 to wchar_t will lose anything not representable in the locale's wchar_t encoding.
So it's probably not worth being portable, and instead you can just read the data into a wchar_t array, or use some other Windows specific facility, such as the _O_U16TEXT mode on files.
This should build and run anywhere, but makes a bunch of assumptions to actually work:
#include <fstream>
#include <sstream>
#include <iostream>
int main ()
{
std::stringstream ss;
std::ifstream fin("filename");
ss << fin.rdbuf(); // dump file contents into a stringstream
std::string const &s = ss.str();
if (s.size()%sizeof(wchar_t) != 0)
{
std::cerr << "file not the right size\n"; // must be even, two bytes per code unit
return 1;
}
std::wstring ws;
ws.resize(s.size()/sizeof(wchar_t));
std::memcpy(&ws[0],s.c_str(),s.size()); // copy data into wstring
}
You should probably at least add code to handle endianess and the 'BOM'. Also Windows newlines don't get converted automatically so you need to do that manually.
It's probably a long-shot that anyone can answer this without seeing all the source code and libraries, etc., but I'll try.
I have a program X written in C++ using boost-1.41. If X outputs with std::cout, then running X from another program using fp=popen("X", "r") allows X's output to be seen via fgets(buff, 1024, fp).
Now if I change X to use printf() instead of std::cout, the output of X is no longer seen. However, running X from bash produces the output as expected.
What could possibly explain this difference?! I suspect boost is involved here, but I don't know much about boost.
Note: I am happy sticking to std::cout and my problem is solved. But I'm trying to understand what the problem was with printf().
The reason is that you probably used std::endl with std::cout. That, in addition to writing a newline character, also flushes the output buffer.
To do the same with printf you can just add fflush(stdout); after the call.
I'm writing a program on a Linux platform that is generating text files that will be viewed on, inevitably, a Windows platform.
Right now, passing std::endl into a ostream generates the CR character only for newlines. Naturally, these text files look wrong in MS Notepad.
Is there a way to change std::endl such that it uses CR+LF for newline instead of LF?
I know I could write my own custom manipulator, like win_endl, for generating my own newlines, but I use the std::endl symbol in a lot of places, and like many programmers, have a tendency to do the thing that requires the least work possible. Could I simply overload std::endl to produce CR+LF, or is this a dumb idea for maintainability?
NB: I checked out this question, but it's asking about going the other way, and the accepted answer seems rather incomplete.
std::endl is basicly:
std::cout << "\n" << std::flush;
So just use "\r\n" instead and omit the flush. It's faster that way, too!
From the ostream header file on endl:
This manipulator is often mistakenly used when a simple newline is desired, leading to poor buffering performance.
Opening a file in text mode should cause std::endl to be converted to the appropriate line ending for your platform. Your problem is that newline is appropriate for your platform, but the files you create aren't intended for your platform.
I'm not sure how you plan on overloading or changing endl, and changing its behavior would certainly be surprising for any developers new to your project. I'd recommend switching to win_endl (should be a simple search-and-replace) or maybe switching from a standard ostream to a Boost.Iostreams filtering stream to do the conversion for you.
Windows Notepad is pretty much the only Windows program you'll find that doesn't handle LF-only files properly. Almost everything else (including WordPad) handles LF-only files just fine.
This problem is a bug in Notepad.
You shouldn't use \r\n. Just use \n, but then open the stream in "text" mode, which than will do the conversion for you. You may not care about cross-platform, but this is the official way of doing it.
That way, the same code will spit-out \n on unix, \r\n on windows, and \r on mac.
Here was my solution to the problem. It's a bit of a mash-up of all the info provided in the answers:
I created a macro in a win_endl.h file for the newline I wanted:
#define win_endl "\r\n"
Then I did a search-and-replace:
sheepsimulator#sheep\_machine > sed -i 's/std::endl/win_endl' *
And ensured all my files included win_endl.h.
Not being able to wrap my head around this one is a real source of shame...
I'm working with a French version of Visual Studio (2008), in a French Windows (XP). French accents put in strings sent to the output window get corrupted. Ditto input from the output window. Typical character encoding issue, I enter ANSI, get UTF-8 in return, or something to that effect. What setting can ensure that the characters remain in ANSI when showing a "hardcoded" string to the output window?
EDIT:
Example:
#include <iostream>
int main()
{
std:: cout << "àéêù" << std:: endl;
return 0;
}
Will show in the output:
óúÛ¨
(here encoded as HTML for your viewing pleasure)
I would really like it to show:
àéêù
Before I go any further, I should mention that what you are doing is not c/c++ compliant. The specification states in 2.2 what character sets are valid in source code. It ain't much in there, and all the characters used are in ascii. So... Everything below is about a specific implementation (as it happens, VC2008 on a US locale machine).
To start with, you have 4 chars on your cout line, and 4 glyphs on the output. So the issue is not one of UTF8 encoding, as it would combine multiple source chars to less glyphs.
From you source string to the display on the console, all those things play a part:
What encoding your source file is in (i.e. how your C++ file will be seen by the compiler)
What your compiler does with a string literal, and what source encoding it understands
how your << interprets the encoded string you're passing in
what encoding the console expects
how the console translates that output to a font glyph.
Now...
1 and 2 are fairly easy ones. It looks like the compiler guesses what format the source file is in, and decodes it to its internal representation. It generates the string literal corresponding data chunk in the current codepage no matter what the source encoding was. I have failed to find explicit details/control on this.
3 is even easier. Except for control codes, << just passes the data down for char *.
4 is controlled by SetConsoleOutputCP. It should default to your default system codepage. You can also figure out which one you have with GetConsoleOutputCP (the input is controlled differently, through SetConsoleCP)
5 is a funny one. I banged my head to figure out why I could not get the é to show up properly, using CP1252 (western european, windows). It turns out that my system font does not have the glyph for that character, and helpfully uses the glyph of my standard codepage (capital Theta, the same I would get if I did not call SetConsoleOutputCP). To fix it, I had to change the font I use on consoles to Lucida Console (a true type font).
Some interesting things I learned looking at this:
the encoding of the source does not matter, as long as the compiler can figure it out (notably, changing it to UTF8 did not change the generated code. My "é" string was still encoded with CP1252 as 233 0 )
VC is picking a codepage for the string literals that I do not seem to control.
controlling what the console shows is more painful than what I was expecting
So... what does this mean to you ? Here are bits of advice:
don't use non-ascii in string literals. Use resources, where you control the encoding.
make sure you know what encoding is expected by your console, and that your font has the glyphs to represent the chars you send.
if you want to figure out what encoding is being used in your case, I'd advise printing the actual value of the character as an integer. char * a = "é"; std::cout << (unsigned int) (unsigned char) a[0] does show 233 for me, which happens to be the encoding in CP1252.
BTW, if what you got was "ÓÚÛ¨" rather than what you pasted, then it looks like your 4 bytes are interpreted somewhere as CP850.
Because I was requested to, I’ll do some necromancy. The other answers were from 2009, but this article still came up on a search I did in 2018. The situation today is very different. Also, the accepted answer was incomplete even back in 2009.
The Source Character Set
Every compiler (including Microsoft’s Visual Studio 2008 and later, gcc, clang and icc) will read UTF-8 source files that start with BOM without a problem, and clang will not read anything but UTF-8, so UTF-8 with a BOM is the lowest common denominator for C and C++ source files.
The language standard doesn’t say what source character sets the compiler needs to support. Some real-world source files are even saved in a character set incompatible with ASCII. Microsoft Visual C++ in 2008 supported UTF-8 source files with a byte order mark, as well as both forms of UTF-16. Without a byte order mark, it would assume the file was encoded in the current 8-bit code page, which was always a superset of ASCII.
The Execution Character Sets
In 2012, the compiler added a /utf-8 switch to CL.EXE. Today, it also supports the /source-charset and /execution-charset switches, as well as /validate-charset to detect if your file is not actually UTF-8. This page on MSDN has a link to the documentation on Unicode support for every version of Visual C++.
Current versions of the C++ standard say the compiler must have both an execution character set, which determines the numeric value of character constants like 'a', and a execution wide-character set that determines the value of wide-character constants like L'é'.
To language-lawyer for a bit, there are very few requirements in the standard for how these must be encoded, and yet Visual C and C++ manage to break them. It must contain about 100 characters that cannot have negative values, and the encodings of the digits '0' through '9' must be consecutive. Neither capital nor lowercase letters have to be, because they weren’t on some old mainframes. (That is, '0'+9 must be the same as '9', but there is still a compiler in real-world use today whose default behavior is that 'a'+9 is not 'j' but '«', and this is legal.) The wide-character execution set must include the basic execution set and have enough bits to hold all the characters of any supported locale. Every mainstream compiler supports at least one Unicode locale and understands valid Unicode characters specified with \Uxxxxxxxx, but a compiler that didn’t could claim to be complying with the standard.
The way Visual C and C++ violate the language standard is by making their wchar_t UTF-16, which can only represent some characters as surrogate pairs, when the standard says wchar_t must be a fixed-width encoding. This is because Microsoft defined wchar_t as 16 bits wide back in the 1990s, before the Unicode committee figured out that 16 bits were not going to be enough for the entire world, and Microsoft was not going to break the Windows API. It does support the standard char32_t type as well.
UTF-8 String Literals
The third issue this question raises is how to get the compiler to encode a string literal as UTF-8 in memory. You’ve been able to write something like this since C++11:
constexpr unsigned char hola_utf8[] = u8"¡Hola, mundo!";
This will encode the string as its null-terminated UTF-8 byte representation regardless of whether the source character set is UTF-8, UTF-16, Latin-1, CP1252, or even IBM EBCDIC 1047 (which is a silly theoretical example but still, for backward-compatibility, the default on IBM’s Z-series mainframe compiler). That is, it’s equivalent to initializing the array with { 0xC2, 0xA1, 'H', /* ... , */ '!', 0 }.
If it would be too inconvenient to type a character in, or if you want to distinguish between superficially-identical characters such as space and non-breaking space or precomposed and combining characters, you also have universal character escapes:
constexpr unsigned char hola_utf8[] = u8"\u00a1Hola, mundo!";
You can use these regardless of the source character set and regardless of whether you’re storing the literal as UTF-8, UTF-16 or UCS-4. They were originally added in C99, but Microsoft supported them in Visual Studio 2015.
Edit: As reported by Matthew, u8" strings are buggy in some versions of MSVC, including 19.14. It turns out, so are literal non-ASCII characters, even if you specify /utf-8 or /source-charset:utf-8 /execution-charset:utf-8. The sample code above works properly in 19.22.27905.
There is another way to do this that worked in Visual C or C++ 2008, however: octal and hexadecimal escape codes. You would have encoded UTF-8 literals in that version of the compiler with:
const unsigned char hola_utf8[] = "\xC2\xA1Hello, world!";
Try this:
#include <iostream>
#include <locale>
int main()
{
std::locale::global(std::locale(""));
std::cout << "àéêù" << std::endl;
return 0;
}
Using _setmode() works¹ and is arguably better than changing the codepage or setting a locale, since it'll actually make your program output in Unicode and thus will be consistent - no matter which codepage or locale are currently set.
Example:
#include <iostream>
#include <io.h>
#include <fcntl.h>
int wmain()
{
_setmode( _fileno(stdout), _O_U16TEXT );
std::wcout << L"àéêù" << std::endl;
return 0;
}
Inside Visual Studio, make sure you set up your project for Unicode (Right-click *Project* -> Click *General* -> *Character Set* = *Use Unicode Character Set*).
MinGW users:
Define both UNICODE and _UNICODE
Add -finput-charset=iso-8859-1 to the compiler options to get around this error: "converting to execution character set: Invalid argument"
Add -municode to the linker options to get around "undefined reference to `WinMain#16" (read more).
**Edit:** The equivalent call to set unicode *input* is: `_setmode( _fileno(stdin), _O_U16TEXT );`
Edit 2: An important piece of information, specially considering the question uses std::cout. This is not supported. The MSDN Docs states (emphasis mine):
Unicode mode is for wide print functions (for example, wprintf) and is
not supported for narrow print functions. Use of a narrow print
function on a Unicode mode stream triggers an assert.
So, don't use std::cout when the console output mode is _O_U16TEXT; similarly, don't use std::cin when the console input is _O_U16TEXT. You must use the wide version of these facilities (std::wcout, std::wcin).
And do note that mixing cout and wcout in the same output is not allowed (but I find it works if you call flush() and then _setmode() before switching between the narrow and wide operations).
I tried this code:
#include <iostream>
#include <fstream>
#include <sstream>
int main()
{
std::wstringstream wss;
wss << L"àéêù";
std::wstring s = wss.str();
const wchar_t* p = s.c_str();
std::wcout << ws.str() << std::endl;
std::wofstream file("C:\\a.txt");
file << p << endl;
return 0;
}
The debugger showed that wss, s and p all had the expected values (i.e. "àéêù"), as did the output file. However, what appeared in the console was óúÛ¨.
The problem is therefore in the Visual Studio console, not the C++. Using Bahbar's excellent answer, I added:
SetConsoleOutputCP(1252);
as the first line, and the console output then appeared as it should.
//Save As Windows 1252
#include<iostream>
#include<windows.h>
int main()
{
SetConsoleOutputCP(1252);
std:: cout << "àéêù" << std:: endl;
}
Visual Studio does not supports UTF 8 for C++, but partially supports for C:
//Save As UTF8 without signature
#include<stdio.h>
#include<windows.h>
int main()
{
SetConsoleOutputCP(65001);
printf("àéêù\n");
}
Make sure you do not forget to change the console's font to Lucida Consolas as mentionned by Bahbar : it was crucial in my case (French win 7 64 bit with VC 2012).
Then as mentionned by others use SetConsoleOutputCP(1252) for C++ but it may fail depending on the available pages so you might want to use GetConsoleOutputCP() to check that it worked or at least to check that SetConsoleOutputCP(1252) returns zero. Changing the global locale also works (for some reason there is no need to do cout.imbue(locale()); but it may break some librairies!
In C, SetConsoleOutputCP(65001); or the locale-based approach worked for me once I had saved the source code as UTF8 without signature (scroll down, the sans-signature choice is way below in the list of pages).
Input using SetConsoleCP(65001); failed for me apparently due to a bad implementation of page 65001 in windows. The locale approach failed too both in C and C++. A more involved solution, not relying on native chars but on wchar_t seems required.
I had the same problem with Chinese input. My source code is utf8 and I added /utf-8 in the compiler option. It works fine under c++ wide-string and wide-char but not work under narrow-string/char which it shows Garbled character/code in Visual Studio 2019 debugger and my SQL database. I have to use the narrow characters because of converting to SQLAPI++'s SAString. Eventually, I find checking the following option (contorl panel->Region->Administrative->Change system locale) can resolve the issue. I know it is not an ideal solution but it does help me.
In visual studio File->Save yourSource.cpp As
then it will pop up a dialog ask you if you want to replace existed file and you choose yes.
Then pop this dialog: select UTF-8 with signature. This solves my giberish output problem both on console and file.
This also comply with #Davislor's answer:
UTF-8 with a BOM is the lowest common denominator for C and C++ source
files