I'm in need of reading both UTF-8 string (std::string) and UTF-16 string (std::u16string) from a file (opened with ifstream).
The UTF-8 string is easy, I think I can just use something like std::getline(stream, str, '\0').
But about UTF-16, I'm not sure how I can actually read it. I know I can maybe loop in the file and read 2 bytes each time until a 0x0000 byte, but I'm not sure if that is the right and best way to do it.
So, how can I read it?
-- edit --
For now, I'm doing it this way, is this ok?
std::string binaryReader::ru16str_n()
{
std::u16string str;
char16_t ch = 0;
while (true)
{
binary.read(reinterpret_cast<char*>(&ch), 2);
if (ch != '\0')
str.push_back(ch);
else break;
}
return std::wstring_convert<
std::codecvt_utf8_utf16<char16_t>, char16_t>{}.to_bytes(str);
}
Related
I am able to successfully read in UTF8 character text files by redirecting input and output on the terminal and then using wcin and wcout
_setmode(_fileno(stdout), _O_U8TEXT);
_setmode(_fileno(stdin), _O_U8TEXT);
Now I'd like to be able to read in UTF8 text using filestreams, but I don't know how to set the mode of the filestreams so that it could read in these characters like I did with stdin and stdout. I've tried using wifstreams/wofstreams and those still read and write garbage, by themselves.
C++'s <iostreams> library doesn't have built-in support for conversions from one text encoding to another. If you need your input text converted from utf-8 into another format (say, for example, the underlying codepoints of the encoding), you'll need to write that conversion manually.
std::string data;
std::ifstream in("utf8.txt");
in.seekg(0, std::ios::end);
auto size = in.tellg();
in.seekg(0, std::ios::beg);
data.resize(size);
in.read(data.data(), size);
//data now contains the entire contents of the file
uint32_t partial_codepoint = 0;
unsigned num_of_bytes = 0;
std::vector<uint32_t> codepoints;
for(char c : data) {
uint8_t byte = uint8_t(c);
if(byte < 128) {
//Character is just a basic ascii character, so we'll just set that as the codepoint value
codepoints.push_back(byte);
if(num_of_bytes > 0) {
//Data was malformed: error handling?
//Codepoint abruptly ended
}
} else {
//Character is part of multi-byte encoding
if(partial_codepoint) {
//We've already begun storing the codepoint
if((byte >> 6) != 0b10) {
//Data was malformed: error handling?
//Codepoint abruptly ended
}
partial_codepoint = (partial_codepoint << 6) | (0b0011'1111 & byte);
num_of_bytes--;
if(num_of_bytes == 0) {
codepoints.emplace_back(partial_codepoint);
partial_codepoint = 0;
}
} else {
//Beginning of new codepoint
if((byte >> 6) == 0b10) {
//Data was malformed: error handling?
//Codepoint did not have proper beginning
}
while(byte & 0b1000'0000) {
num_of_bytes++;
byte = byte << 1;
}
partial_codepoint = byte >> num_of_bytes;
}
}
}
This code will reliably convert from [correctly-encoded] utf-8 to utf-32, which is usually the easiest form to convert directly into glyphs + characters—though remember that codepoints are not characters.
To keep things consistent in your code, my recommendation is that utf-8 encoded text be stored in your program using std::string, and utf-32 encoded text be stored as std::vector<uint32_t>.
There is a csv file which has the many different languages encoded in utf-8. I have to parse the file and validate for invalid characters.
I have written a sample program below as shown…
int main(void)
{
string invalidUTF8Chars = ""; // Invalid UTF-8 Chars array.
invalidUTF8Chars+= "\u00A0";
invalidUTF8Chars+= "\u005E";
invalidUTF8Chars+= "\u00FE";
invalidUTF8Chars+= "\u00BA";
invalidUTF8Chars+= "\u00AF";
FILE* fp;
char ch;
fp = fopen("unicodeUTF8TextFile.txt","r");
if(fp != NULL)
{
while(( ch = fgetc(fp) ) != EOF ) // Reading byte by byte form input file.
{
//if (strchr(invalidUTF8Chars.c_str(), ch)) // How do I validate here?
{
printf("Invalid character\n");
}
}
}
return 0;
}
How do I compare the data read from the file against the invalid chars?
When strchr() fails to find a character it returns a NULL-pointer. What you need to do is to check if the return was a NULL-pointer or not:
if(strchr(invalidUTF8Chars.c_str(), ch) == nullptr){
printf("Invalid character\n");
}
Here's the strchr() reference for your convenience.
Invalid character for UTF-8 may either mean that the UTF-8 encoding is invalid and doesn't correspond to any character, or that the UTF-8 decoding will lead to a character that you don't want.
You are interested in the second variant, where each character is encoded as one or more bytes in UTF-8, specifically "\u005E" is one byte in UTF-8 and the others are 2 bytes.
Thus you cannot reject individual bytes in your example, but would either need to decode to Unicode-characters or read everything as UTF-8 and then find the issues using something like:
if (strstr(readFile, u8"\u00A0") != nullptr || strstr(readFile, u8"\u005E") != nullptr ... ) printf("Found bad character\n");
std::string is commonly interpreted as UTF8, hence has a variable length encoding. In my font renderer I've hit a problem in that I'm not sure how to get a "character" from a std::string and convert it into a Freetype FT_ULong in order to get a glyph with FT_Get_Char_Index. That is to say, I am not sure that what I'm doing is "correct" as I'm just iterating through std::string and casting the resulting chars over (surely this is incorrect, although it works with my OS defaults).
So is there a "correct" way of doing this and more importantly has someone written a library that implements this "correct" way that I can use off the shelf?
You should first check how UTF8 is encoded, and would know that what kind of start bits are with how many bytes.
See http://en.wikipedia.org/wiki/UTF8
And then you can write code like this:
if ((byte & 0x80) == 0x00) {
// 1 byte UTF8 char
}
else if ((byte & 0xE0) == 0xC0) {
// 2 bytes UTF8 char
}
else if ...
Then you can iterates each UTF8 characters in the std::string with correct bytes.
My company use some code like this:
std::string(CT2CA(some_CString)).c_str()
which I believe it converts a Unicode string (whose type is CString)into ANSI encoding, and this string is for a email's subject. However, header of the email (which includes the subject) indicates that the mail client should decode it as a unicode (this is how the original code does). Thus, some German chars like "ä ö ü" will not be properly displayed as the title.
Is there anyway that I can put this header back to UTF8 and store into a std::string or const char*?
I know there are a lot of smarter ways to do this, but I need to keep the code sticking to its original one (i.e. sent the header as std::string or const char*).
Thanks in advance.
Becareful : it's '|' and not '&' !
*buffer++ = 0xC0 | (c >> 6);
*buffer++ = 0x80 | (c & 0x3F);
This sounds like a plain conversion from one encoding to another encoding: You can use std::codecvt<char, char, mbstate_t> for this. Whether your implementation ships with a suitable conversion, I don't know, however. From the sounds of it you just try to convert ISO-Latin-1 into Unicode. That should be pretty much trivial: the first 128 characters map (0 to 127) identically to UTF-8 and the second half conveniently map to the corresponding Unicode code points, i.e., you just need to encode the corresponding value into UTF-8. Each character will be replaced by two characters. That it, I think the conversion is something like that:
// Takes the next position and the end of a buffer as first two arguments and the
// character to convert from ISO-Latin-1 as third argument.
// Returns a pointer to end of the produced sequence.
char* iso_latin_1_to_utf8(char* buffer, char* end, unsigned char c) {
if (c < 128) {
if (buffer == end) { throw std::runtime_error("out of space"); }
*buffer++ = c;
}
else {
if (end - buffer < 2) { throw std::runtime_error("out of space"); }
*buffer++ = 0xC0 | (c >> 6);
*buffer++ = 0x80 | (c & 0x3f);
}
return buffer;
}
I want to read Unicode file (UTF-8) character by character, but I don't know how to read from a file one by one character.
Can anyone to tell me how to do that?
First, look at how UTF-8 encodes characters: http://en.wikipedia.org/wiki/UTF-8#Description
Each Unicode character is encoded to one or more UTF-8 byte. After you read first next byte in the file, according to that table:
(Row 1) If the most significant bit is 0 (char & 0x80 == 0) you have your character.
(Row 2) If the three most significant bits are 110 (char & 0xE0 == 0xc0), you have to read another byte, and the bits 4,3,2 of the first UTF-8 byte (110YYYyy) make the first byte of the Unicode character (00000YYY) and the two least significant bits with 6 least significant bits of the next byte (10xxxxxx) make the second byte of the Unicode character (yyxxxxxx); You can do the bit arithmetic using shifts and logical operators of C/C++ easily:
UnicodeByte1 = (UTF8Byte1 << 3) & 0xE0;
UnicodeByte2 = ( (UTF8Byte1 << 6) & 0xC0 ) | (UTF8Byte2 & 0x3F);
And so on...
Sounds a bit complicated, but it's not difficult if you know how to modify the bits to put them in proper place to decode a UTF-8 string.
UTF-8 is ASCII compatible, so you can read a UTF-8 file like you would an ASCII file. The C++ way to read a whole file into a string is:
#include <iostream>
#include <string>
#include <fstream>
std::ifstream fs("my_file.txt");
std::string content((std::istreambuf_iterator<char>(fs)), std::istreambuf_iterator<char>());
The resultant string has characters corresponding to UTF-8 bytes. you could loop through it like so:
for (std::string::iterator i = content.begin(); i != content.end(); ++i) {
char nextChar = *i;
// do stuff here.
}
Alternatively, you could open the file in binary mode, and then move through each byte that way:
std::ifstream fs("my_file.txt", std::ifstream::binary);
if (fs.is_open()) {
char nextChar;
while (fs.good()) {
fs >> nextChar;
// do stuff here.
}
}
If you want to do more complicated things, I suggest you take a peek at Qt. I've found it rather useful for this sort of stuff. At least, less painful than ICU, for doing largely practical things.
QFile file;
if (file.open("my_file.text") {
QTextStream in(&file);
in.setCodec("UTF-8")
QString contents = in.readAll();
return;
}
In theory strlib.h has a function mblen which shell return length of multibyte symbol. But in my case it returns -1 for first byte of multibyte symbol and continue it returns all time. So I write the following:
{
if(i_ch == nullptr) return -1;
int l = 0;
char ch = *i_ch;
int mask = 0x80;
while(ch & mask) {
l++;
mask = (mask >> 1);
}
if (l < 4) return -1;
return l;
}
It's take less time than research how shell using mblen.
try this: get the file and then loop through the text based on it's length
Pseudocode:
String s = file.toString();
int len = s.length();
for(int i=0; i < len; i++)
{
String the_character = s[i].
// TODO : Do your thing :o)
}