How to parse std::string containing unicode literals? - c++

Question

I have std::string which stores characters encoded in UTF. Example:
std::string a = "\\u00c1\\u00c4\\u00d3";
Note that the length of a is 18 (3 characters, 6 ASCII symbols for each UTF character).
Question: How can I convert a into C++ string that have only 3 characters? Are there any standard functions (libraries) to do that?

Answer 1

There is nothing in the standard C++ library to handle this kind of conversion automatically for you. You are going to have to parse this string yourself, manually converting each 6-char "\uXXXX" substring into a 1-wchar value 0xXXXX that you can then store into a std::wstring or std::u16string as needed.
For example:
std::string a = "\\u00c1\\u00c4\\u00d3";
std::wstring ws;
ws.reserve(a.size());
for(size_t i = 0; i < a.size();)
{
char ch = a[i++];
if ((ch == '\\') && (i < a.size()) && (a[i] == 'u'))
{
wchar_t wc = static_cast<wchar_t>(std::stoi(a.substr(++i, 4), nullptr, 16));
i += 4;
ws.push_back(wc);
}
else
{
// depending on the charset used for encoding the string,
// this may or may not need to be decoded further...
ws.push_back(static_cast<wchar_t>(ch));
}
}
Live Demo
Alternatively:
std::string a = "\\u00c1\\u00c4\\u00d3";
std::wstring ws;
ws.reserve(a.size());
size_t start = 0;
do
{
size_t found = a.find("\\u", start);
if (found == std::string::npos) break;
if (start < found)
{
// depending on the charset used for encoding the string,
// this may or may not need to be decoded further...
ws.insert(ws.end(), a.begin()+start, a.begin()+found);
}
wchar_t wc = static_cast<wchar_t>(std::stoi(a.substr(found+2, 4), nullptr, 16));
ws.push_back(wc);
start = found + 6;
}
while (true);
if (start < a.size())
{
// depending on the charset used for encoding the string,
// this may or may not need to be decoded further...
ws.insert(ws.end(), a.begin()+start, a.end());
}
Live Demo
Otherwise, use a 3rd party library that already does this kind of translation for you.