String.length() returns a wrong result - c++

Question

I'm trying to write a program in C++ to normalize a string by converting every uppercase letter into lowercase.
Also, I'm dealing with some special characters since my native language is Spanish, and it is supposed to work with Spanish words, too. For some reason, I was returning a string from my normalize() but couldn't cout it.
So, to make it work, I had to print it as if it was an array, and it worked for most cases because I was using word.length(). However, when I switched to result.length() it was giving me a straight 0 every time. I can't figure out what the problem is, maybe I have to add a null terminator to result so length() can do its work?
#include <iostream>
#include <string>
using namespace std;
string normalize(string word)
{
string result;
int j = 0;
for (int i = 0; i < word.length(); i++)
{
if (word[i] >= 'A' && word[i] <= 'Z')
{
result[j] = tolower(word[i]);
}
else
{
if (word[i] == char(0xC3))
{
switch (word[i + 1])
{
case char(0xA1):
word[j] = 'a';
break;
case char(0xA9):
word[j] = 'e';
break;
case char(0xAD):
word[j] = 'i';
break;
case char(0xB3):
word[j] = 'o';
break;
case char(0xBA):
word[j] = 'u';
break;
case char(0xBC):
word[j] = 'u';
break;
case char(0x81):
word[j] = 'a';
break;
case char(0x89):
word[j] = 'e';
break;
case char(0x8D):
word[j] = 'i';
break;
case char(0x93):
word[j] = 'o';
break;
case char(0x9A):
word[j] = 'u';
break;
case char(0x9C):
word[j] = 'u';
break;
}
i++;
}
else
result[j] = result[i];
}
j++;
}
return result;
}
int main()
{
int counter = 0;
string word;
while (cin >> word)
{
counter++;
string result = normalize(word);
cout << counter << ". ";
for (int i = 0; i < result.length(); i++)
{
cout << result[i];
}
cout << endl;
}
return 0;
}

Answer 1

normalize() expects a UTF-8 string as input. When handling the "special" characters, you are not writing any characters to result at all, you are writing them back to word instead. Even if you were writing them to result, you are not writing them correctly, as you haven't allocated any memory for result before populating it. You should be using operator+= instead of operator[], or at least call result.resize(word.length()) before entering the loop, and then call result.resize(j) after exiting the loop.
Try something more like this instead:
string normalize(const string &word)
{
string result;
result.reserve(word.length());
int i = 0;
while (i < word.length())
{
char ch = word[i++];
if (ch <= 0x7F)
{
result += (char) tolower(ch);
}
else if ((ch == 0xC3) && (i < word.length()))
{
ch = word[i++];
switch (ch)
{
case 0x81:
case 0xA1:
result += 'a';
break;
case 0x89:
case 0xA9:
result += 'e';
break;
case 0x8D:
case 0xAD:
result += 'i';
break;
case 0x93:
case 0xB3:
result += 'o';
break;
case 0x9A:
case 0x9C:
case 0xBA:
case 0xBC:
result += 'u';
break;
default:
result += '?';
break;
}
}
else
result += '?';
}
return result;
}
However, that being said, what normalize() is doing is not the correct way to handle UTF-8 in general. What you are looking for is called "transliteration", which is much more involved than your simple implementation. You should use a dedicated Unicode library like ICONV or ICU instead. But if you are going to do it manually, at least decode and process the UTF-8 properly, eg:
string normalize(const string &word)
{
// TODO: normalize word using Unicode Normalization Form NFC first...
string result;
result.reserve(word.length());
int i = 0;
while (i < word.length())
{
uint8_t ch = (uint8_t) word[i++];
int32_t cp;
int count;
if ((ch & 0x80) == 0x00)
{
cp = (ch & 0x7F);
count = 0;
}
else if ((ch & 0xE0) == 0xC0)
{
cp = ch & 0x1F;
count = 1;
}
else if ((ch & 0xF0) == 0xE0)
{
cp = ch & 0x0F;
count = 2;
}
else if ((ch & 0xF8) == 0xF0)
{
cp = ch & 0x07;
count = 3;
}
else
{
result += '?';
continue;
}
bool ok = ((i+count) <= word.length());
for (int j = 0; (ok) && (j < count); ++j)
{
ch = (uint8_t) word[i++];
if ((ch & 0xC0) != 0x80)
{
ok = false;
break;
}
cp <<= 6;
cp |= (ch & 0x3F);
}
if (!ok)
{
result += '?';
}
else
{
switch (cp)
{
case 0x00C1:
case 0x00E1:
result += 'a';
break;
case 0x00C9:
case 0x00E9:
result += 'e';
break;
case 0x00CD:
case 0x00ED:
result += 'i';
break;
case 0x00D3:
case 0x00F3:
result += 'o';
break;
case 0x00DA:
case 0x00DC:
case 0x00FA:
case 0x00FC:
result += 'u';
break;
default:
if (cp <= 0x007F)
result += (char) tolower(cp);
else
result += '?';
break;
}
}
}
return result;
}
Alternatively, if you are using C++11 or later:
string normalize(const string &word)
{
u32string u32 = codecvt_utf8<char32_t>{}.from_bytes(word);
// TODO: normalize u32 using Unicode Normalization Form NFC first...
string result;
result.reserve(u32.length());
for (char32_t cp : u32)
{
switch (cp)
{
case 0x00C1:
case 0x00E1:
result += 'a';
break;
case 0x00C9:
case 0x00E9:
result += 'e';
break;
case 0x00CD:
case 0x00ED:
result += 'i';
break;
case 0x00D3:
case 0x00F3:
result += 'o';
break;
case 0x00DA:
case 0x00DC:
case 0x00FA:
case 0x00FC:
result += 'u';
break;
default:
if (cp <= 0x007F)
result += (char) tolower(cp);
else
result += '?';
break;
}
}
return result;
}