QString(const char* p) constructor misrepresents non ASCII characters - c++

Question

I am trying to upgrade a bigger C++ program from Qt4 to Qt5 or higher and have some problems with the legacy code that was written in ISO-LATIN1. From Qt5, the code is expected to be present in UTF8 and that is what we tried to do. We use a own String-class (let's call it myQString here), that was basically a char* under Qt4 and went to a QString-derived class in Qt5. So far so good.
The cases where I still have some problems is when I try to pass char* variables to the myQString class, that includes non-ASCII characters (like the letter characters with diaeresis for example, 'ä', 'Ä', 'ö', 'Ö', etc.).
I tried to write a mini program that reproduces/illustrates the problem. To make it clearer I could post some code but a picture would be better in this case:
Zoom Debugger output
Here we can see via the Debugger-View, that the desired end-products (cyan and yellow color: "mstr2, mstr4, qstr2, qstr4), that should be something that store "Äa", misrepresent the first byte "Ä". All of them use the green marked constructor, myQString(const char* p).
The function that illustrates the problem (in the Debugger) is charPointerToQstring(). It is part of the file main.cpp (see last code block).
If you want to run that "mini" program, I will also post all the files (four) needed to do so. I am using QtCreator and have a project file, which you can call how you like, let's say "testingQString.pro"
QT -= gui
CONFIG += c++17 console
SOURCES += \
main.cpp \
myqstring.cpp
HEADERS += \
myqstring.h
Then we have a "stripped" myQString class, with the two files:
myqstring.h:
#ifndef MYQSTRING_H
#define MYQSTRING_H
#include <QString>
class myQString : public QString
{
public:
myQString();
myQString(const QString& str);
myQString(char c);
myQString(const char* p);
myQString(const QByteArray& ba);
};
#endif // MYQSTRING_H
and "stripped" myqstring.cpp:
#include "myqstring.h"
#include <QDebug>
#define ENTER_FUNCTION qDebug() << "========== Entering:" << Q_FUNC_INFO
myQString::myQString()
{
ENTER_FUNCTION;
}
myQString::myQString(const QString& str) : QString(str)
{
ENTER_FUNCTION;
}
myQString::myQString(char c) : QString(QChar(c))
{
ENTER_FUNCTION;
}
myQString::myQString(const char* p) : QString(p)
{
ENTER_FUNCTION;
}
myQString::myQString(const QByteArray& ba)
{
ENTER_FUNCTION;
foreach (auto c, ba) {
#if QT_VERSION_MAJOR == 5
append(QChar(c));
#endif
#if QT_VERSION_MAJOR == 6
append(char(c));
#endif
}
}
The file main.cpp is also "stripped" here and only shows that one specific problem:
#include "myqstring.h"
#include <QDebug>
#include <array>
// -----------------------------------------------------------------------------
// -----------------------------------------------------------------------------
void charPointerToQstring() {
// case 1 - const char* with string as initialiser
const char* buf1("Äa");
myQString mstr1(buf1);
QString qstr1(buf1);
// case 2 - char* with char assignment
const int len = 2;
char* buf2 = new char[len+1];
buf2[0] = char(0xC4); // 0xC4 == 196 == AE (umlaut)
buf2[1] = 'a';
buf2[len] = '\0';
myQString mstr2(buf2);
QString qstr2(buf2);
// case 3 - str
myQString mstr3("Äa");
QString qstr3("Äa");
// case 4 - std::array<char>
std::array<char, len+1> stda1;
stda1[0] = char(0xC4);
stda1[1] = 'a';
stda1[len] = '\0';
myQString mstr4(stda1.data());
QString qstr4(stda1.data());
qDebug() << "Set a breakpoint exactly on ME (nr 3) and check the results via Debugger!!!";
}
// -----------------------------------------------------------------------------
// -----------------------------------------------------------------------------
int main(int argc, char *argv[])
{
Q_UNUSED(argc)
Q_UNUSED(argv)
// missing code with more tests here...
charPointerToQstring();
}
The big question is: Why isn't Qt handling a single char of a char* argument right but a string as argument (with the same info) goes well? If we have a char* as an argument then we can only go for each char from 0x00 to 0xFF (unsinged). Why not make 0x0000 to 0x00FF out of it?
Edit:
The answer of Artyer explains the behavior for buf1 but not for buf2. buf2 is a char[3] { 0xC4, 0x61, '\0' } which get's converted (with Artyers help) to a QString with elements QChar{ 0x00C4, 0x0061 }. So Qt can easily convert those 0xC4 characters to 0x00C4. In fact qstr1 shows that it can convert the two chars { 0xC3, 0x84 } from 'Ä' to one correct QChar {0x00C4}. If we have a char* as an argument then we can only go for each char from 0x00 to 0xFF (unsinged). Why not make 0x0000 to 0x00FF out of it?
And btw, I can't accept that approach yet because it now breaks mstr1 and mstr3. They then get exactly the "same" elements as buf1 but in QChar (so, without the closing '\0', from char[3] { 0xC3, 0x84, 0x61 } to QChar { 0x00C3, 0x0084, 0x0061 } but it should get QChar { 0x00C4, 0x0061 })

Answer 1

What is probably the case is that "Äa" is three UTF-8 encoded bytes in the source file (Equivalent to char[4]{ 0xC3, 0x84, 'a', '\0' }), and the QString constructor expects UTF-8 encoded data.
The 65533 character (U+FFFD) is the replacement character for the invalid UTF-8 data.
Use QString::fromLatin1:
myQString::myQString(const char* p) : QString(QString::fromLatin1(p, std::strlen(p)))
{
ENTER_FUNCTION;
}
myQString::myQString(const QByteArray& ba) : QString(QString::fromLatin1(ba))
{
ENTER_FUNCTION;
}
Also consider using QLatin1StringView instead of char* to avoid getting confused about encoding (might be called QLatin1String in older QT versions)