Qt: Safe parsing of Windows format data under Linux - c++

Question

I have a Server-Client application in which JSON data is send between those. The Client has a Linux and a Windows version, while the Server application runs under Linux.
The Linux Client communicates just find, but I have problems with the Windows Client.
The problematic JSON data contains a text field with an apostrophe. Let's say the content is "a dog`s name", then the Windows client sends this as "a dog\x92s name", while the Linux client goes for "a dog\xE2\x80\x99s name", at least that is what qDebug() shows me.
I parse the JSON data with the lines
QJsonDocument document = QJsonDocument::fromJson(body);
if(document.isArray()) json_data = document.array();
if(document.isObject()) json_data.append(document.object());
where body is a QByteArray and json_data is a QJsonArray.
If the Windows data is fed into this, it seems that the Qt JSON parser does not recognize it as valid JSON and thus json_data end up being empty.
I really don't want to do anything manually with that text specific to those very characters, as I want it not only to work with that apostrophe but with all kinds of special characters that a user might enter in general. Is there some way to handle this in general? I assume the Windows is in something like the Windows-1252 encoding?

Answer 1

I think windows client sends strings encoded in CP1251 or CP1252. And json decoder expects utf-8.
Maybe source code is not in utf-8 and has string literals. Qt4 has QTextCodec::setCodecForCStrings. Qt5 assume string literals encoded in utf-8.
$ echo -n "’" | iconv -f utf-8 -t cp1251 | xxd
00000000: 92
$ echo -n "’" | xxd
00000000: e280 99
If you don't want to fix windows client the proper way (fixing it's output encoding) you can deal with this situation by converting all input from windows client to unicode before building QJsonDocument on server.
QByteArray bodycp1252;
QTextCodec* cp1252 = QTextCodec::codecForName("CP1252");
QTextCodec* utf8 = QTextCodec::codecForName("UTF-8");
QByteArray body = utf8->fromUnicode(cp1252->toUnicode(bodycp1252));
QJsonDocument document = QJsonDocument::fromJson(body);
It's possible to check if QByteArray contains valid utf-8 data with QUtf8::isValidUtf8(const char *chars, qsizetype len) function. It is defined in private headers, so you need to add QT += core-private. Unfortunately implementation is not visible by linker (not exported from QtCore.lib) so you need to add qutfcodec.cpp from qt sources to your project to resolve linker errors.
////////////////// is-valid-utf8.pro
QT -= gui
QT += core core-private
CONFIG += c++11 console
CONFIG -= app_bundle
qt_src = "C:/Qt/5.15.1/Src"
SOURCES += \
main.cpp \
$$qt_src/qtbase/src/corelib/codecs/qutfcodec.cpp
////////////////// main.cpp
#include <QCoreApplication>
#include <private/qutfcodec_p.h>
#include <QTextCodec>
#include <QDebug>
bool isValidUtf8(const QByteArray& data) {
return QUtf8::isValidUtf8(data.data(), data.size()).isValidUtf8;
}
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QTextCodec* utf8 = QTextCodec::codecForName("UTF-8");
QTextCodec* cp1251 = QTextCodec::codecForName("CP1251");
QByteArray utf8data1 = utf8->fromUnicode("Привет мир");
QByteArray cp1251data1 = cp1251->fromUnicode("Привет мир");
QByteArray utf8data2 = utf8->fromUnicode("Hello world");
QByteArray cp1251data2 = cp1251->fromUnicode("Hello world");
Q_ASSERT(isValidUtf8(utf8data1));
Q_ASSERT(isValidUtf8(cp1251data1) == false);
Q_ASSERT(isValidUtf8(utf8data2));
Q_ASSERT(isValidUtf8(cp1251data2));
qDebug() << "test passed";
return 0;
}
source