QJsonDocument::fromJson fails on "foreign" characters

[VRonin]

can you replace jsontext = "{\"string\": \"æøå\"}"; with jsontext =QString::fromWCharArray(L"{\"string\": \"æøå\"}"); or try

jsontext =QString::fromWCharArray(
                L"{\"string\": \""
             L"\u00E6"
             L"\u00F8"
                L"\u00E5"
                L"\"}"
                );

What I mean is that the problem is not how you read the string but how you create it

[Per Gunnar Holm]

@VRonin
Thank you for looking into it!
Unfortunately the result is the same in both cases:

jsontext = "{\"string\": \"æøå\"}"
msg = {"string": "æøå"}
(Broken json), error code = invalid UTF8 string

This is a bit of a surprise to me, I thought the second option would force the characters down to single byte representation?

Here's the actual code I ran (first suggestion commented out):

int main()
{
    QString jsontext = "{\"string\": \"abc\"}";
    textMessageReceived(jsontext);
//    jsontext = QString::fromWCharArray(L"{\"string\": \"æøå\"}");
    jsontext = QString::fromWCharArray(L"{\"string\": \"" L"\u00E6" L"\u00F8" L"\u00E5" L"\"}");
    qDebug() << "jsontext =" << jsontext;
    textMessageReceived(jsontext);

    return 0;
}

[VRonin]

This works for me. Qt 5.5.1 on MSVC2013

#include <QtCore/QCoreApplication>
#include <QtCore/QJsonDocument>

#include <QDebug>

void textMessageReceived(const QString &msg)
{
    qDebug() << "msg = " << msg << '\n';
    QJsonParseError error;

    // Create a Json document from text. Fails for foreign characters!
    QJsonDocument doc = QJsonDocument::fromJson(msg.toUtf8(), &error);

    if(doc.isNull())
        qDebug() << "(Broken json), error code = "
                  << error.errorString() << '\n';
    else
        qDebug() << "(Valid json) : " << QString(doc.toJson()) << '\n';
}

int main()
{

    QString jsontext = "{\"string\": \"abc\"}";
    textMessageReceived(jsontext);

    jsontext = QString::fromWCharArray(
                L"{\"string\": \""
             L"\u00E6"
             L"\u00F8"
                L"\u00E5"
                L"\"}"
                );
    textMessageReceived(jsontext);

    return 0;

}

Output:

msg =  "{\"string\": \"abc\"}" 

(Valid json) :  "{\n    \"string\": \"abc\"\n}\n" 

msg =  "{\"string\": \"æøå\"}" 

(Valid json) :  "{\n    \"string\": \"æøå\"\n}\n" 

[Per Gunnar Holm]

Thanks again @VRonin !

That means this is platform dependent I suppose. As far as I can see there are no differences between your code and mine (except you use qDebug).
It would also explain why there hasn't been a torrent of complaints to Qt if this is a bug and not me making a mistake; probably not too many on my platform.
As stated initially I have Qt 5.6.0 and I'm on Linux (CentOS 6.8).

[VRonin]

Even on Qt5.7 works for me.
2 questions:

• did you try my code?
• What encoding do you use on your .cpp file? (in QtCreator you can see it in edit->select encoding

[Per Gunnar Holm]

@VRonin

Now I did :-)
Same result (unfortunately):

msg =  "{\"string\": \"abc\"}" 
(Valid json) :  "{\n    \"string\": \"abc\"\n}\n" 
msg =  "{\"string\": \"æøå\"}" 
(Broken json), error code =  "invalid UTF8 string" 

I checked the encoding as instructed, and the "Text Encoding" window comes up with "UTF-8" high-lighted.
I am assuming this is OK?
-- Gunnar

[mrjj]

hi
window 7, Qt 5.7 also shows it ok.

Linux Mint, Qt 5.5

Notice it dont render as æøå but still consider it valid.

[Per Gunnar Holm]

Thanks @mrjj
It looks increasingly as if this is a problem specific to my platform.
That's good news for the world, but bad news for me, of course, since it decreases the likelihood of getting it fixed :-(

[Per Gunnar Holm]

Today I have tested on virtual installations of Ubuntu 16.04 and CentOS-7, both using QT 5.6.1.
The test program passes without problems there, so this is definitely a CentOS-6 problem.

[mrjj]

@Per-Gunnar-Holm
well maybe Qt 5.6.1 has a bug on that distro.

I hope Is it an option to use CentOs-7 instead :)

[kshegunov]

@Per-Gunnar-Holm

QByteArray ba (msg.toStdString().c_str());

Bug or no, the above line doesn't seem correct. You should enforce the required encoding (as @VRonin has done) instead of relying on the internal representation of std::string and/or QString.
The above should be:

QByteArray ba = msg.toUtf8();

If you need to output QStrings to the standard streams, attach a QTextStream to them instead of converting the objects to std::string:

QTextStream cout(stdout);
QTextStream cerr(stderr);
QTextStream cin(stdin);

Kind regards.

[Per Gunnar Holm]

Thanks @kshegunov !

The c_str() was just something we tested along the way!
The original code was

    // Create a Json document from text. Fails for foreign characters!
    QJsonDocument doc = QJsonDocument::fromJson(msg.toUtf8(), &error);

However, all the variations we/I have tried display the same problem (on CentOS 6.5, as we have discovered).