Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. QJsonDocument::fromJson fails on "foreign" characters
Forum Updated to NodeBB v4.3 + New Features

QJsonDocument::fromJson fails on "foreign" characters

Scheduled Pinned Locked Moved Unsolved General and Desktop
qjsondocumentutf8fromjson
17 Posts 4 Posters 10.7k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • V Offline
    V Offline
    VRonin
    wrote on 17 Aug 2016, 14:56 last edited by VRonin
    #6

    can you replace jsontext = "{\"string\": \"æøå\"}"; with jsontext =QString::fromWCharArray(L"{\"string\": \"æøå\"}"); or try

    jsontext =QString::fromWCharArray(
                    L"{\"string\": \""
                 L"\u00E6"
                 L"\u00F8"
                    L"\u00E5"
                    L"\"}"
                    );
    

    What I mean is that the problem is not how you read the string but how you create it

    "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
    ~Napoleon Bonaparte

    On a crusade to banish setIndexWidget() from the holy land of Qt

    P 1 Reply Last reply 17 Aug 2016, 15:17
    2
    • V VRonin
      17 Aug 2016, 14:56

      can you replace jsontext = "{\"string\": \"æøå\"}"; with jsontext =QString::fromWCharArray(L"{\"string\": \"æøå\"}"); or try

      jsontext =QString::fromWCharArray(
                      L"{\"string\": \""
                   L"\u00E6"
                   L"\u00F8"
                      L"\u00E5"
                      L"\"}"
                      );
      

      What I mean is that the problem is not how you read the string but how you create it

      P Offline
      P Offline
      Per Gunnar Holm
      wrote on 17 Aug 2016, 15:17 last edited by
      #7

      @VRonin
      Thank you for looking into it!
      Unfortunately the result is the same in both cases:

      jsontext = "{\"string\": \"æøå\"}"
      msg = {"string": "æøå"}
      (Broken json), error code = invalid UTF8 string
      

      This is a bit of a surprise to me, I thought the second option would force the characters down to single byte representation?

      Here's the actual code I ran (first suggestion commented out):

      int main()
      {
          QString jsontext = "{\"string\": \"abc\"}";
          textMessageReceived(jsontext);
      //    jsontext = QString::fromWCharArray(L"{\"string\": \"æøå\"}");
          jsontext = QString::fromWCharArray(L"{\"string\": \"" L"\u00E6" L"\u00F8" L"\u00E5" L"\"}");
          qDebug() << "jsontext =" << jsontext;
          textMessageReceived(jsontext);
      
          return 0;
      }
      
      1 Reply Last reply
      0
      • V Offline
        V Offline
        VRonin
        wrote on 17 Aug 2016, 15:24 last edited by VRonin
        #8

        This works for me. Qt 5.5.1 on MSVC2013

        #include <QtCore/QCoreApplication>
        #include <QtCore/QJsonDocument>
        
        #include <QDebug>
        
        void textMessageReceived(const QString &msg)
        {
            qDebug() << "msg = " << msg << '\n';
            QJsonParseError error;
        
            // Create a Json document from text. Fails for foreign characters!
            QJsonDocument doc = QJsonDocument::fromJson(msg.toUtf8(), &error);
        
            if(doc.isNull())
                qDebug() << "(Broken json), error code = "
                          << error.errorString() << '\n';
            else
                qDebug() << "(Valid json) : " << QString(doc.toJson()) << '\n';
        }
        
        int main()
        {
        
            QString jsontext = "{\"string\": \"abc\"}";
            textMessageReceived(jsontext);
        
            jsontext = QString::fromWCharArray(
                        L"{\"string\": \""
                     L"\u00E6"
                     L"\u00F8"
                        L"\u00E5"
                        L"\"}"
                        );
            textMessageReceived(jsontext);
        
            return 0;
        
        }
        

        Output:

        msg =  "{\"string\": \"abc\"}" 
        
        (Valid json) :  "{\n    \"string\": \"abc\"\n}\n" 
        
        msg =  "{\"string\": \"æøå\"}" 
        
        (Valid json) :  "{\n    \"string\": \"æøå\"\n}\n" 
        

        "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
        ~Napoleon Bonaparte

        On a crusade to banish setIndexWidget() from the holy land of Qt

        P 1 Reply Last reply 17 Aug 2016, 15:38
        5
        • V VRonin
          17 Aug 2016, 15:24

          This works for me. Qt 5.5.1 on MSVC2013

          #include <QtCore/QCoreApplication>
          #include <QtCore/QJsonDocument>
          
          #include <QDebug>
          
          void textMessageReceived(const QString &msg)
          {
              qDebug() << "msg = " << msg << '\n';
              QJsonParseError error;
          
              // Create a Json document from text. Fails for foreign characters!
              QJsonDocument doc = QJsonDocument::fromJson(msg.toUtf8(), &error);
          
              if(doc.isNull())
                  qDebug() << "(Broken json), error code = "
                            << error.errorString() << '\n';
              else
                  qDebug() << "(Valid json) : " << QString(doc.toJson()) << '\n';
          }
          
          int main()
          {
          
              QString jsontext = "{\"string\": \"abc\"}";
              textMessageReceived(jsontext);
          
              jsontext = QString::fromWCharArray(
                          L"{\"string\": \""
                       L"\u00E6"
                       L"\u00F8"
                          L"\u00E5"
                          L"\"}"
                          );
              textMessageReceived(jsontext);
          
              return 0;
          
          }
          

          Output:

          msg =  "{\"string\": \"abc\"}" 
          
          (Valid json) :  "{\n    \"string\": \"abc\"\n}\n" 
          
          msg =  "{\"string\": \"æøå\"}" 
          
          (Valid json) :  "{\n    \"string\": \"æøå\"\n}\n" 
          
          P Offline
          P Offline
          Per Gunnar Holm
          wrote on 17 Aug 2016, 15:38 last edited by
          #9

          Thanks again @VRonin !

          That means this is platform dependent I suppose. As far as I can see there are no differences between your code and mine (except you use qDebug).
          It would also explain why there hasn't been a torrent of complaints to Qt if this is a bug and not me making a mistake; probably not too many on my platform.
          As stated initially I have Qt 5.6.0 and I'm on Linux (CentOS 6.8).

          1 Reply Last reply
          0
          • V Offline
            V Offline
            VRonin
            wrote on 17 Aug 2016, 15:45 last edited by
            #10

            Even on Qt5.7 works for me.
            2 questions:

            • did you try my code?
            • What encoding do you use on your .cpp file? (in QtCreator you can see it in edit->select encoding

            "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
            ~Napoleon Bonaparte

            On a crusade to banish setIndexWidget() from the holy land of Qt

            P M 2 Replies Last reply 17 Aug 2016, 15:52
            0
            • V VRonin
              17 Aug 2016, 15:45

              Even on Qt5.7 works for me.
              2 questions:

              • did you try my code?
              • What encoding do you use on your .cpp file? (in QtCreator you can see it in edit->select encoding
              P Offline
              P Offline
              Per Gunnar Holm
              wrote on 17 Aug 2016, 15:52 last edited by
              #11

              @VRonin

              Now I did :-)
              Same result (unfortunately):

              msg =  "{\"string\": \"abc\"}" 
              (Valid json) :  "{\n    \"string\": \"abc\"\n}\n" 
              msg =  "{\"string\": \"æøå\"}" 
              (Broken json), error code =  "invalid UTF8 string" 
              

              I checked the encoding as instructed, and the "Text Encoding" window comes up with "UTF-8" high-lighted.
              I am assuming this is OK?
              -- Gunnar

              1 Reply Last reply
              0
              • V VRonin
                17 Aug 2016, 15:45

                Even on Qt5.7 works for me.
                2 questions:

                • did you try my code?
                • What encoding do you use on your .cpp file? (in QtCreator you can see it in edit->select encoding
                M Offline
                M Offline
                mrjj
                Lifetime Qt Champion
                wrote on 18 Aug 2016, 06:15 last edited by mrjj
                #12

                hi
                window 7, Qt 5.7 also shows it ok.

                Linux Mint, Qt 5.5

                Notice it dont render as æøå but still consider it valid.

                P 1 Reply Last reply 18 Aug 2016, 09:35
                1
                • M mrjj
                  18 Aug 2016, 06:15

                  hi
                  window 7, Qt 5.7 also shows it ok.

                  Linux Mint, Qt 5.5

                  Notice it dont render as æøå but still consider it valid.

                  P Offline
                  P Offline
                  Per Gunnar Holm
                  wrote on 18 Aug 2016, 09:35 last edited by
                  #13

                  Thanks @mrjj
                  It looks increasingly as if this is a problem specific to my platform.
                  That's good news for the world, but bad news for me, of course, since it decreases the likelihood of getting it fixed :-(

                  1 Reply Last reply
                  0
                  • P Offline
                    P Offline
                    Per Gunnar Holm
                    wrote on 18 Aug 2016, 12:48 last edited by
                    #14

                    Today I have tested on virtual installations of Ubuntu 16.04 and CentOS-7, both using QT 5.6.1.
                    The test program passes without problems there, so this is definitely a CentOS-6 problem.

                    M K 2 Replies Last reply 18 Aug 2016, 12:52
                    1
                    • P Per Gunnar Holm
                      18 Aug 2016, 12:48

                      Today I have tested on virtual installations of Ubuntu 16.04 and CentOS-7, both using QT 5.6.1.
                      The test program passes without problems there, so this is definitely a CentOS-6 problem.

                      M Offline
                      M Offline
                      mrjj
                      Lifetime Qt Champion
                      wrote on 18 Aug 2016, 12:52 last edited by
                      #15

                      @Per-Gunnar-Holm
                      well maybe Qt 5.6.1 has a bug on that distro.

                      I hope Is it an option to use CentOs-7 instead :)

                      1 Reply Last reply
                      0
                      • P Per Gunnar Holm
                        18 Aug 2016, 12:48

                        Today I have tested on virtual installations of Ubuntu 16.04 and CentOS-7, both using QT 5.6.1.
                        The test program passes without problems there, so this is definitely a CentOS-6 problem.

                        K Offline
                        K Offline
                        kshegunov
                        Moderators
                        wrote on 18 Aug 2016, 14:02 last edited by kshegunov
                        #16

                        @Per-Gunnar-Holm

                        QByteArray ba (msg.toStdString().c_str());
                        

                        Bug or no, the above line doesn't seem correct. You should enforce the required encoding (as @VRonin has done) instead of relying on the internal representation of std::string and/or QString.
                        The above should be:

                        QByteArray ba = msg.toUtf8();
                        

                        If you need to output QStrings to the standard streams, attach a QTextStream to them instead of converting the objects to std::string:

                        QTextStream cout(stdout);
                        QTextStream cerr(stderr);
                        QTextStream cin(stdin);
                        

                        Kind regards.

                        Read and abide by the Qt Code of Conduct

                        P 1 Reply Last reply 18 Aug 2016, 14:45
                        2
                        • K kshegunov
                          18 Aug 2016, 14:02

                          @Per-Gunnar-Holm

                          QByteArray ba (msg.toStdString().c_str());
                          

                          Bug or no, the above line doesn't seem correct. You should enforce the required encoding (as @VRonin has done) instead of relying on the internal representation of std::string and/or QString.
                          The above should be:

                          QByteArray ba = msg.toUtf8();
                          

                          If you need to output QStrings to the standard streams, attach a QTextStream to them instead of converting the objects to std::string:

                          QTextStream cout(stdout);
                          QTextStream cerr(stderr);
                          QTextStream cin(stdin);
                          

                          Kind regards.

                          P Offline
                          P Offline
                          Per Gunnar Holm
                          wrote on 18 Aug 2016, 14:45 last edited by
                          #17

                          Thanks @kshegunov !

                          The c_str() was just something we tested along the way!
                          The original code was

                              // Create a Json document from text. Fails for foreign characters!
                              QJsonDocument doc = QJsonDocument::fromJson(msg.toUtf8(), &error);
                          

                          However, all the variations we/I have tried display the same problem (on CentOS 6.5, as we have discovered).

                          1 Reply Last reply
                          0

                          15/17

                          18 Aug 2016, 12:52

                          • Login

                          • Login or register to search.
                          15 out of 17
                          • First post
                            15/17
                            Last post
                          0
                          • Categories
                          • Recent
                          • Tags
                          • Popular
                          • Users
                          • Groups
                          • Search
                          • Get Qt Extensions
                          • Unsolved