Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. QByteArray to QVariantMap while conserving numerical type

QByteArray to QVariantMap while conserving numerical type

Scheduled Pinned Locked Moved Unsolved General and Desktop
qjsonobjectqvariantmapqjsondocumentqbytearrayqjsonvalue
10 Posts 5 Posters 522 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • E Offline
    E Offline
    erwan_.-
    wrote on 8 Aug 2024, 11:46 last edited by
    #1

    Hello all,

    I am trying to convert the content of an HTTP request to a QVariantMap, perform some actions, and then save the QVariantMap as a Json file.
    The QJsonObject allows to do this very easily with the following code snippet:

        QJsonParseError err;
        QJsonDocument body;
    
        body = QJsonDocument::fromJson(request->getRequest()->collectedData(), &err);
        if (!(err.error == QJsonParseError::NoError)) {
            qInfo(LogCat()) << "Could not decode the body into a Json format. Error: " << err.error;
            return;
        };
        QVariantMap userConfig = body.object().toVariantMap();
     
     //then do some processing and save the Json
    
    

    My problem is in the numerical conversions. As Json only has a single 'number' type and does not differentiate 'float' or 'int' (and I believe Qt is following the same rule), this is leading to some issues regarding the type of the values.

    Take for instance the following Json that we might read from the request:

    {
      "pi": 3.14,
      "almost_pi": 3.0 // float
    }
    

    This is correctly read in the QByteArray from the request, (i.e. we read almost_pi as 3.0), but when converting it to a QJsonObject, this would be regarded as:

    {
      "pi": 3.14,
      "almost_pi": 3 // int
    }
    

    Then, if we were to save this as a Json file, this would be saved as presented above.
    From a Json aspect, they are exactly the same. However, from my side this is leading to errors in the next steps of my pipeline, because this file will be read by Python scripts that do make the difference between int and float in Jsons.
    Is there a way around that that does not require to create our own parser ?
    Also, the request I receive could be anything, and does not follow a specific structure, which makes it impossible to use to<Type>() in the QVariantMap.

    Thanks

    J S 2 Replies Last reply 8 Aug 2024, 12:32
    0
    • E erwan_.-
      8 Aug 2024, 11:46

      Hello all,

      I am trying to convert the content of an HTTP request to a QVariantMap, perform some actions, and then save the QVariantMap as a Json file.
      The QJsonObject allows to do this very easily with the following code snippet:

          QJsonParseError err;
          QJsonDocument body;
      
          body = QJsonDocument::fromJson(request->getRequest()->collectedData(), &err);
          if (!(err.error == QJsonParseError::NoError)) {
              qInfo(LogCat()) << "Could not decode the body into a Json format. Error: " << err.error;
              return;
          };
          QVariantMap userConfig = body.object().toVariantMap();
       
       //then do some processing and save the Json
      
      

      My problem is in the numerical conversions. As Json only has a single 'number' type and does not differentiate 'float' or 'int' (and I believe Qt is following the same rule), this is leading to some issues regarding the type of the values.

      Take for instance the following Json that we might read from the request:

      {
        "pi": 3.14,
        "almost_pi": 3.0 // float
      }
      

      This is correctly read in the QByteArray from the request, (i.e. we read almost_pi as 3.0), but when converting it to a QJsonObject, this would be regarded as:

      {
        "pi": 3.14,
        "almost_pi": 3 // int
      }
      

      Then, if we were to save this as a Json file, this would be saved as presented above.
      From a Json aspect, they are exactly the same. However, from my side this is leading to errors in the next steps of my pipeline, because this file will be read by Python scripts that do make the difference between int and float in Jsons.
      Is there a way around that that does not require to create our own parser ?
      Also, the request I receive could be anything, and does not follow a specific structure, which makes it impossible to use to<Type>() in the QVariantMap.

      Thanks

      J Offline
      J Offline
      JonB
      wrote on 8 Aug 2024, 12:32 last edited by
      #2

      @erwan_
      As you say, JSON has a numeric type and no int vs float. Sorry, but that means you are wrong to be attempting to read the bytes in the text file and distinguish between 3 vs 3.0. Either one is possible from JSON.

      because this file will be read by Python scripts that do make the difference between int and float in Jsons.

      I don't understand. If you have a JSON file you would only read it with a JSON library. Even from Python. If you read it with pure Python parsing not JSON parsing you are asking for trouble.

      I really think you are trying to fit a square peg into a round hole if you pursue this way. If you are using JSON stick with what JSON provides you with.

      1 Reply Last reply
      2
      • V Offline
        V Offline
        VRonin
        wrote on 8 Aug 2024, 17:25 last edited by
        #3

        Any chance you can change "almost_pi": 3.0 to "almost_pi": 3.0e+0? that should fix your issue

        "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
        ~Napoleon Bonaparte

        On a crusade to banish setIndexWidget() from the holy land of Qt

        1 Reply Last reply
        0
        • A Online
          A Online
          Axel Spoerl
          Moderators
          wrote on 10 Aug 2024, 07:29 last edited by Axel Spoerl 8 Oct 2024, 07:30
          #4

          It's an equation with one too many unknowns: You know it's a number, but you don't know if the type is int or float.
          That means, you have to build the variant map programmatically and figure out the type programmatically.
          That's actually not a big deal. There are multiple ways do achieve it.

          Assumption:
          If you know that almost_pi is an int, assign that type if you find the JSON key.
          In other words, you rely on the HTTP request to stick to a specific format.
          That's usually done by making the format/version part of the JSON content.

          Heuristics:
          If no specific format is required, just try whether the number fits into an int without deteriorating its value.

          This example implements the heuristic approach:

          QVariantMap YourClass::toVariantMap(const QJsonObject &obj)
          {
              QVariantMap map;
              const QStringList keys = obj.keys();
              for (const auto &key : keys) {
                  const QJsonValue &value = obj.value(key);
                  switch (value.type()) {
                  case QJsonValue::Type::Object:
                      // Recurse, just for the fun of it
                      map.insert(key, toVariantMap(value.toObject());
                      break;
                  case QJsonValue::Type::Array:
                      // Ooops, this code isn't prepared to handle an array
                      break;
                  case QJsonValue::Type::Double: {
                      const int intVal = value.toInt();
                      const QString strVal = value.toString();
                      // Test integer fit
                      if (QString::number(intVal) == strVal)
                          map.insert(key, intVal);
                      else
                          map.insert(key, value.toDouble());
                  }
                      break;
                  case QJsonValue::Type::Bool:
                      map.insert(key, value.toBool());
                      break;
                  case QJsonValue::Type::String:
                      map.insert(key, value.toString());
                      break;
                  case QJsonValue::Type::Undefined:
                      // Handle format error
                      break;
                  case QJsonValue::Type::Null:
                      // Handle or ignore null value
                      break;
                  }
              }
              return map;
          }
          

          Software Engineer
          The Qt Company, Oslo

          J 1 Reply Last reply 10 Aug 2024, 07:46
          0
          • A Axel Spoerl
            10 Aug 2024, 07:29

            It's an equation with one too many unknowns: You know it's a number, but you don't know if the type is int or float.
            That means, you have to build the variant map programmatically and figure out the type programmatically.
            That's actually not a big deal. There are multiple ways do achieve it.

            Assumption:
            If you know that almost_pi is an int, assign that type if you find the JSON key.
            In other words, you rely on the HTTP request to stick to a specific format.
            That's usually done by making the format/version part of the JSON content.

            Heuristics:
            If no specific format is required, just try whether the number fits into an int without deteriorating its value.

            This example implements the heuristic approach:

            QVariantMap YourClass::toVariantMap(const QJsonObject &obj)
            {
                QVariantMap map;
                const QStringList keys = obj.keys();
                for (const auto &key : keys) {
                    const QJsonValue &value = obj.value(key);
                    switch (value.type()) {
                    case QJsonValue::Type::Object:
                        // Recurse, just for the fun of it
                        map.insert(key, toVariantMap(value.toObject());
                        break;
                    case QJsonValue::Type::Array:
                        // Ooops, this code isn't prepared to handle an array
                        break;
                    case QJsonValue::Type::Double: {
                        const int intVal = value.toInt();
                        const QString strVal = value.toString();
                        // Test integer fit
                        if (QString::number(intVal) == strVal)
                            map.insert(key, intVal);
                        else
                            map.insert(key, value.toDouble());
                    }
                        break;
                    case QJsonValue::Type::Bool:
                        map.insert(key, value.toBool());
                        break;
                    case QJsonValue::Type::String:
                        map.insert(key, value.toString());
                        break;
                    case QJsonValue::Type::Undefined:
                        // Handle format error
                        break;
                    case QJsonValue::Type::Null:
                        // Handle or ignore null value
                        break;
                    }
                }
                return map;
            }
            
            J Offline
            J Offline
            JonB
            wrote on 10 Aug 2024, 07:46 last edited by JonB 8 Oct 2024, 07:49
            #5

            @Axel-Spoerl said in QByteArray to QVariantMap while conserving numerical type:

                    const int intVal = value.toInt();
                    const QString strVal = value.toString();
                    // Test integer fit
                    if (QString::number(intVal) == strVal)
            

            But this does not do/does the opposite of what the OP wants. (And you could probably achieve exactly the same more simply without string via conversion via if (value.toInt() != value.toDouble()), with the same problem/flaw.)

            The OP wants that he should be to distinguish the following in the JSON input text:

            "almost_pi": 3.0 // float
            "almost_pi": 3 // int
            

            Using your approach both of these deliver the same "number", because JSON should return (untested) the same QJsonValue::Type::Double value. But their "requirement" is to treat these two differently. Which (so far as I know) can only be done by looking at the actual characters in the JSON input text. (Once JSON has parsed that you just get back a conversion to QJsonValue::Type::Double in all cases.) And that is what the OP wishes to do, which I suggest is not a good idea for a number of reasons. He is not, or should not be, in control of the character output format of the generated JSON, that is up to the JSON library used by the sender and should be allowed to vary as that pleases so long as it is legal JSON.

            A 1 Reply Last reply 10 Aug 2024, 08:08
            2
            • J JonB
              10 Aug 2024, 07:46

              @Axel-Spoerl said in QByteArray to QVariantMap while conserving numerical type:

                      const int intVal = value.toInt();
                      const QString strVal = value.toString();
                      // Test integer fit
                      if (QString::number(intVal) == strVal)
              

              But this does not do/does the opposite of what the OP wants. (And you could probably achieve exactly the same more simply without string via conversion via if (value.toInt() != value.toDouble()), with the same problem/flaw.)

              The OP wants that he should be to distinguish the following in the JSON input text:

              "almost_pi": 3.0 // float
              "almost_pi": 3 // int
              

              Using your approach both of these deliver the same "number", because JSON should return (untested) the same QJsonValue::Type::Double value. But their "requirement" is to treat these two differently. Which (so far as I know) can only be done by looking at the actual characters in the JSON input text. (Once JSON has parsed that you just get back a conversion to QJsonValue::Type::Double in all cases.) And that is what the OP wishes to do, which I suggest is not a good idea for a number of reasons. He is not, or should not be, in control of the character output format of the generated JSON, that is up to the JSON library used by the sender and should be allowed to vary as that pleases so long as it is legal JSON.

              A Online
              A Online
              Axel Spoerl
              Moderators
              wrote on 10 Aug 2024, 08:08 last edited by
              #6

              @JonB
              value.toInt() == value.toDouble() is true for "3.0" and "3".
              The example would always make "3.0" a double.

              That set aside, the solid implementations I have seen solve ambiguous types by format versions or, in complex cases, by additional key/value pairs to specify the format of other keys. It's not entirely clear to me, how the OP intends to interact between Python and C++.
              The example was just meant to illustrate a heuristic implementation.

              Software Engineer
              The Qt Company, Oslo

              J 1 Reply Last reply 10 Aug 2024, 08:14
              0
              • A Axel Spoerl
                10 Aug 2024, 08:08

                @JonB
                value.toInt() == value.toDouble() is true for "3.0" and "3".
                The example would always make "3.0" a double.

                That set aside, the solid implementations I have seen solve ambiguous types by format versions or, in complex cases, by additional key/value pairs to specify the format of other keys. It's not entirely clear to me, how the OP intends to interact between Python and C++.
                The example was just meant to illustrate a heuristic implementation.

                J Offline
                J Offline
                JonB
                wrote on 10 Aug 2024, 08:14 last edited by JonB 8 Oct 2024, 09:31
                #7

                @Axel-Spoerl said in QByteArray to QVariantMap while conserving numerical type:

                The example would always make "3.0" a double.

                I really don't understand. May I ask whether you have actually tested you code against two different lines of input:

                "almost_pi": 3.0
                "almost_pi": 3
                

                You are saying your code distinguishes these two different lines of original text in the JSON are you? I am "surprised", but have not tested. Since I am expecting either 3 or 3.0 in input to deliver identical QJsonValue::Type::Double types and values, but your code relies on them resulting in some difference....

                P.S.

                by additional key/value pairs to specify the format of other key

                That is the only safe and "clean" solution I know of. An alternative, if you really want to receive a different value for a sender floating point versus integer, is to send the values as strings (QJsonValue::Type::String) not numbers (QJsonValue::Type::Double) and do the conversion to int/double yourself accordingly, so that you actually see the incoming string.

                1 Reply Last reply
                1
                • A Online
                  A Online
                  Axel Spoerl
                  Moderators
                  wrote on 10 Aug 2024, 14:03 last edited by
                  #8

                  Turns out that things are a bit different than I had originally thought, sorry for that.
                  QJsonValue::toString() returns QString() for both cases:

                  "almost_pi": 3.0
                  "almost_pi": 3
                  

                  There is no way, to get the difference between both, because numeric JSON values can't be retrieved as a string.
                  That eliminates the heuristic option, and leaves the remaining two: Either interpret almost_pi as a float all the time, or use another key/value pair to specify a type.

                  Thanks @JonB for spotting!

                  Software Engineer
                  The Qt Company, Oslo

                  1 Reply Last reply
                  2
                  • E erwan_.-
                    8 Aug 2024, 11:46

                    Hello all,

                    I am trying to convert the content of an HTTP request to a QVariantMap, perform some actions, and then save the QVariantMap as a Json file.
                    The QJsonObject allows to do this very easily with the following code snippet:

                        QJsonParseError err;
                        QJsonDocument body;
                    
                        body = QJsonDocument::fromJson(request->getRequest()->collectedData(), &err);
                        if (!(err.error == QJsonParseError::NoError)) {
                            qInfo(LogCat()) << "Could not decode the body into a Json format. Error: " << err.error;
                            return;
                        };
                        QVariantMap userConfig = body.object().toVariantMap();
                     
                     //then do some processing and save the Json
                    
                    

                    My problem is in the numerical conversions. As Json only has a single 'number' type and does not differentiate 'float' or 'int' (and I believe Qt is following the same rule), this is leading to some issues regarding the type of the values.

                    Take for instance the following Json that we might read from the request:

                    {
                      "pi": 3.14,
                      "almost_pi": 3.0 // float
                    }
                    

                    This is correctly read in the QByteArray from the request, (i.e. we read almost_pi as 3.0), but when converting it to a QJsonObject, this would be regarded as:

                    {
                      "pi": 3.14,
                      "almost_pi": 3 // int
                    }
                    

                    Then, if we were to save this as a Json file, this would be saved as presented above.
                    From a Json aspect, they are exactly the same. However, from my side this is leading to errors in the next steps of my pipeline, because this file will be read by Python scripts that do make the difference between int and float in Jsons.
                    Is there a way around that that does not require to create our own parser ?
                    Also, the request I receive could be anything, and does not follow a specific structure, which makes it impossible to use to<Type>() in the QVariantMap.

                    Thanks

                    S Offline
                    S Offline
                    SimonSchroeder
                    wrote on 12 Aug 2024, 06:40 last edited by
                    #9

                    @erwan_ said in QByteArray to QVariantMap while conserving numerical type:

                    Then, if we were to save this as a Json file, this would be saved as presented above.

                    Maybe this is your actual problem: Saving the floating point type from C++ writes out 3 instead of 3.0. In that case you need to specify some formatting to make sure the number is written out in a way that makes it clear it is floating point. Because JSON does not distinguish between integer and floating point numbers there is no reason for QJsonDocument to make that distinction by default.

                    J 1 Reply Last reply 12 Aug 2024, 07:42
                    0
                    • S SimonSchroeder
                      12 Aug 2024, 06:40

                      @erwan_ said in QByteArray to QVariantMap while conserving numerical type:

                      Then, if we were to save this as a Json file, this would be saved as presented above.

                      Maybe this is your actual problem: Saving the floating point type from C++ writes out 3 instead of 3.0. In that case you need to specify some formatting to make sure the number is written out in a way that makes it clear it is floating point. Because JSON does not distinguish between integer and floating point numbers there is no reason for QJsonDocument to make that distinction by default.

                      J Offline
                      J Offline
                      JonB
                      wrote on 12 Aug 2024, 07:42 last edited by JonB 8 Dec 2024, 07:43
                      #10

                      @SimonSchroeder said in QByteArray to QVariantMap while conserving numerical type:

                      In that case you need to specify some formatting to make sure the number is written out in a way that makes it clear it is floating point.

                      I don't follow you here. How would any kind of formatting of the number at sender side make any difference to the result received at the receiver side using a JSON parser to read it which will return a "numeric"/QJsonValue::Type::Double? Other than sending it is a string type and doing the numeric conversion at receiver side, which is quite different.

                      1 Reply Last reply
                      1

                      1/10

                      8 Aug 2024, 11:46

                      • Login

                      • Login or register to search.
                      1 out of 10
                      • First post
                        1/10
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • Users
                      • Groups
                      • Search
                      • Get Qt Extensions
                      • Unsolved