'Garbage at the end of the document' error on parsing QJsonDocument

returnx

i want to sending json object over tcp socket and, i serializing the json object like this :

void message_shooter::send_packet(QJsonObject packet)
{
    QJsonDocument json_doc(packet);
    //QByteArray packet_bytes=json_doc.toBinaryData();
    QByteArray packet_bytes=json_doc.toJson();
    m_client_socket->write(packet_bytes);
    m_client_socket->waitForBytesWritten();
    m_client_socket->flush();
}

and i reading data like this :

void connectivity_layer::on_socket_ready_read()
{
    qDebug()<<"Ready read...";
    qint64 incomming_packet_size=m_socket->bytesAvailable();
    if (incomming_packet_size > 0)
    {
        QByteArray received_bytes=m_socket->read(incomming_packet_size);
        m_buffer.append(received_bytes);

        m_buffer.simplified();
        QJsonParseError parse_error;
        QJsonDocument json_doc=QJsonDocument::fromJson(m_buffer,&parse_error);

        if (parse_error.error == QJsonParseError::NoError)
        {
            QJsonObject i_packet=json_doc.object();
            m_buffer.clear();
            analys_packet(i_packet);

        }
        else
        {
            qDebug()<<"Error!";
            qDebug()<<QString::fromUtf8(m_buffer);
            qDebug()<<parse_error.errorString();
        }
    }
}

But this doesn't work and print 'Garbage at the end of the document' error! [when the data received completely]
how can i solve this problem?

mrjj

Hi
code look ok , and you seem to append data until
all have been received.
Have you tried to dump the incoming data so you
can see what is wrong with document?

returnx

@mrjj hi,
i receive this:

"{\"contacts\":[{\"first_name\":\"\",\"last_name\":\"\",\"nick_name\":\"test\",\"user_name\":\"test\"}],\"event\":1,\"type\":7}{\"conversations\":[{\"id\":68,\"members\":[\"test\"],\"starter\":\"test1\",\"title\":\"\",\"type\":1}],\"event\":3,\"type\":7}"
"garbage at the end of the document"

but i expect see this , if contacts object parse without any error:

{\"conversations\":[{\"id\":68,\"members\":[\"test\"],\"starter\":\"test1\",\"title\":\"\",\"type\":1}],\"event\":3,\"type\":7}"
"garbage at the end of the document"

and if contacts object parse with error:

"{\"contacts\":[{\"first_name\":\"\",\"last_name\":\"\",\"nick_name\":\"test\",\"user_name\":\"test\"}],\"event\":1,\"type\":7}
"Any error report"

because i send tow separated request for getting contacts and conversations !

mrjj

@returnx
hmm but there seems to be no garbage at end.
is that directly from m_buffer ?

kshegunov

@returnx
Hello,
If you parse the data each time you get a read on your socket, and if your data doesn't come into one packet you'll naturally get an error. You could send an integer from the server side before sending the data through the socket, so on the client side you know how much data to expect. Then you read the data, and only when all the data has arrived (your buffer has a size that matches the integer you had sent) you parse the JSON data. Additionally I see no reason to call QByteArray::simplified(); on the client side at all, if a call is to be made, it certainly should be put before sending the data. Moreover, you probably are getting several packets as one (sockets have buffers and there is no guarantee that what you send as separate write calls will be read from separate reads). That's why you have to implement a simple protocol by which your server and client will communicate.

Kind regards.

returnx

@mrjj
"is that directly from m_buffer ?"
yes! i'm just copied from Qt Creator application output window and paste here!

mrjj

@returnx
ok seems fine.
As @kshegunov is talking about,

Are you sure this error comes even when all data have been sent/read?
It will/might come in several blocks so on_socket_ready_read() will be called
more than one time for say "contacts"

If you check packet_bytes.size() and
m_socket->bytesAvailable(); ( when reading) , do they match up?
also please look at
http://doc.qt.io/qt-5/qjsonparseerror.html#offset-var
to get hint of where it think error is.

returnx

@mrjj
i checked it!
when i write tow time,for example size of the first data block is 123 bytes and size of the second data is 118,on client side i received total of the data (241) just one time!

and for this case:

"{\"contacts\":[{\"first_name\":\"test\",\"last_name\":\"test\",\"nick_name\":\"test\",\"user_name\":\"test\"}],\"event\":1,\"type\":7}{\"conversations\":[{\"id\":68,\"members\":[\"test1\"],\"starter\":\"test\",\"title\":\"\",\"type\":1}],\"event\":3,\"type\":7}"

error offset is :112

mrjj

@returnx
well it seems u get both docs in same string
so there is 2 JSON root elements which I think it will not like
if u paste it into
https://jsonformatter.curiousconcept.com/
you will see it not happy with that so my guess
is that the qt json parser don't like either.
so you should not send them as you do now.
Or at least do as @kshegunov suggest and make a small protocol so u know which is which.

{  
   "contacts":[  
      {  
         "first_name":"test",
         "last_name":"test",
         "nick_name":"test",
         "user_name":"test"
      }
   ],
   "event":1,
   "type":7
}
/////// new root 
{  
   "conversations":[  
      {  
         "id":68,
         "members":[  
            "test1"
         ],
         "starter":"test",
         "title":"",
         "type":1
      }
   ],
   "event":3,
   "type":7
}

kshegunov

@returnx
Hello,
Then your data is buffered before sending/receiving (either on server or client side, it doesn't really matter). This is very characteristic of network communications. The simplest way to solve your problem is either to send a header with each write (as my suggestion for an integer) or split the written data by a special character, for example the NULL character should suffice.

Kind regards.

returnx

I appreciate for your helps...
i choose second approach because i have short time for implementation...

mrjj

@returnx
Ok. super. please mark as solved :)

Final note:
When you deploy,
there might be more reads for full json string.
The code you shown, will try to parse on each read. Make sure the code can
handle that it comes in blocks and not bail out if parse fails.