QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior

[oldevel]

Hi,

I am experiencing some behavior in the included code which I find odd. I am somewhat new to Qt and am hoping someone with in depth knowledge can assist me with this.

This function reads a text file which contains hex as presented in the comment above the function. It iterates the opened QTextStream, reading line by line, until the end of file.

The intension is to translate each textual line of hex into an array of bytes by using the QByteArray::fromHex(QString.toUtf8()) conversion.

Upon inspection of the resulting QByteArray ba, the very first value is incorrect. I expect 177d or 7Fh, yet the array presents the value 0x07 and the second character is 0xF4 instead of 0x46. The rest of the conversion is corrupt, as all characters are shifted right by 1.

I have also attempted to iterate over the line of text in the hexstring QString, yet the last value which is represented by 80 becomes corrupt, and is converted to 0x08 and '\0' , which is definitely not desired.

Any light shed on these issues and how to correct them will be appreciated.

Thanks.

// Test file contents
// 7F46000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F30313233343536373839380
// 7F46000202030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F30313233343536373839380
//
QList<QByteArray> FileIO::LoadFile()
{
    QString filePath;
    int lastSeparatorIndex = 0;
    QString fileName = "";
    QString feedback;
    QString hexstring;
    QString hexchar;
    int i, j = 0;
    QByteArray ba;

    ba.clear();

    // open a file dialog which allows the user to browse to and select a file
    filePath = QFileDialog::getOpenFileName(nullptr, tr("Select Data File"),
                            "C:/", tr("*.txt"), nullptr, QFileDialog::ReadOnly);

    // continue if a file path is returned
    if (filePath != NULL)
    {
        // continue if the file path contains a separator
        if (filePath.contains('/') == true)
        {
            // get the index of the last separator ie. before the file name
            lastSeparatorIndex = filePath.lastIndexOf('/');

            // the index is valid
            if (lastSeparatorIndex != -1)
            {
                // extract the file name from the returned path
                fileName = filePath.right(filePath.length() - (lastSeparatorIndex + 1));

                // the filename is valid
                if (fileName != "")
                {
                    // return the fileName to the UI
                    emit fileIOUiFileNameMessage(fileName);
					
                    // initialize a file
                    QFile dataFile(filePath);

                    // open the file, error if not available
                    if (!dataFile.open(QIODevice::ReadOnly | QIODevice::Text))
                    {
                        feedback = "FileIO: Data file failed to open";
                    }
                    else
                    {
                        // read the data file
                        QTextStream textStream(&dataFile);
                        // append to container line by line
                        while (!textStream.atEnd())
                        {
                            hexstring = textStream.readLine();
                            ba.append(QByteArray::fromHex(hexstring.toUtf8()));
                        }
                        // done with the file
                        dataFile.close();

                        // assign feedback
                        if (!ba.isEmpty())
                        {
                            feedback = QString("File with %1 records loaded").arg(ba.count());
                            emit fileIOFileLoaded(ba.count());
                        }
                        else
                        {
                            feedback = "FileIO: File read returned no data";
                        }
                    }
                }
                else
                {
                    feedback = "FileIO: No file selected";
                }
            }
            else
            {
                feedback = "FileIO: File name could not be extracted";
            }
        }
        else
        {
            feedback = "FileIO: File separator not found in file path";
        }
    }
    else
    {
        feedback = "FileIO: File path is empty";
    }

    emit fileIOUiMessage(feedback);

    return ba;
}

[Christian Ehrlicher]

Your hex string has an odd number of characters.

[JonB]

@oldevel said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

The rest of the conversion is corrupt, as all characters are shifted right by 1.

We only have your word for what you say is in the file, we cannot see. For all we know, just maybe there is a byte marker or something.

So help yourself to help us:

while (!textStream.atEnd())
{
    hexstring = textStream.readLine();
    qDebug() << hexString;
    auto hexstring2 = hexstring.toUtf8();
    qDebug() << hexstring2;
    auto ba2 = QByteArray::fromHex(hexstring2);
    ba.append(ba2);
}

Meanwhile, I would replace your reading from file with a literal

hexstring = "7F46000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F30313233343536373839380";

and confirm behaviour with known input.

UPDATE
Crossed with @Christian-Ehrlicher's observation, obviously that is what needs attending to!

[JonB]

@Christian-Ehrlicher said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

Your hex string has an odd number of characters.

Interesting (you are good at counting!) :) Slightly strange that causes it to shift at left rather than at right?

[Christian Ehrlicher]

@JonB said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

Slightly strange that causes it to shift at left rather than at right?

The he decoding starts from the back. Don't know why though but the documentation is explicit: Input is not checked for validity

[JonB]

@Christian-Ehrlicher said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

he decoding starts from the back.

I sit and stare at

    hexstring = textStream.readLine();
    ba.append(QByteArray::fromHex(hexstring.toUtf8()));

and just cannot see that!? :(

Though anyway of course it doesn't matter, he needs to fix the character count.

[Christian Ehrlicher]

@JonB said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

and just cannot see that!? :(

Forgot a Tat the start :D

[KroMignon]

@JonB said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

I sit and stare at
hexstring = textStream.readLine();
ba.append(QByteArray::fromHex(hexstring.toUtf8()));

and just cannot see that!? :(

when reading from stream, you got an QByteArray.
You have to transform it to QString to be able to use QByteArray::fromHex().

I would to it with hexstring.toLatin1() but it doesn't really matter, because there are only '0'-'9' or 'A'-'F' or 'a'-'f' bytes.

[Christian Ehrlicher]

@KroMignon said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

when reading from stream

I wouldn't use QTextStream at all but QIODevice::readLine() to avoid the two useless conversions completely :)

[oldevel]

Wow, many thanks for your reply's and suggestions. I will provide further observations shortly.

[JonB]

@KroMignon , @Christian-Ehrlicher
Sorry, I just don't see what either of you are saying. Nothing to do with QByteArray versus QString. I simply don't understand where @Christian-Ehrlicher said:

[T]he decoding starts from the back.

as an answer to why the OP says the first byte he gets, which he & I expect to be the 7F at the left-hand side of the string, comes out as 0x07 [and the second character is 0xF4]

With an odd number of characters in the string, I would have expected the first byte to be 7F and it to go wrong at the right-hand end of the string.

Since you two seem to understand and I do not, we can leave this if you wish...

[KroMignon]

@JonB said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

Since you two seem to understand and I do not, we can leave this if you wish...

If you take a look at QByteArray::fromHex() source code, you will see that the string if read backwards.
This is why he got this.

[Christian Ehrlicher]

@JonB said in QByteArray::fromHex(QString.toUtf8()) presenting undesired behavior:

I simply don't understand where @Christian-Ehrlicher said:

Hehe.
It was the only meaningful explanation for the described problem and when you look at the code (hey, it's opensource ;)) you see that it's the correct one.

[JonB]

@KroMignon , @Christian-Ehrlicher

If you take a look at QByteArray::fromHex() source code, you will see that the string if read backwards.

Indeed, that makes sense for behaviour reported, I merely expressed my surprise as it was not the direction I expected. I expected it would naturally work left-to-right, that's all!

So with a missing byte at the end the code does not allow decoding of all the bytes up to the last one, so you can mostly see what is in there, instead it makes them all wrong if, say, the input is prematurely curtailed, for whatever reason. Potentially a shame/confusing. That's all. I agree the

the documentation is explicit: Input is not checked for validity

means it can do what it likes with this bad input, as I say I was merely surprised that it does turn out to do right-to-left.....

[oldevel]

I gained insight from all your posts. It turns out that simply correcting the line length sorted the problem, thank you.