How to write entire QVector to a binary file?

CJha

Hi, I am trying to write a QVector<double> to a binary file using QDataStream. According to Qt documentation

Reading and Writing Qt Collection Classes
The Qt container classes can also be serialized to a QDataStream. These include QList, QLinkedList, QVector, QSet, QHash, and QMap. The stream operators are declared as non-members of the classes.

When I do something like this:

QFile file("BinTest.bin");
file.open(QIODevice::WriteOnly);
QDataStream out(&file);
QVector<double> vec;
for(int ii = 0; ii < 10; ++ii){
    vec << ii * 0.33;
}
out << vec;
file.close();

The output contains garbage. I tried serializing:

QDataStream& operator <<(QDataStream& out, QVector<double> vec);

But this is not working as well. I know I can write data to my file from QVector by going through it one by one, but my data vectors are way too long (in millions) and so I am looking for a faster way to write it to a binary file. Is there any way I can write an entire QVector to a binary file without iterating through it?

J.Hilk

@CJha it works, you see "garbage" because there is a header to identify it as a QVector, how many entries the type and what not. You're serializing a complex class and not an array

CJha

@J-Hilk I am unable to find any examples to tell me how to get the header. The QDataStream class description only says just a little about serializing QVectors do you know any examples that I can follow so that I have an idea how to read my binary file?

Christian Ehrlicher

@CJha said in How to write entire QVector to a binary file?:

so that I have an idea how to read my binary file?

Read it with QDataStream. The format is internal.

CJha

@Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

Christian Ehrlicher

@CJha said in How to write entire QVector to a binary file?:

I have to read it in Matlab.

Then create your own format or check what matlab needs to read it with it's built-in functions.

JonB

@CJha
Then do not use a Qt serialization operator! Use whatever is going to be acceptable at the MatLab side, e.g. just a plain array's-worth of the data, or whatever. Qt serialization is for Qt programs!

J.Hilk

@CJha you can take a look into the source code

no idea what version you're using, but I found this

link

Christian Ehrlicher

@J-Hilk But this is internal and may change (although the chance is very very low)

CJha

@J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
@Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

Christian Ehrlicher

@CJha said in How to write entire QVector to a binary file?:

and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

VRonin

@CJha said in How to write entire QVector to a binary file?:

I have to read it in Matlab.

If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

QFile file("BinTest.bin");
file.open(QIODevice::WriteOnly | QIODevice::Text);
QTextStream out(&file);
QVector<double> vec;
for(int ii = 0; ii < 10; ++ii){
    vec << ii * 0.33;
}
for(auto&& val : vec)
out << val << ',';
file.close();

CJha

@Christian-Ehrlicher I didn't know that before I saw the source code for QDataStream. I assumed since for a QVector data points are in adjacent memory positions pushing an entire vector into a binary file must be faster than iterating over it.

JonB

@CJha said in How to write entire QVector to a binary file?:

The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points

Yep. That's what has to be done, and it's what the << serializer does, as @J-Hilk showed you. The only other way would be if you can get the address of contiguous QVector memory and save from there, which I'm guessing can be done. However...

...If @VRonin's latest post is correct and you're supposed to produce text instead to export then you cannot help but do it one-by-one....

P.S.
BTW, you'd have to test, but my guess is that code to output data points one-by-one instead of in a contiguous clump is not what would be slow over 1,000,000 points --- rather, the size of the output written to file will be what is significant....

Christian Ehrlicher

You're correct with the adjacent memory but QVector can also hold objects so memcpy'ing it out will not work there. You can use memcpy if you want in your case but QDataStream is generic and has no optimizations for such things.

CJha

@VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

JonB

@CJha
OK, so STOP and tell us exactly what format MatLab will accept, and take it from there...

jsulm

@CJha But does Matlab support any binary format? You have to write in a format supported by Matlab.

J.Hilk

@CJha said in How to write entire QVector to a binary file?:

I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

actually stop right here!

if this program is used more than once, you're going to destroy your HD/SSD very quickly!

I'm sure there's an other - in memory - way to hand over those data points

CJha

@JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on