Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. How to write entire QVector to a binary file?
Forum Updated to NodeBB v4.3 + New Features

How to write entire QVector to a binary file?

Scheduled Pinned Locked Moved Unsolved General and Desktop
qvectorbinary formatqdatastream
46 Posts 8 Posters 13.1k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Christian EhrlicherC Offline
    Christian EhrlicherC Offline
    Christian Ehrlicher
    Lifetime Qt Champion
    wrote on last edited by
    #6

    @CJha said in How to write entire QVector to a binary file?:

    I have to read it in Matlab.

    Then create your own format or check what matlab needs to read it with it's built-in functions.

    Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
    Visit the Qt Academy at https://academy.qt.io/catalog

    1 Reply Last reply
    2
    • CJhaC CJha

      @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

      JonBJ Offline
      JonBJ Offline
      JonB
      wrote on last edited by JonB
      #7

      @CJha
      Then do not use a Qt serialization operator! Use whatever is going to be acceptable at the MatLab side, e.g. just a plain array's-worth of the data, or whatever. Qt serialization is for Qt programs!

      1 Reply Last reply
      0
      • CJhaC CJha

        @J-Hilk I am unable to find any examples to tell me how to get the header. The QDataStream class description only says just a little about serializing QVectors do you know any examples that I can follow so that I have an idea how to read my binary file?

        J.HilkJ Offline
        J.HilkJ Offline
        J.Hilk
        Moderators
        wrote on last edited by J.Hilk
        #8

        @CJha you can take a look into the source code

        no idea what version you're using, but I found this

        00717b7a-3822-470e-a26a-220eb197ba6e-image.png

        066e8007-1c4a-4394-aec7-7138631cfe9e-image.png

        link


        Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


        Q: What's that?
        A: It's blue light.
        Q: What does it do?
        A: It turns blue.

        Christian EhrlicherC 1 Reply Last reply
        0
        • J.HilkJ J.Hilk

          @CJha you can take a look into the source code

          no idea what version you're using, but I found this

          00717b7a-3822-470e-a26a-220eb197ba6e-image.png

          066e8007-1c4a-4394-aec7-7138631cfe9e-image.png

          link

          Christian EhrlicherC Offline
          Christian EhrlicherC Offline
          Christian Ehrlicher
          Lifetime Qt Champion
          wrote on last edited by
          #9

          @J-Hilk But this is internal and may change (although the chance is very very low)

          Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
          Visit the Qt Academy at https://academy.qt.io/catalog

          1 Reply Last reply
          1
          • CJhaC Offline
            CJhaC Offline
            CJha
            wrote on last edited by
            #10

            @J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
            @Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

            JonBJ 1 Reply Last reply
            0
            • Christian EhrlicherC Offline
              Christian EhrlicherC Offline
              Christian Ehrlicher
              Lifetime Qt Champion
              wrote on last edited by
              #11

              @CJha said in How to write entire QVector to a binary file?:

              and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

              Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

              Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
              Visit the Qt Academy at https://academy.qt.io/catalog

              CJhaC 1 Reply Last reply
              1
              • CJhaC CJha

                @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

                VRoninV Offline
                VRoninV Offline
                VRonin
                wrote on last edited by
                #12

                @CJha said in How to write entire QVector to a binary file?:

                I have to read it in Matlab.

                If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

                QFile file("BinTest.bin");
                file.open(QIODevice::WriteOnly | QIODevice::Text);
                QTextStream out(&file);
                QVector<double> vec;
                for(int ii = 0; ii < 10; ++ii){
                    vec << ii * 0.33;
                }
                for(auto&& val : vec)
                out << val << ',';
                file.close();
                

                "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
                ~Napoleon Bonaparte

                On a crusade to banish setIndexWidget() from the holy land of Qt

                CJhaC 1 Reply Last reply
                1
                • Christian EhrlicherC Christian Ehrlicher

                  @CJha said in How to write entire QVector to a binary file?:

                  and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                  Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

                  CJhaC Offline
                  CJhaC Offline
                  CJha
                  wrote on last edited by
                  #13

                  @Christian-Ehrlicher I didn't know that before I saw the source code for QDataStream. I assumed since for a QVector data points are in adjacent memory positions pushing an entire vector into a binary file must be faster than iterating over it.

                  1 Reply Last reply
                  0
                  • CJhaC CJha

                    @J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
                    @Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                    JonBJ Offline
                    JonBJ Offline
                    JonB
                    wrote on last edited by JonB
                    #14

                    @CJha said in How to write entire QVector to a binary file?:

                    The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points

                    Yep. That's what has to be done, and it's what the << serializer does, as @J-Hilk showed you. The only other way would be if you can get the address of contiguous QVector memory and save from there, which I'm guessing can be done. However...

                    ...If @VRonin's latest post is correct and you're supposed to produce text instead to export then you cannot help but do it one-by-one....

                    P.S.
                    BTW, you'd have to test, but my guess is that code to output data points one-by-one instead of in a contiguous clump is not what would be slow over 1,000,000 points --- rather, the size of the output written to file will be what is significant....

                    1 Reply Last reply
                    0
                    • Christian EhrlicherC Offline
                      Christian EhrlicherC Offline
                      Christian Ehrlicher
                      Lifetime Qt Champion
                      wrote on last edited by
                      #15

                      You're correct with the adjacent memory but QVector can also hold objects so memcpy'ing it out will not work there. You can use memcpy if you want in your case but QDataStream is generic and has no optimizations for such things.

                      Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                      Visit the Qt Academy at https://academy.qt.io/catalog

                      1 Reply Last reply
                      3
                      • VRoninV VRonin

                        @CJha said in How to write entire QVector to a binary file?:

                        I have to read it in Matlab.

                        If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

                        QFile file("BinTest.bin");
                        file.open(QIODevice::WriteOnly | QIODevice::Text);
                        QTextStream out(&file);
                        QVector<double> vec;
                        for(int ii = 0; ii < 10; ++ii){
                            vec << ii * 0.33;
                        }
                        for(auto&& val : vec)
                        out << val << ',';
                        file.close();
                        
                        CJhaC Offline
                        CJhaC Offline
                        CJha
                        wrote on last edited by
                        #16

                        @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                        JonBJ jsulmJ J.HilkJ 3 Replies Last reply
                        0
                        • CJhaC CJha

                          @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                          JonBJ Offline
                          JonBJ Offline
                          JonB
                          wrote on last edited by
                          #17

                          @CJha
                          OK, so STOP and tell us exactly what format MatLab will accept, and take it from there...

                          1 Reply Last reply
                          2
                          • CJhaC CJha

                            @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                            jsulmJ Offline
                            jsulmJ Offline
                            jsulm
                            Lifetime Qt Champion
                            wrote on last edited by
                            #18

                            @CJha But does Matlab support any binary format? You have to write in a format supported by Matlab.

                            https://forum.qt.io/topic/113070/qt-code-of-conduct

                            1 Reply Last reply
                            0
                            • CJhaC CJha

                              @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                              J.HilkJ Offline
                              J.HilkJ Offline
                              J.Hilk
                              Moderators
                              wrote on last edited by
                              #19

                              @CJha said in How to write entire QVector to a binary file?:

                              I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                              actually stop right here!

                              if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                              I'm sure there's an other - in memory - way to hand over those data points


                              Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                              Q: What's that?
                              A: It's blue light.
                              Q: What does it do?
                              A: It turns blue.

                              CJhaC 1 Reply Last reply
                              1
                              • CJhaC Offline
                                CJhaC Offline
                                CJha
                                wrote on last edited by
                                #20

                                @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                                jsulmJ 1 Reply Last reply
                                0
                                • CJhaC CJha

                                  @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                                  jsulmJ Offline
                                  jsulmJ Offline
                                  jsulm
                                  Lifetime Qt Champion
                                  wrote on last edited by
                                  #21

                                  @CJha said in How to write entire QVector to a binary file?:

                                  Matlab supports the binary format

                                  Do you have its specification?

                                  https://forum.qt.io/topic/113070/qt-code-of-conduct

                                  CJhaC 1 Reply Last reply
                                  0
                                  • J.HilkJ J.Hilk

                                    @CJha said in How to write entire QVector to a binary file?:

                                    I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                                    actually stop right here!

                                    if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                    I'm sure there's an other - in memory - way to hand over those data points

                                    CJhaC Offline
                                    CJhaC Offline
                                    CJha
                                    wrote on last edited by
                                    #22

                                    @J-Hilk I am not sure what you mean by

                                    if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                    Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                    JonBJ J.HilkJ 2 Replies Last reply
                                    0
                                    • CJhaC CJha

                                      @J-Hilk I am not sure what you mean by

                                      if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                      Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                      JonBJ Offline
                                      JonBJ Offline
                                      JonB
                                      wrote on last edited by JonB
                                      #23

                                      @CJha
                                      You may (well) know more than I, but can MatLab read and process 8MB of new data per second, at the same time as something else is producing it? And, separately, do you really generate 1 million new data points per second?

                                      Also, as @J-Hilk said, wouldn't sending a pipe stream (e.g. a socket?) be better than writing to file and reading back? Does Matlab accept incoming data elsewhere than in a file?

                                      CJhaC 1 Reply Last reply
                                      0
                                      • jsulmJ jsulm

                                        @CJha said in How to write entire QVector to a binary file?:

                                        Matlab supports the binary format

                                        Do you have its specification?

                                        CJhaC Offline
                                        CJhaC Offline
                                        CJha
                                        wrote on last edited by
                                        #24

                                        @jsulm It is highly compatible. Here are the links for fopen, fread, fseek. I all these I can specify the format, ByteOrder, size of data (such as int, double, etc), and quite a few other things.
                                        I don't think Matlab is the restrictive thing here, I can read any type of binary file in Matlab as long as I know how it is written.

                                        JonBJ 1 Reply Last reply
                                        0
                                        • CJhaC CJha

                                          @jsulm It is highly compatible. Here are the links for fopen, fread, fseek. I all these I can specify the format, ByteOrder, size of data (such as int, double, etc), and quite a few other things.
                                          I don't think Matlab is the restrictive thing here, I can read any type of binary file in Matlab as long as I know how it is written.

                                          JonBJ Offline
                                          JonBJ Offline
                                          JonB
                                          wrote on last edited by JonB
                                          #25

                                          @CJha
                                          The code to write a QVector to file in the way you want, as fast as possible in one blob not one-by-one, is given in e.g. https://www.qtcentre.org/threads/65713-Output-a-QVector-of-doubles-to-a-binary-file-(without-using-QDatastream)?p=289540#post289540 :

                                          qint64 bytesWritten = file.write(reinterpret_cast<const char*>(vec.constData()), sizeof(double) * vec.size());
                                          

                                          EDIT I think you will want reinterpret_cast<> rather than static_cast<> here as shown in that post, so I have altered the code line to use that.

                                          CJhaC 1 Reply Last reply
                                          4

                                          • Login

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Users
                                          • Groups
                                          • Search
                                          • Get Qt Extensions
                                          • Unsolved