Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. How to write entire QVector to a binary file?
QtWS25 Last Chance

How to write entire QVector to a binary file?

Scheduled Pinned Locked Moved Unsolved General and Desktop
qvectorbinary formatqdatastream
46 Posts 8 Posters 12.8k Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J J.Hilk
    17 Nov 2020, 09:39

    @CJha it works, you see "garbage" because there is a header to identify it as a QVector, how many entries the type and what not. You're serializing a complex class and not an array

    C Offline
    C Offline
    CJha
    wrote on 17 Nov 2020, 09:47 last edited by
    #3

    @J-Hilk I am unable to find any examples to tell me how to get the header. The QDataStream class description only says just a little about serializing QVectors do you know any examples that I can follow so that I have an idea how to read my binary file?

    J 1 Reply Last reply 17 Nov 2020, 09:54
    0
    • C Offline
      C Offline
      Christian Ehrlicher
      Lifetime Qt Champion
      wrote on 17 Nov 2020, 09:51 last edited by
      #4

      @CJha said in How to write entire QVector to a binary file?:

      so that I have an idea how to read my binary file?

      Read it with QDataStream. The format is internal.

      Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
      Visit the Qt Academy at https://academy.qt.io/catalog

      C 1 Reply Last reply 17 Nov 2020, 09:52
      1
      • C Christian Ehrlicher
        17 Nov 2020, 09:51

        @CJha said in How to write entire QVector to a binary file?:

        so that I have an idea how to read my binary file?

        Read it with QDataStream. The format is internal.

        C Offline
        C Offline
        CJha
        wrote on 17 Nov 2020, 09:52 last edited by
        #5

        @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

        J V 2 Replies Last reply 17 Nov 2020, 09:53
        0
        • C Offline
          C Offline
          Christian Ehrlicher
          Lifetime Qt Champion
          wrote on 17 Nov 2020, 09:53 last edited by
          #6

          @CJha said in How to write entire QVector to a binary file?:

          I have to read it in Matlab.

          Then create your own format or check what matlab needs to read it with it's built-in functions.

          Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
          Visit the Qt Academy at https://academy.qt.io/catalog

          1 Reply Last reply
          2
          • C CJha
            17 Nov 2020, 09:52

            @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

            J Offline
            J Offline
            JonB
            wrote on 17 Nov 2020, 09:53 last edited by JonB
            #7

            @CJha
            Then do not use a Qt serialization operator! Use whatever is going to be acceptable at the MatLab side, e.g. just a plain array's-worth of the data, or whatever. Qt serialization is for Qt programs!

            1 Reply Last reply
            0
            • C CJha
              17 Nov 2020, 09:47

              @J-Hilk I am unable to find any examples to tell me how to get the header. The QDataStream class description only says just a little about serializing QVectors do you know any examples that I can follow so that I have an idea how to read my binary file?

              J Offline
              J Offline
              J.Hilk
              Moderators
              wrote on 17 Nov 2020, 09:54 last edited by J.Hilk
              #8

              @CJha you can take a look into the source code

              no idea what version you're using, but I found this

              00717b7a-3822-470e-a26a-220eb197ba6e-image.png

              066e8007-1c4a-4394-aec7-7138631cfe9e-image.png

              link


              Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


              Q: What's that?
              A: It's blue light.
              Q: What does it do?
              A: It turns blue.

              C 1 Reply Last reply 17 Nov 2020, 09:59
              0
              • J J.Hilk
                17 Nov 2020, 09:54

                @CJha you can take a look into the source code

                no idea what version you're using, but I found this

                00717b7a-3822-470e-a26a-220eb197ba6e-image.png

                066e8007-1c4a-4394-aec7-7138631cfe9e-image.png

                link

                C Offline
                C Offline
                Christian Ehrlicher
                Lifetime Qt Champion
                wrote on 17 Nov 2020, 09:59 last edited by
                #9

                @J-Hilk But this is internal and may change (although the chance is very very low)

                Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                Visit the Qt Academy at https://academy.qt.io/catalog

                1 Reply Last reply
                1
                • C Offline
                  C Offline
                  CJha
                  wrote on 17 Nov 2020, 10:02 last edited by
                  #10

                  @J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
                  @Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                  J 1 Reply Last reply 17 Nov 2020, 10:07
                  0
                  • C Offline
                    C Offline
                    Christian Ehrlicher
                    Lifetime Qt Champion
                    wrote on 17 Nov 2020, 10:03 last edited by
                    #11

                    @CJha said in How to write entire QVector to a binary file?:

                    and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                    Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

                    Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                    Visit the Qt Academy at https://academy.qt.io/catalog

                    C 1 Reply Last reply 17 Nov 2020, 10:06
                    1
                    • C CJha
                      17 Nov 2020, 09:52

                      @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

                      V Offline
                      V Offline
                      VRonin
                      wrote on 17 Nov 2020, 10:04 last edited by
                      #12

                      @CJha said in How to write entire QVector to a binary file?:

                      I have to read it in Matlab.

                      If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

                      QFile file("BinTest.bin");
                      file.open(QIODevice::WriteOnly | QIODevice::Text);
                      QTextStream out(&file);
                      QVector<double> vec;
                      for(int ii = 0; ii < 10; ++ii){
                          vec << ii * 0.33;
                      }
                      for(auto&& val : vec)
                      out << val << ',';
                      file.close();
                      

                      "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
                      ~Napoleon Bonaparte

                      On a crusade to banish setIndexWidget() from the holy land of Qt

                      C 1 Reply Last reply 17 Nov 2020, 10:10
                      1
                      • C Christian Ehrlicher
                        17 Nov 2020, 10:03

                        @CJha said in How to write entire QVector to a binary file?:

                        and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                        Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

                        C Offline
                        C Offline
                        CJha
                        wrote on 17 Nov 2020, 10:06 last edited by
                        #13

                        @Christian-Ehrlicher I didn't know that before I saw the source code for QDataStream. I assumed since for a QVector data points are in adjacent memory positions pushing an entire vector into a binary file must be faster than iterating over it.

                        1 Reply Last reply
                        0
                        • C CJha
                          17 Nov 2020, 10:02

                          @J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
                          @Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                          J Offline
                          J Offline
                          JonB
                          wrote on 17 Nov 2020, 10:07 last edited by JonB
                          #14

                          @CJha said in How to write entire QVector to a binary file?:

                          The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points

                          Yep. That's what has to be done, and it's what the << serializer does, as @J-Hilk showed you. The only other way would be if you can get the address of contiguous QVector memory and save from there, which I'm guessing can be done. However...

                          ...If @VRonin's latest post is correct and you're supposed to produce text instead to export then you cannot help but do it one-by-one....

                          P.S.
                          BTW, you'd have to test, but my guess is that code to output data points one-by-one instead of in a contiguous clump is not what would be slow over 1,000,000 points --- rather, the size of the output written to file will be what is significant....

                          1 Reply Last reply
                          0
                          • C Offline
                            C Offline
                            Christian Ehrlicher
                            Lifetime Qt Champion
                            wrote on 17 Nov 2020, 10:08 last edited by
                            #15

                            You're correct with the adjacent memory but QVector can also hold objects so memcpy'ing it out will not work there. You can use memcpy if you want in your case but QDataStream is generic and has no optimizations for such things.

                            Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                            Visit the Qt Academy at https://academy.qt.io/catalog

                            1 Reply Last reply
                            3
                            • V VRonin
                              17 Nov 2020, 10:04

                              @CJha said in How to write entire QVector to a binary file?:

                              I have to read it in Matlab.

                              If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

                              QFile file("BinTest.bin");
                              file.open(QIODevice::WriteOnly | QIODevice::Text);
                              QTextStream out(&file);
                              QVector<double> vec;
                              for(int ii = 0; ii < 10; ++ii){
                                  vec << ii * 0.33;
                              }
                              for(auto&& val : vec)
                              out << val << ',';
                              file.close();
                              
                              C Offline
                              C Offline
                              CJha
                              wrote on 17 Nov 2020, 10:10 last edited by
                              #16

                              @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                              J J J 3 Replies Last reply 17 Nov 2020, 10:10
                              0
                              • C CJha
                                17 Nov 2020, 10:10

                                @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                                J Offline
                                J Offline
                                JonB
                                wrote on 17 Nov 2020, 10:10 last edited by
                                #17

                                @CJha
                                OK, so STOP and tell us exactly what format MatLab will accept, and take it from there...

                                1 Reply Last reply
                                2
                                • C CJha
                                  17 Nov 2020, 10:10

                                  @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                                  J Offline
                                  J Offline
                                  jsulm
                                  Lifetime Qt Champion
                                  wrote on 17 Nov 2020, 10:11 last edited by
                                  #18

                                  @CJha But does Matlab support any binary format? You have to write in a format supported by Matlab.

                                  https://forum.qt.io/topic/113070/qt-code-of-conduct

                                  1 Reply Last reply
                                  0
                                  • C CJha
                                    17 Nov 2020, 10:10

                                    @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                                    J Offline
                                    J Offline
                                    J.Hilk
                                    Moderators
                                    wrote on 17 Nov 2020, 10:13 last edited by
                                    #19

                                    @CJha said in How to write entire QVector to a binary file?:

                                    I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                                    actually stop right here!

                                    if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                    I'm sure there's an other - in memory - way to hand over those data points


                                    Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                                    Q: What's that?
                                    A: It's blue light.
                                    Q: What does it do?
                                    A: It turns blue.

                                    C 1 Reply Last reply 17 Nov 2020, 10:18
                                    1
                                    • C Offline
                                      C Offline
                                      CJha
                                      wrote on 17 Nov 2020, 10:14 last edited by
                                      #20

                                      @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                                      J 1 Reply Last reply 17 Nov 2020, 10:16
                                      0
                                      • C CJha
                                        17 Nov 2020, 10:14

                                        @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                                        J Offline
                                        J Offline
                                        jsulm
                                        Lifetime Qt Champion
                                        wrote on 17 Nov 2020, 10:16 last edited by
                                        #21

                                        @CJha said in How to write entire QVector to a binary file?:

                                        Matlab supports the binary format

                                        Do you have its specification?

                                        https://forum.qt.io/topic/113070/qt-code-of-conduct

                                        C 1 Reply Last reply 17 Nov 2020, 10:23
                                        0
                                        • J J.Hilk
                                          17 Nov 2020, 10:13

                                          @CJha said in How to write entire QVector to a binary file?:

                                          I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                                          actually stop right here!

                                          if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                          I'm sure there's an other - in memory - way to hand over those data points

                                          C Offline
                                          C Offline
                                          CJha
                                          wrote on 17 Nov 2020, 10:18 last edited by
                                          #22

                                          @J-Hilk I am not sure what you mean by

                                          if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                          Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                          J J 2 Replies Last reply 17 Nov 2020, 10:21
                                          0

                                          12/46

                                          17 Nov 2020, 10:04

                                          topic:navigator.unread, 34
                                          • Login

                                          • Login or register to search.
                                          12 out of 46
                                          • First post
                                            12/46
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Users
                                          • Groups
                                          • Search
                                          • Get Qt Extensions
                                          • Unsolved