Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. QTemporaryFile takes forever
Forum Updated to NodeBB v4.3 + New Features

QTemporaryFile takes forever

Scheduled Pinned Locked Moved Solved General and Desktop
qtemporaryfile
12 Posts 5 Posters 3.1k Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • jsulmJ Offline
    jsulmJ Offline
    jsulm
    Lifetime Qt Champion
    wrote on last edited by
    #2

    So, you want to store each object in its own file?
    And you have "very big number of objects", right?
    Then you will create a very big number of temporary files, I don't think such an approach will deliver a good performance.

    https://forum.qt.io/topic/113070/qt-code-of-conduct

    1 Reply Last reply
    1
    • VRoninV Offline
      VRoninV Offline
      VRonin
      wrote on last edited by
      #3

      That's not exactly the point, if I simulate the same behaviour using QFile the results is MUCH faster and creating the file does not depend on the number of files already in the folder

           QList<QFile*> allfiles;
           QTime timeCounter;
           QString tempFileName;
           QFile file("C:/Temp/ChangeAverageTime.txt");
           file.open(QIODevice::WriteOnly | QIODevice::Text);
           QTextStream stream(&file);
           timeCounter.start();
           for (int i = 0; i < 2000; ++i) {
               if (i % 100 == 0) {
                   stream << QString("%1 Files. Time: %2").arg(i).arg(static_cast<double>(timeCounter.restart()) / 1000.0, 0, 'f', 2) << '\n';
               }
               tempFileName.clear();
               do { // Generate random file name
                   for (int j = 0; j < 15; ++j) {
                       tempFileName.append(static_cast<char>(97 + (qrand() % 26)));
                   }
               } while (QFile::exists("C:/temp/TempTest/" + tempFileName));
               allfiles.append(new QFile("C:/temp/TempTest/" + tempFileName));
               allfiles.last()->open(QIODevice::WriteOnly);
           }
           file.close();
           while (!allfiles.isEmpty()) {
               auto tempDel = allfiles.takeLast();
               tempDel->remove();
               delete tempDel;
           }
           return 0;
      

      0 Files. Time: 0.00
      100 Files. Time: 0.03
      200 Files. Time: 0.03
      300 Files. Time: 0.03
      400 Files. Time: 0.03
      500 Files. Time: 0.03
      600 Files. Time: 0.03
      700 Files. Time: 0.03
      800 Files. Time: 0.03
      900 Files. Time: 0.03
      1000 Files. Time: 0.03
      1100 Files. Time: 0.03
      1200 Files. Time: 0.03
      1300 Files. Time: 0.03
      1400 Files. Time: 0.03
      1500 Files. Time: 0.03
      1600 Files. Time: 0.03
      1700 Files. Time: 0.03
      1800 Files. Time: 0.03
      1900 Files. Time: 0.03

      "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
      ~Napoleon Bonaparte

      On a crusade to banish setIndexWidget() from the holy land of Qt

      JonBJ 1 Reply Last reply
      0
      • jsulmJ Offline
        jsulmJ Offline
        jsulm
        Lifetime Qt Champion
        wrote on last edited by
        #4

        I think creating temporary files is slower compared to creating new files, because Qt framework has to ensure that the file names are unique.

        https://forum.qt.io/topic/113070/qt-code-of-conduct

        1 Reply Last reply
        1
        • VRoninV Offline
          VRoninV Offline
          VRonin
          wrote on last edited by
          #5

          I'm doing that too in: while (QFile::exists("C:/temp/TempTest/" + tempFileName)); but it's lightning fast

          "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
          ~Napoleon Bonaparte

          On a crusade to banish setIndexWidget() from the holy land of Qt

          1 Reply Last reply
          0
          • jsulmJ Offline
            jsulmJ Offline
            jsulm
            Lifetime Qt Champion
            wrote on last edited by
            #6

            Why shouldn't it be fast? You only check whether a file exists or not. What QTemporaryFile is doing: it creates a unique file name and ensures nobody else creates one with the same name in the same directory. This, for sure, involves some OS calls to make the creation of a temp file an atomic operation. Else several processes could create and use a temp file with same file name.

            https://forum.qt.io/topic/113070/qt-code-of-conduct

            1 Reply Last reply
            3
            • M Offline
              M Offline
              Mecanica
              wrote on last edited by
              #7

              I know this thread is too old, but I would resolve this by using an inmemory database like sqlite, where you can have two columns one for the file name an another for the file data stored as a QByteArray.

              JonBJ 1 Reply Last reply
              0
              • M Mecanica

                I know this thread is too old, but I would resolve this by using an inmemory database like sqlite, where you can have two columns one for the file name an another for the file data stored as a QByteArray.

                JonBJ Offline
                JonBJ Offline
                JonB
                wrote on last edited by
                #8

                @Mecanica
                Whatever the merits of your suggestion, that would not be a suitable replacement for a QTemporaryFile method in the Qt framework!

                1 Reply Last reply
                1
                • VRoninV VRonin

                  That's not exactly the point, if I simulate the same behaviour using QFile the results is MUCH faster and creating the file does not depend on the number of files already in the folder

                       QList<QFile*> allfiles;
                       QTime timeCounter;
                       QString tempFileName;
                       QFile file("C:/Temp/ChangeAverageTime.txt");
                       file.open(QIODevice::WriteOnly | QIODevice::Text);
                       QTextStream stream(&file);
                       timeCounter.start();
                       for (int i = 0; i < 2000; ++i) {
                           if (i % 100 == 0) {
                               stream << QString("%1 Files. Time: %2").arg(i).arg(static_cast<double>(timeCounter.restart()) / 1000.0, 0, 'f', 2) << '\n';
                           }
                           tempFileName.clear();
                           do { // Generate random file name
                               for (int j = 0; j < 15; ++j) {
                                   tempFileName.append(static_cast<char>(97 + (qrand() % 26)));
                               }
                           } while (QFile::exists("C:/temp/TempTest/" + tempFileName));
                           allfiles.append(new QFile("C:/temp/TempTest/" + tempFileName));
                           allfiles.last()->open(QIODevice::WriteOnly);
                       }
                       file.close();
                       while (!allfiles.isEmpty()) {
                           auto tempDel = allfiles.takeLast();
                           tempDel->remove();
                           delete tempDel;
                       }
                       return 0;
                  

                  0 Files. Time: 0.00
                  100 Files. Time: 0.03
                  200 Files. Time: 0.03
                  300 Files. Time: 0.03
                  400 Files. Time: 0.03
                  500 Files. Time: 0.03
                  600 Files. Time: 0.03
                  700 Files. Time: 0.03
                  800 Files. Time: 0.03
                  900 Files. Time: 0.03
                  1000 Files. Time: 0.03
                  1100 Files. Time: 0.03
                  1200 Files. Time: 0.03
                  1300 Files. Time: 0.03
                  1400 Files. Time: 0.03
                  1500 Files. Time: 0.03
                  1600 Files. Time: 0.03
                  1700 Files. Time: 0.03
                  1800 Files. Time: 0.03
                  1900 Files. Time: 0.03

                  JonBJ Offline
                  JonBJ Offline
                  JonB
                  wrote on last edited by
                  #9

                  @VRonin said in QTemporaryFile takes forever:

                       } while (QFile::exists("C:/temp/TempTest/" + tempFileName));
                       allfiles.append(new QFile("C:/temp/TempTest/" + tempFileName));
                       allfiles.last()->open(QIODevice::WriteOnly);
                  

                  @VRonin I see that you wrote this in 2015. Leaving aside that if the timing you describe still holds then I am not sure why and yours is an interesting finding, looking at this code now would you recommend it for exclusively creating a file, in a multi-process environment? ;-)

                  Christian EhrlicherC 1 Reply Last reply
                  0
                  • JonBJ JonB

                    @VRonin said in QTemporaryFile takes forever:

                         } while (QFile::exists("C:/temp/TempTest/" + tempFileName));
                         allfiles.append(new QFile("C:/temp/TempTest/" + tempFileName));
                         allfiles.last()->open(QIODevice::WriteOnly);
                    

                    @VRonin I see that you wrote this in 2015. Leaving aside that if the timing you describe still holds then I am not sure why and yours is an interesting finding, looking at this code now would you recommend it for exclusively creating a file, in a multi-process environment? ;-)

                    Christian EhrlicherC Online
                    Christian EhrlicherC Online
                    Christian Ehrlicher
                    Lifetime Qt Champion
                    wrote on last edited by
                    #10

                    @JonB The code above is not correct and can create a race condition since another thread/process can create exactly the same filename between QFile::exists() and open(). That's also the reason why the Qt implementation is slower - it opens a file and if it fails it checks some error conditions.
                    Also the possible space for the file names is lower in QTemporaryFile (only six digits by default) while the other loop is using 15 characters, so the chance of a collision is much lower.

                    Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                    Visit the Qt Academy at https://academy.qt.io/catalog

                    JonBJ 1 Reply Last reply
                    2
                    • Christian EhrlicherC Christian Ehrlicher

                      @JonB The code above is not correct and can create a race condition since another thread/process can create exactly the same filename between QFile::exists() and open(). That's also the reason why the Qt implementation is slower - it opens a file and if it fails it checks some error conditions.
                      Also the possible space for the file names is lower in QTemporaryFile (only six digits by default) while the other loop is using 15 characters, so the chance of a collision is much lower.

                      JonBJ Offline
                      JonBJ Offline
                      JonB
                      wrote on last edited by
                      #11

                      @Christian-Ehrlicher
                      It was the race condition that I had noticed. I was bringing that to @VRonin 's attention and seeing what he said about it, it's sometimes interesting to look back on your own code from years ago :)

                      1 Reply Last reply
                      0
                      • VRoninV Offline
                        VRoninV Offline
                        VRonin
                        wrote on last edited by
                        #12

                        The bug was solved in Qt 5.8 so a while ago. I'll mark the thread as solved as well

                        "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
                        ~Napoleon Bonaparte

                        On a crusade to banish setIndexWidget() from the holy land of Qt

                        1 Reply Last reply
                        0

                        • Login

                        • Login or register to search.
                        • First post
                          Last post
                        0
                        • Categories
                        • Recent
                        • Tags
                        • Popular
                        • Users
                        • Groups
                        • Search
                        • Get Qt Extensions
                        • Unsolved