Weird collateral effect when building a QT application using shapelib

bleriot13

Hi!

Since 2018 I have been developing a suite of applications using Qt5 on Windows. Many of these applications read the so-called "ESRI shapefile" format. This is a quite old format that is still very actively used in geo-applications.

In fact, a shapefile is a set of several files, one of them being stored in the venerable dBase format. The extension for dBase files is “.dbf”.

To read and write shapefiles (and the several kinds of files these are composed of) in C or C++ a seasoned library, the “shapelib”, has been used for about two decades now. Of course, my applications use the shapelib library too to do it. The shapelib may be obtained here.

This combination, C++, Qt5, shapelib and other open-source libraries has worked as a charm for 5 years now. Until now, that I decided to migrate these applications to Linux.

The problem I’m facing is that, in some cases that I will explain later, the digits after the decimal point for all the float data stored in a shapelib are lost. In short, if the file stores something like 123.4567, what I get when I read (or write) this value is just 123.0000.

Weird, isn’t it?

After carefully debugging my applications, I concluded that instantiating a QApplication (GUI apps) or QCoreApplication (command line, no GUI apps) was the reason why this problem happened.

The problem is that I can live without a QCoreApplication object in a command-line, non-GUI app, but it is impossible to avoid QApplication objects in GUI-based apps.

I have attached the source code required to build a very simple command line application by yourselves and see how this happens. (this is not a minimalistic application devised to show the problem, not the real ones that I'm trying to migrate to Linux).

The six source files making the shapelib may be downloaded from the link above (to avoid cluttering too much this post). The names of the files required are listed in the .pro file, but I reproduce these here for the sake of convenience:

dbfopen.c, shapefil.h, safileio.c, shpopen.c, sbnsearch.c and shptree.c

The quickest way to rebuild the project is to copy the whole set of files (the ones shown later in this post plus the six from the shapelib) in the same folder. The .pro file is prepared to work this way.

The problem is that I had prepared a very simple .dbf file (5 rows, 5 float values each row) to let you check the app but I’ve found no way to upload it. I can send it via email, Google Driver or whatever means you prefer, but tell me how.

Below I’ll copy the source code as well as some screenshots showing the results of executing several versions (Windows vs. Linux, using QCoreApplication or not...). If any of you has any idea about how to solve this problem, please let me know.

Oh! I forgot to say: I'm using QT 5.15.2 in both platforms (Windows, Linux), Visual Studio 2015 (Windows 10) and gcc (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0 in my Linux Mint 20.3 Cinnamon box.

Thank you very much for your help...

File: dbf_reader_cmd.pro

QT -= gui

TARGET = dbf_reader_cmd
CONFIG += c++11 console
CONFIG -= app_bundle
DEFINES += QT_DEPRECATED_WARNINGS

#
# Configure the destination directories here.
#
# Destinations are dependent on the configuration and platform
# being used.
#

ROOT_DIRECTORY = $$PWD

CONFIG(release, debug|release): CURRENT_CONFIG = Release
else: CONFIG(debug, debug|release): CURRENT_CONFIG = Debug

# We only compile 64-bit executables.

PLATFORM = x64

# Build and other directories.

win32: BUILD_DIRECTORY = $${ROOT_DIRECTORY}/$${PLATFORM}/$${CURRENT_CONFIG}
unix:!macx: BUILD_DIRECTORY = $${ROOT_DIRECTORY}/$${CURRENT_CONFIG}

OBJECTS_DIR = $${BUILD_DIRECTORY}
MOC_DIR     = $${BUILD_DIRECTORY}
RCC_DIR     = $${BUILD_DIRECTORY}
UI_DIR      = $${BUILD_DIRECTORY}
DESTDIR     = $${BUILD_DIRECTORY}

HEADERS += \
    dbf_utils.hpp \
    shapefil.h

SOURCES += \
    dbf_utils.cpp \
    dbfopen.c \
    main_cmd.cpp \
    safileio.c \
    sbnsearch.c \
    shpopen.c \
    shptree.c

File: dbf_utils.cpp

/** \file dbf_utils.cpp
\brief Implementation file for dbf_utils.hpp
*/

#include "dbf_utils.hpp"

DBF_utils::DBF_utils(void)
{
  {
    // Intentionally left blank.
  }
}

bool
DBF_utils::
read_DBF_header
(DBFHandle&	hDBF,
 string&    field_names,
 int&       n_rows,
 int&       n_columns)
{
  {
    char*        field_name;
    DBFFieldType field_type;
    string       s_field_name;

    //
    // Get the number of fields in the DBF file
    // (that is, the number of columns in the file).
    //

    n_columns = DBFGetFieldCount(hDBF);

    // Number of rows.

    n_rows = DBFGetRecordCount(hDBF);

    //
    // Iterate for all the fields in the header to
    // get their names. These are stored in our
    // output "header" parameter.
    //

    field_names = "";

    for (int i = 0; i < n_columns; i++)
    {
      field_name = new char[12];

      field_type = DBFGetFieldInfo(hDBF, i, field_name, nullptr, nullptr);
      if (field_type == FTInvalid) return false;

      s_field_name = field_name;

      field_names += s_field_name;
      if (i != (n_columns-1)) field_names += " ";
    }

    // That's all.

    return true;
  }
}

void
DBF_utils::
read_DBF_row
(      DBFHandle&      hDBF,
 const int             row_number,
 const int             columns_to_read,
       vector<double>& values)
{
  {
    double value;

    // Reset the output vector.

    values.clear();

    //
    // Read as many columns as requested from the row selected.
    // Store the successively read values into the output vector.
    //

    for (int field_column = 0; field_column < columns_to_read; field_column++)
    {
      value =  DBFReadDoubleAttribute(hDBF, row_number, field_column);
      values.push_back(value);
    }
  }

  return;
}

File: dbf_utils.hp

/** \file dbf_utils.hpp
\brief Simple utilities for reading still simpler dbf files.
*/

#ifndef DBF_UTILS_HPP
#define DBF_UTILS_HPP

#include "shapefil.h"

#include <vector>
#include <string>

using namespace std;


class DBF_utils
{
  public:

    /// \brief Default constructor.

    DBF_utils (void);

    /// \brief Read the DBF header to retrieve the list of
    ///        field names, the number of fields (columns)
    ///        and rows (data records).
    /**
       \param hDBF Handle to an already opened DBF file.
       \param field_names A string containing the list of
              field names (separated by blanks).
       \param Number of data rows in the DBF file.
       \param Number of data columns (fields) in the DBF file.
       \return True if the header may be read, false otherwise.

      Note that the DBF file must have been opened before calling
      this method.
     */

    bool     read_DBF_header (DBFHandle&	hDBF,
                              string&     field_names,
                              int&        n_rows,
                              int&        n_columns);

    /// \brief Read a single row from the DBF file.
    /**
      \param hDBF A handle to the already opened DBF file.
      \param row_number Number of the row that must be read.
      \param columns_to_read Number of columns (starting from the
             first one) that must be read from the given row.
      \param values The set of values read from the DBF file

      Note that the DBF file must have been opened before calling
      this method.

     */

    void     read_DBF_row    (      DBFHandle&       hDBF,
                              const int              row_number,
                              const int              columns_to_read,
                                    vector<double>&  values);
};

#endif // DBF_UTILS_HPP

File: main_cmd.cpp - Look for the "QCoreApplication a(argc, argv)" sentence and comment it (to get an app that works properly) or uncomment it (to make all decimal values become 0).

/** \file main.cpp
\brief Main entry point for the dump_dbf_cmd application.
*/

//
// The following inclusion is not really needed. It is included
// here to let you uncomment the "QCoreApplication a(argc, argv)"
// at the beginning of the main function.
//

#include <QCoreApplication>

#include "dbf_utils.hpp"
#include "shapefil.h"

#include <iostream>
#include <vector>

using namespace std;

/// \brief ADAfinder application's main function.
/**
  \param argc Number of command line parameters.
  \param argv The command line parameters.
  \return 0 if the process finished successfully, any other
         value otherwise.

  This is the dbf_reader_cmd application's main entry point.
  At least one parameter must be provided: the name of the
  dbf file to dump.
 */

int
main
(int   argc,
 char* argv[])
{
  {
    //
    // If the following sentence is uncommented, the program stops
    // working properly. All the values after the decimal point
    // simply vanish. For instance, a value that should be
    // 1.2 becomes 1.0.
    //

    //QCoreApplication a(argc, argv);

    string    dbfile;
    DBF_utils dbu;
    DBFHandle hDBF;
    string    header;
    int       n_cols;
    int       n_rows;
    string    the_list_of_fields;

    //
    // Check that we've got at least one parameter: the name
    // of the dbf file to dump.
    //

    if (argc < 2)
    {
      cout << "dump_dbf_cmd <dbf_file_name>" << endl;
      return 1;
    }

    // Name of the dbf file.

    dbfile = argv[1];

    // Open it.

    hDBF = DBFOpen(dbfile.c_str(), "rb");
    if (hDBF == nullptr)
    {
      // Unable to open the DBF file.

      cout << "Unable to open the input dbf file" << endl;
      return 1;
    }

    //
    // Read the header, thus retrieving the list of field
    // names as well as the number of rows (data records)
    // and columns (data fields) in the file.
    //

    if (!dbu.read_DBF_header (hDBF, the_list_of_fields, n_rows, n_cols))
    {
      cout << "Error reading the header" << endl;
      return 1;
    }

    // Print the header.

    cout << the_list_of_fields << endl;

    //
    // Read as many rows (records) and columns (fields per record)
    // as stated by the header. Print them.
    //

    for (int the_row = 0; the_row < n_rows; the_row++)
    {
      vector<double> values;
      dbu.read_DBF_row(hDBF, the_row, n_cols, values);

      for (int the_col = 0; the_col < n_cols; the_col++)
        cout << values[the_col] << " ";

      cout << endl;
    }

    // Close the dbf file.

    DBFClose(hDBF);

  }
}

The next picture shows the sample dbf file used for all the tests (opened with LibreOffice Calc). Note how all values have at least one decimal:

The next one shows a Linux GUI-based application (using the same code to read the dbf file as the command line one whose source code you've seen above). Note how all numbers end in ".0000".

But if this very same code is run on Windows, the numbers are read properly:

The same problem shows up when the app whose code has been posted above is run on Linux if the QCoreApplication sentence is present in the code (no decimal part!!!)

But If the QCoreApplication sentence is removed, then everything works as expected:

When running these command line on Windows, it does not matter whether the QCoreApplication sentence is present or not, the result is always correct:

SGaist

Hi,

Can you provide a sample dbf file that allows to reproduce this behaviour ?

bleriot13

Hi, thanks for your interest!

I solved the problem avoiding the use of threads and embedding the command-line versions of the software in the GUI ones, using processes instead.

I destroyed the whole set of files (code, data) used for the example I'd posted, but, in fact, the problem was a generalised one. Reading any .dbf should produce the weird results, since it happened to me no matter what file I tried to load. To guarantee that the code I uploaded would work, the .dbf to read, however, should have just a series of an arbitrary number of DOUBLE fields to work (adding, for instance, boolean, integer or char attributes would make my example fail, because the example code expects finding only double values).

I would have liked to be able to upload a sample, but, as I said above, I solved the problem using a different approach and now I'm assigned to a different project... so, I have to apologize, I have not the time to prepare a synthetic dataset again. Please, accept my most sincere apologies. The pressure to finish my current tasks is too strong.

Thanks again for your interest.

SGaist

No worries !

You manage to find a solution so that's the essential point.