Sharing data between threads and QML

[Kniebou]

Hi,

I'd like to have your advices on how to implement a complex architecture. Here is what it looks like :

I have some custom form of data (let's call it MyData), which can come from several sources (a web server, a USB device, and a local database in my case).
My desktop application's role is to show these data, and to enable the user to edit it, delete it, and to create new data. To do that, I use QML for the interface, and C++ for the logic.
As the communication with the external data sources is slow (web and USB communication for example), it has to be put in different threads so that it doesn't impact the GUI. So, from a C++ point of view, I have the main GUI thread, and one thread per data source.
The QML part communicates only with the main thread, as the doc says it must be. I use QAbstractItemModels which contain QLists of MyData to display the data in QML. So the MyData class has to be QObject-inherited and its type exposed to QML (with qmlRegisterType) to make that possible.

To make it a little more clear, here is a typical use-case :
A USB device is plugged in. The USB thread retrieves the data from the device (as binary data), and parses it to organize the information in the form of an instance of the class MyData. Then, the data must be transferred to the main thread, which stores it on a model (a QList in a QAbstractItemModels for example). The user now sees the data (thanks to QML bindings), and edits some parts of it. So we have a new data, that must be transferred to the USB thread, which will send it to the device (as binary data).

Ok ! So my problem is : what is the best way to share the data between the threads and QML ?

The easier way would be to have 2 classes : a MyData struct which contains the data and is used for communication between threads via signals/slots, and a MyDataObject class inheriting QObject to expose the data to QML. The problem of this method is that the data is copied when passed from one thread to another. It is no big deal with light data, but if MyData is very heavy, it could really be a problem (or does it ?).

So I'd like to use pointers. But as MyData must inherit QObject (for QML communication), I have the thread affinity problem. My idea is to create the object in the USB thread, to change its ownership to the main thread (with moveToThread), and then send the pointer with signals/slots. But I don't know if it is a good idea, and I can't do that when I want to send MyData from the main thread to the USB thread, as the main thread must keep the data to display it. I thought about QSharedPointers that would be a good solution, but it seems that it is not well handled when dealing with QML (according to this wiki page and the associated discussion : https://wiki.qt.io/SharedPointersAndQmlOwnership).

I hope that everything is clear, and that someone can give me some advice.

Thanks !

Kniebou

Edit : I use Qt 5.4 and I try to always use the latest version

[JKSH]

Hi @Kniebou,

But as MyData must inherit QObject (for QML communication), I have the thread affinity problem. My idea is to create the object in the USB thread, to change its ownership to the main thread (with moveToThread), and then send the pointer with signals/slots. But I don't know if it is a good idea, and I can't do that when I want to send MyData from the main thread to the USB thread, as the main thread must keep the data to display it.

You've already identified yourself why this approach is a bad idea. I would recommend the other approach.

I have some custom form of data (let's call it MyData), which can come from several sources (a web server, a USB device, and a local database in my case).
My desktop application's role is to show these data, and to enable the user to edit it, delete it, and to create new data. To do that, I use QML for the interface, and C++ for the logic.

It's hard to tell what's best without knowing what your data looks like. Could you provide a sample?

It is no big deal with light data, but if MyData is very heavy, it could really be a problem (or does it ?).

The only way to know for sure is to write the code and profile it.

I think this shouldn't be an issue if your data is implicitly shared (e.g. QJsonObject, or a QVariantMap of implicitly shared items)

[Kniebou]

Hi @JKSH, and thank you for your answer.

It's hard to tell what's best without knowing what your data looks like. Could you provide a sample?

I deal with musical data.
For example, one of the entities is a "sound", which corresponds to an instrument. It contains the instrument's description (its name, category, size, etc.), the samples (.wav files), and the note mapping (which sample it uses for a given note : for example, use the sample 1 for notes between C1 and E1, sample 2 between F1 and A2, etc.).
To keep it simple, it is a big set of strings, numeric values, and binary data (for the .wav files). From an object point of view, the class "Sound" contains QStrings, QByteArrays, ints and floats, and containers of the above types (QLists, QSets...). I also have other classes to keep an organized object tree, for example the class "Sound" contains a QList of "Sample". "Sample" contains a QByteArray with the .wav, two ints to represent the note range (which could also be a "NoteRange" class), and QStrings for the description.
The total can represent several Mo of data (a really big sound is around 15 Mo), and I deal with a lot of sounds...

The only way to know for sure is to write the code and profile it.
I think this shouldn't be an issue if your data is implicitly shared (e.g. QJsonObject, or a QVariantMap of implicitly shared items)

I already have looked into the implicitly shared data concept. But I can't make my MyData class implicitly shared, because I have to handle it as a QObject-inherited class pointer (for QML), and I didn't find a way to make both coexist : implicitly shared classes must have a copy constructor, and QObjects are not copyable. Did have another way to use implicitly shared classes in mind ?

[JKSH]

Hi @Kniebou,

I can't make my MyData class implicitly shared, because I have to handle it as a QObject-inherited class pointer (for QML)

Philosophically, your data structure should not be a QObject. Rather, think of your QObject as a "device" that transfers data between QML and C++. Your QObject contains a copy of your data structure (as a member variable), and feeds parts of that structure to QML. You don't need to expose all of your data to QML -- only the parts that need to be displayed on the GUI. For example, I doubt that QML needs access to your audio samples.

Anyway, you don't need to make your custom structure itself implicitly shared either. If it consists mainly of QStrings and QByteArrays, then it is quite cheap to copy it (because its internals are mostly implicitly shared)

[Kniebou]

So, what you're saying is that I should have something like this :

struct SampleData {
    QString name;
    QByteArray data;
    int firstNote;
    int lastNote;
    // ...
}

struct SoundData {
    QList<SampleData> samples;
    QString name;
    int id;
    // ...
}

class SoundObject : public QObject {
    Q_OBJECT

    Q_PROPERTY(QString name READ name WRITE setName NOTIFY nameChanged)
    Q_PROPERTY(int id READ id WRITE setId NOTIFY idChanged)
    // ...

public:
    SoundObject(const SoundData& value, QObject* parent = 0) :
        QObject(parent),
        data(value)
    {}

    QString name() const { return data.name; }
    int id() const { return data.id; }
    // ...

public slots:
    void setName(const QString& value) {
        if (value != data.name) {
            data.name = value;
            emit nameChanged(value);
        }
    }
    void setId(int value) { /* ... */ }
    // ...

signals:
    void nameChanged(const QString& value);
    void idChanged(int value);
    // ...

private:
    SoundData data;
}

Then I would use SoundData to transfer data between threads, and store it in a SoundObject in the main thread to expose it to QML.

But then, how do I expose the QList<SampleData> ? Do I have to also make a SampleObject class ?
If so, where do I put the QList<SampleObject> ?
And what if my SampleData class has a QList of other custom data-type too ?

[JKSH]

Let's take a step back first: How do you want to display your data in the GUI? (Your answer will determine how you expose your data)

[Kniebou]

First, I have a view of all sounds as a list (with ListView for example). In this list, we only see minimal information about the sounds, e.g. the name, the size, maybe other minor things. The user can modify the name by double-clicking on a row.

Then, the user can select a sound in that list to see its details. From a GUI point of view, it will show it in an area on the right of the window (in a similar way as Finder on Mac OS) with the informations about the sound. From this area, the user can modify more things (the category for example), and see the list of samples.

Finally, I will have a button "see more details" that will open a separate window, in which the user can edit, add and remove samples, and see and edit even more sound settings.

When the user modifies something, I want the data to be modified accordingly in the source (for example the USB device, or the local database). So I need the corresponding threads to be connected to signals emitted by the data model when it changes. For slow operations (for example change a sample), I will have an "Apply" button to trigger the sending to source, but for everything else, it has to be transparent for the user.

[JKSH]

(The following is simply one suggestion; there are probably other ways you could do this)

Then I would use SoundData to transfer data between threads, and store it in a SoundObject in the main thread to expose it to QML.

Yep, that works just fine for the first ListView.

But then, how do I expose the QList<SampleData> ? Do I have to also make a SampleObject class ?

Yes, you would make a SampleObject class (derived from QObject) to expose data to the other ListView.

If so, where do I put the QList<SampleObject> ?

One way is to put QList<SampleObject*> in the same place where you put QList<SoundObject*>. Then, you expose both lists to QML.

This is the key idea: In your data structure, a SoundData object contains a QList<SampleData>. However, a SoundObject does not have to contain a QList<SampleObject*>. Your GUI structure is different from your data structure.

When the user clicks on a sound, you can delete the items in the existing QList<SoundObject*> and build a new list. Or, you can preserve the existing SampleObjects and only replace their internal SampleData (grow or shrink the list if necessary). Strictly speaking, you only need one QList<SampleObject*> in existence at any given time.

When the user edits sample info, the SampleObject automatically signals for the modifications to be copied back into the SoundData, and the SoundData is copied across to the other thread.

And what if my SampleData class has a QList of other custom data-type too ?

Then you follow the same design process again: Decide how you want that data list to appear in your GUI, and then expose it appropriately.

First, I have a view of all sounds as a list (with ListView for example). In this list, we only see minimal information about the sounds, e.g. the name, the size, maybe other minor things.

Ok, this is where you use a QList<SoundObject*> to provide data to your first ListView.

From a GUI point of view, it will show it in an area on the right of the window (in a similar way as Finder on Mac OS) with the informations about the sound. From this area, the user can modify more things (the category for example), and see the list of samples.

There are two different elements here:

1. A collection of more fields from one SoundData
2. Basic info from a list of SampleData objects

For #1, you can simply link the GUI fields to the properties of the SoundObject that was selected.
For #2, this is where your QList<SampleObject*> comes in.

Finally, I will have a button "see more details" that will open a separate window, in which the user can edit, add and remove samples, and see and edit even more sound settings.

Again, you can link to the same SoundObject to access "even more sound settings".

I'm not entirely sure what you meant by "edit, add and remove samples, but it sounds like this window would link to the same QList<SampleObject*> too.

What do you think?

[Kniebou]

So the main idea would be that the QObject-derivated classes reflect the GUI structure, and the xxxData classes reflect the data structure.
I like the idea :)

I'm not entirely sure what you meant by "edit, add and remove samples, but it sounds like this window would link to the same QList<SampleObject*> too.

Let's consider a use-case :

• I plug a USB device. The USB thread gets all the sounds that are on the device, convert them from binary to SoundData objects, and sends it via a signal to the main thread. The main thread receives them in a slot, and populates a QAbstractItemModel containing a QList<SoundObject*> with the received QList<SoundData>.
• I have my list of sounds displayed, which is a ListView with a model bound to the previous QAbstractItemModel. In a SoundObject, I have the attribute SoundData data, and in a SoundData, I have the attribute QList<SampleData> samples. The delegate of the ListView is bound to the properties of its associated SoundObject*. So if I want to display the amount of samples, I will have a Q_PROPERTY named "nbSamples" in my SoundObject which returns this->data->samples->size().
• I click on a row. The right area will show up, and the data it shows will be bound to the selected SoundObject*. It also shows a list of samples, which model is bound to another QAbstractItemModelcontaining a QList<SampleObject*>. So at the moment of the selection, I will have to populate this model with the QList<SampleData> of the selected SoundData.
• I click on "More details". A new window shows up, same thing as above.

Things get complicated when I want to edit things.

If I change the name of a sound, I have to forward the action to the USB thread, so that the sound is modified into the USB device.
My idea would be to call the WRITE function of the SoundObject's corresponding Q_PROPERTY, which will emit the NOTIFY signal. This signal will be connected to the USB thread at the SoundObject's creation. When adding or removing a sound, we call a C++ method from QML which will emit another signal connected to the USB thread. When the USB thread is done sending the modifications to the device, it will emit a signal to the main thread to tell if everything worked. If we were editing a sound's name, this signal will contain the SoundData of this sound as it is in the device (if the edition failed, it assures that the displayed informations always corresponds to the effective state in the device).

If I change a sample (for example its note range), I will call the WRITE function of the SampleObject's corresponding Q_PROPERTY, which will emit the NOTIFY signal. This signal will be connected to the corresponding SoundObject at the SampleObject's creation. Then, the SoundObject will emit a signal to the USB thread as it does when you edit one of its other "normal" properties.

Does it seem good ? Or do you see an easier way to communicate the changes to the USB thread ?

[JKSH]

@Kniebou said:

So the main idea would be that the QObject-derivated classes reflect the GUI structure, and the xxxData classes reflect the data structure.
I like the idea :)

Me too! :)

The main thread receives them in a slot, and populates a QAbstractItemModel containing a QList<SoundObject*> with the received QList<SoundData>.

You don't actually need to subclass QAbstractItemModel with your current approach. Your QList<SoundObject*> already acts as the model.

Subclassing QAbstractItemModel is a different approach that doesn't involve a QList<QObject*>. You can see examples of both approaches at http://doc.qt.io/qt-5/qtquick-modelviewsdata-cppmodels.html

Does it seem good ? Or do you see an easier way to communicate the changes to the USB thread ?

Yep, sounds good.

It does take a bit of code to set up, as you say. However, bear in mind that your data structure is quite complex too, and multithreading usually involves some extra effort to make sure everything works safely. And at the end of the day, you will have a clean yet extensible architecture -- the first ListView only needs to worry about SoundObject, the second ListView only needs to worry about SampleObject, and your USB thread only needs to worry about SoundData and SampleData.

I'm sure other people can come up with alternative architectures, but this is how I'd naturally do it. When you are implementing this code, you might come up with better ideas. If you do, please share!

[Kniebou]

You don't actually need to subclass QAbstractItemModel with your current approach. Your QList<SoundObject*> already acts as the model.

In fact, I want to use a QSortFilterProxyModel in C++ to filter what I want to display, and a TableView in QML to enable multiple selection, which is not available in ListView :(
So I had to subclass QAbstractItemModel, and add a custom role that I named "data", which returns the QObject*.

Thank you for everything, I'm going to implement this right away. If I find something useful, I will share it here. And if someone has another idea/vision of how to solve the problem, don't hesitate to share too !