[solved] Serialize heap objects pointed to by QList

[A Former User]

Hi there!
To explain my problem, let me present the following example:
I have a class A for which I have provided QDataStream &operator<<(QDataStream &out, const A* my_a);
I want to serialize a QList<A*> to write the entire contents of my_a into a file. When I write, everything seems fine:
QDataStream out(&file);
out<<list_of_As;
Qt somehow seems to follow the pointer, as I can see fragments of the QStrings in my_a appearing in the file.
However, when it comes to retrieve the file, I don't know how to handle the memory management: How does qt behave when unserializing a QList<A*>? Will it do the memory allocation for me and I can completely forget about it? Or will I have to call the constructor for class A by myself (if so, where and how)?
Thx in advance!
Cheers, Kalsan

[SGaist]

Hi,

You have to provide the counterpart operator (so operator>>) and handle the creation of the objects yourself

[A Former User]

Hi Samuel,
thanks for your reply! The question then now is how to implement operator>>. That appearently isn't obvious for pointers, see http://stackoverflow.com/a/23697747
Now, I have a list of pointers, so it's even worse... how do I do that properly? Should I copy all objects to the stack before serializing?
Cheers,
Kalsan

[SGaist]

What are you storing about A ? By the way, do you really need a pointer to your class ?

[A Former User]

A contains a QString and a QList<B*>. B contains an int and two distinct QList<C*>. C contains an int and a QString. Note that this model is simplified, the actual datastructure is more complex than that.
There are many helper functions in various files that prepare, update, verify alter, create and remove Elements in those lists. I decided to use dynamic memory management (on the heap and not the stack) so that I could pass those datastructures around more easily and efficiently, so yes, I think I need them to be pointers.

[SGaist]

Did you consider Qt's implicit sharing helper classes ?

[Chris Kawa]

As @SGaist said in the first response you just need to provide the operator>> for the pointer. The syntax is not obvious but it's quite logical when you see it. Here's an example:

struct Foo {
    Foo(const QString& foo = QString()) : data(foo) {}
    QString data;
};

QDataStream& operator<<(QDataStream& out, const Foo* foo) { out << foo->data; return out;  }
QDataStream& operator>>(QDataStream& in, Foo*& foo) { foo = new Foo(); in >> foo->data; return in;  }
//yes, that's a reference to a pointer ;)

and then...

QByteArray storage;

QList<Foo*> orig { new Foo("bar"), new Foo("bazz") };
QDataStream storeStream(&storage, QIODevice::WriteOnly);
storeStream << orig;
qDeleteAll(orig);

QList<Foo*> retr;
QDataStream retrStream(&storage, QIODevice::ReadOnly);
retrStream >> retr;
qDeleteAll(retr);

[A Former User]

Hi guys! Thanks for your replies,
@SGaist : No, I wasn't aware of that feature. Will check it out!
@Chris-Kawa : Sounds aweseommely simple, but doesn't this contradict Samuel's former statment: "You have to [...] handle the creation of the objects yourself" ?
I'm talking about the part:
QList<Foo*> retr;
QDataStream retrStream(&storage, QIODevice::ReadOnly);
retrStream >> retr; -> does Qt call my constructor for every object in the list here?
qDeleteAll(retr);
Cheers,
Kalsan

[Chris Kawa]

@kalsan said:

Sounds aweseommely simple, but doesn't this contradict Samuel's former statment: "You have to [...] handle the creation of the objects yourself" ?

No. I am handling it myself by calling new Foo() in the operator>>.

There's no magic here. When Qt deserializes a list it resizes it to the size value stored in the data and initializes each of its element by calling operator>> on it. In this case the element type happens to be a pointer, but it's exactly the same as if it were an int or a QString. We need to initialize the variable in any case and in case of a pointer we do it by assigning a new Foo() to it and filling its data from the stream.

[Chris Kawa]

To answer your question specifically - no, it does not call a constructor for your objects. It calls a (default) constructor for a pointer to your object.
In pseudocode something like this:

stream >> typeid;
if(typeid == a_type_id_of_a_list) {
    stream >> sizeOfTheList
    QList<Type> list;
    for(int i=0; i<sizeOfTheList; ++i ) {
        Type variable; //creates a default constructed value of Type
        stream >> variable;
        list << variable;
    }
}

The important thing to notice is that the Type in this case is Foo*, a pointer. If it were Foo then it would default construct Foo object, but here all it does is default construct a pointer, which simply creates a null pointer. Then an operator>> is called and the pointer variable is passed to it by reference and we initialize it.

[A Former User]

Hey Chris,
So I spend hours trying to understand why the hell the code I generated from what I learned from you and the Qt documentation wouldn't even compile. The answer is simple: Deserializing QString works fine, but if I try to do that with a qint32... BAM!
So the Qt documentation for QDataStream says, tralala (yes, I AM slowly getting nuts with this), just go QString str; qint32 a; in >> str >> a; So far for the expectation.
Now, let's talk about reality. Please change in your Foo example the type of data from QString to qint32.
This is what I am getting (and no, even throwing the computer out of the window and picking it up 4200 floors beyond didn't help):
-> Error: invalid conversion from 'qint32 {aka int}' to 'UnitStruct*' [-fpermissive] QDataStream& operator>>(QDataStream& in, UnitStruct*& foo) { foo = new UnitStruct(); in >> foo->data; return in; } ^
-> note: initializing argument 2 of 'QDataStream& operator>>(QDataStream&, UnitStruct*&)' QDataStream& operator>>(QDataStream& in, UnitStruct*& foo) { foo = new UnitStruct(); in >> foo->data; return in; } ^
-> Error: cannot bind rvalue '(UnitStruct*)((long int)foo->UnitStruct::data)' to 'UnitStruct*&' QDataStream& operator>>(QDataStream& in, UnitStruct*& foo) { foo = new UnitStruct(); in >> foo->data; return in; } ^
By experience, I know I can be veeeery slow to get things. So please pardon me if I, once again, don't see the huge bar right in front of my head, but I am completely hopeless about this right now.
Thanks for your patience...
Kalsan

[Chris Kawa]

Yup, works just as well:

struct UnitStruct {
    UnitStruct(qint32 foo = 0) : data(foo) {}
    qint32 data;
};

QDataStream& operator<<(QDataStream& out, const UnitStruct* foo) { out << foo->data; return out;  }
QDataStream& operator>>(QDataStream& in, UnitStruct*& foo) { foo = new UnitStruct(); in >> foo->data; return in;  }

QByteArray storage;

QList<UnitStruct*> orig { new UnitStruct(42), new UnitStruct(66) };
QDataStream storeStream(&storage, QIODevice::WriteOnly);
storeStream << orig;
qDeleteAll(orig);

QList<UnitStruct*> retr;
QDataStream retrStream(&storage, QIODevice::ReadOnly);
retrStream >> retr;
qDeleteAll(retr);

[A Former User]

Well I seriously wonder if I might have a corrupt Qt version or something... I copy-pasted the first 6 lines of your code above (up to an without QByteArray storate;) and I get the three compile errors I pasted above. I did clean and rebuild the entire project.
I'm using unpatched (direct from repo) Qt 5.4.1-9 with Qt Creator on Arch Linux, default gcc configuration.
Looks to me like the exact same code will compile on your machine but not on mine. Should I consider filing a bug report?
Cheers,
Kalsan

[Chris Kawa]

Should I consider filing a bug report?

You might, but I'm not sure to whom exactly. This doesn't look like a Qt bug. More like a compiler one. What version of gcc are you using? I don't have a linux to test this but it works fine on both VS2013 and MinGW 4.9.1(which is basically gcc).

Oh, and btw.
Error: invalid conversion from 'qint32 {aka int}' to 'UnitStruct*'
This looks like the wrong thing is passed to the operator>>. Are you sure you didn't by mistake try to read the data into QList<qint32> instead of QList<UnitStruct*>?

[A Former User]

I couldn't quite believe that this could be an actual bug, so I kept testing.
And the result... oh well... I'm truly ashamed... I have included QDataStream in every file... except the one I actually tested.
So the solution is: #include <QDataStream> Facepalm
Thanks for your tips, I'm now gonna implement what you told me.