QNetworkAccessManager - first GET very slow

[SGaist]

Do you have any proxy or firewall in your network setup ?

[Aerius]

No, none that I'm aware of. Regular home network without special routing and/or protection. I might get around checking it in a different network, but regardless of the result, locking up the UI for a second for a GET request is not exactely great. I'm fine with waiting longer if it really is a pure networking issue, but the whole UI lockup is just not something I'm willing to accept if I can avoid it.

[SGaist]

GUI freeze ? On all the platforms you mentioned ?

[Aerius]

Yes, as I mentioned in the first post, during the slow GET, the gui is completely frozen up, waiting for it to finish. On all platforms. During that time on android, the application throws those concurrent GC 'warnings'.

[SGaist]

Can you share a sample request ? i.e. the URL that you are trying to get ?

[kshegunov]

@Aerius
What about DNS lookups? Could you try retrieving a file directly with a server's IP, so not to cause the whole DNS fetching nonsense to happen?

locking up the UI for a second for a GET request is not exactely great.

Unless QML uses some threading internally, which I don't believe to be the case, your whole network request is executed in the main thread. Fetching any data through the main thread could potentially cause the UI to freeze, so I don't see a reason to be surprised.

One additional remark:

I call all that from a QML File, that causes a singleton c++ object to call DownloadManager::downloadFile. I doubt this matters, as the c++ code is very much working on it's own.

You are not supposed to create QObject instances before the root QObject is alive (application object, QML engine or however the QML runtime is being set up), so depending on your (singleton) implementation you may be getting weird results from that as well.

[Aerius]

@SGaist said:

Can you share a sample request ? i.e. the URL that you are trying to get ?

https://forum.qt.io/logo.png (qt logo)
http://spiegel.de/static/sys/v10/logo/spiegel_online_logo_460_64.png (logo of a large german news website)
http://mediadb.kicker.de/library/image/logo-kicker.png (logo of a german football website)

I tried several others from several other sources and servers to no difference. And again, for instance, if I take these three files, whatever file is the first one to get downloaded triggers the lockup, subsequent ones do not. It doesn't matter if I download the qt logo first, for instance, or the kicker one. Whatever's the first in line locks up everything.

@kshegunov said:

@Aerius
What about DNS lookups? Could you try retrieving a file directly with a server's IP, so not to cause the whole DNS fetching nonsense to happen?

I'm not sure about how I would go about that, it's not like I can just replace the URL with an IP string. Willing to do so, but I'd need a pointer on where to look that up.

Unless QML uses some threading internally, which I don't believe to be the case, your whole network request is executed in the main thread. Fetching any data through the main thread could potentially cause the UI to freeze, so I don't see a reason to be surprised.

The thing is, QNAM explicitely states that it handles it's own requests asynchronously on other threads - and from what I can see, this is very much the case for every request but the first one . or alternatively, the first one triggers something that causes the lockup. Am I mistaken in my assumption here?

One additional remark:

I call all that from a QML File, that causes a singleton c++ object to call DownloadManager::downloadFile. I doubt this matters, as the c++ code is very much working on it's own.

You are not supposed to create QObject instances before the root QObject is alive (application object, QML engine or however the QML runtime is being set up), so depending on your (singleton) implementation you may be getting weird results from that as well.

The qml engine creates the c++ singleton object, it's registered via qmlRegisterSingletonType<>(). I do not programatically create any c++ objects before the one singleton object (which holds my entire c++ stuff).

On an important sidenote, even if the QNetworkReply::error() signal is emitted, the lockup/GC calls still happen before it's emitted. I'm pretty sure that the lockup doesn't have anything to do with the download itself, but happens as soon as QNetworkAccessManager starts any kind of get request for the very first time, regardless of what it tries to download.

Also: Thanks for your answers and thoughts so far. Communication over different timezones is a little difficult, but I'm happy about any help I can get here.

[kshegunov]

@Aerius
Hello,

I'm not sure about how I would go about that, it's not like I can just replace the URL with an IP string. Willing to do so, but I'd need a pointer on where to look that up.

Only replace the server name with the IP. If there is no special scheme to serving the requests this should be sufficient.

For example, the google logo:
http://www.google.bg/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png
On my machine google.bg resolves to 216.58.212.35, so using that IP I can access the image as well:
http://216.58.212.35/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png

The thing is, QNAM explicitely states that it handles it's own requests asynchronously on other threads - and from what I can see, this is very much the case for every request but the first one.

I admit, I don't know the documentation by heart, but where does it state that?

The qml engine creates the c++ singleton object, it's registered via qmlRegisterSingletonType<>(). I do not programatically create any c++ objects before the one singleton object (which holds my entire c++ stuff).

Fair enough. Then it's tied to and is handled by the QML engine so my guess is you're just fine with it.

[Aerius]

@kshegunov said:

@Aerius
Hello,

I'm not sure about how I would go about that, it's not like I can just replace the URL with an IP string. Willing to do so, but I'd need a pointer on where to look that up.

Only replace the server name with the IP. If there is no special scheme to serving the requests this should be sufficient.

For example, the google logo:
http://www.google.bg/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png
On my machine google.bg resolves to 216.58.212.35, so using that IP I can access the image as well:
http://216.58.212.35/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png

Interesting. For google this works for me too, for other servers it did not. Might be a server configuration thing? Not my area of expertise, if you couldn't guess that by now.
In any case - resolving the url to an ip adress in the get request doesn't change anything with my problem - still the same issue, still only for the first get.

The thing is, QNAM explicitely states that it handles it's own requests asynchronously on other threads - and from what I can see, this is very much the case for every request but the first one.

I admit, I don't know the documentation by heart, but where does it state that?

http://doc.qt.io/qt-5/qnetworkaccessmanager.html

QNetworkAccessManager has an asynchronous API. When the replyFinished slot above is called, the parameter it takes is the QNetworkReply object containing the downloaded data as well as meta-data (headers, etc.).

Moreover - later get requests work fine asynchronously. It's always only the first one that locks up the ui (which leads me to believe it has nothing to do with the actual download, as I stated before).

[kshegunov]

QNetworkAccessManager has an asynchronous API.

This doesn't necessarily mean that the code is threaded. In any case I got curious and opened the source to see what's going on. So yes, I can confirm that a thread is started, but for asynchronous request a network access manager global thread is used, meaning all replies are processed in one thread. This is one possibility as to why your first request is slower than the others.

In the request creation/reply setup I don't see anything special for android except an "assets" protocol. One thing that looks that might be locking the main thread is the backend and session initialization as they seem to be performed in the main thread, if I'm reading the source correctly.

Another thing I saw is that a proxy lookup is done in the main thread and while you're not using a proxy it might also take some time to refresh/resolve or possibly the caching is starting up a slower than expected.

So you see there are multiple possible reasons and I suggest investigating each one of them separately by fiddling with the access manager's properties a bit. For example you could try disabling/forcing caching. Or trying to trace where in the Qt source the delay could be occurring (you'd want to build Qt from source for that).
Although it's not really a direct answer to your problem, I hope this helps.

Kind regards.

[djee]

This is a bit old, but one of the first results on Google and neither the actual reason nor the workaround seem to be explained, so here they are: on first get(), the network manager will lazily load some library (DLL), which can take quite some time and freeze the UI if the code expects (rightly) that get() is purely asynchronous and will return quickly. The workaround is to pre-connect to the given server with connectToHost() or connectToHostEncrypted() earlier. After doing so the first get() has the same performance than subsequent ones.

[harry159821]

@djee In My Way connectToHost To Http Server Not Working, So I Call QSslConfiguration::defaultConfiguration(); Instead.