cc1plus.exe: out of memory | 60MB encrypted resource file

[Slv007]

I'm trying to embed 60MB of encrypted data as resource file of app. But building it for windows breaks compilation with error "cc1plus.exe: out of memory allocating 1073745919 bytes" (yes, I have a lot of free memory 8 Gb in total, building machine is Win10 x64)

This file is required to be integrated into executable binary because of distribution as *.apk file for Android and same for iOS.
Compiling for Android also fails with "qrc_qml.cpp:2796221:3: internal compiler error: Segmentation fault
0x9,0xec,0xa5,0x71,0xc7,0x27,0x18,0xd9,0xd,0xaa,0xd7,0x75,0xfc,0xe8,0xb1,0x2b,"

I noticed that rcc extends my encrypted 60MB file into 260MB xxx.cpp one and I think this the reason. But how I can embed that file and why rcc makes its *.cpp version so big ? How to avoid this?
makefile for my project has such auto-generated line "C:\Qt\5.5\mingw492_32\bin\rcc.exe -name qml ..\TestProject\qml.qrc -o debug\qrc_qml.cpp" Runing the same command but with -binary option gives 70MB output. So, seems that -name option makes resulting *.cpp files x4 bigger and cause out of memory errors because compiler is not designed for such big files.
Still do not know how resolve this. Android has limitation 200Mb of *.apk file, so Qt must have such option to generate such big files. But in my case I can't generate 60 MB executable binary , very disappointed.

[ambershark]

Well when you turn a binary file into a cpp file it is basically taking the bytes of said binary and doing something like:

unsigned char bytes[] = { 0x00, 0x01, 0x02 };

So as you can see a single byte gets turned into at least 3 bytes in code. Technically more since you have the comma and space too.

A way to solve this (and not necessarily the best way, just the first one that pops into my mind) is to break up that byte array over multiple cpp files. So you can have:

File 1.cpp:
unsigned char bytesPart1[] = { ... };

File 2.cpp
unsigned char bytesPart2[] = { ... };

...

File X.cpp
unsigned char bytesPartX[] = { ... };

NOTE: I wouldn't necessarily use the stack for all those bytes. I would make that allocated heap memory. You are definitely going to run out of stack space doing huge files like that.

[Slv007]

@ambershark
Hello, thank you for your comment !
I have to notice you that every resource file in Qt project is translated into *.cpp and then used by compiler automatically. The only difference of how this works on different platforms. On iOS and Android binary package in the market store is just zipped archive with separate resources and executable file, so I can add almost any size resource file without touching compiler but Qt uses different way (starts from desktop OS way) uses translation resources into *.cpp files and this cause the problems.
By the way your idea seems may solve the problem, I'll try to split my encrypted file into several smaller ones.
Another thing I was not thought about that Qt may keep all integrated resources in memory, hm...this encrypted file is just read-only database I have distribute it within app on mobile platforms, but I do not need to keep it in RAM.

[ambershark]

@Slv007 I haven't used Qt for mobile platforms yet. Nor do I have much experience in mobile but couldn't you just include that file in the .aar or whatever archive you have for your application without converting it as a resource to a cpp file first?

It seems like there should be some way to distribute support files without making them a resource. I rarely ever include big resource files in my desktop Qt projects.

Like if I have a lot of images I would distribute an images/ directory with all my images in it and load them as needed rather than have them in resources.

And yes my guess is the resources would be loaded into memory. I do not know the underlying functionality of Qt there, but my guess/assumption is that it is loaded at all times into memory since it's part of the code. There isn't really a way to not have it loaded since you are setting a variable or piece of memory to the data you represent. Unless on program start they wrote it out to a temp file and then used shared memory to access it or something. Seems far fetched though.

[jsulm]

@Slv007 "This file is required to be integrated into executable binary because of distribution as *.apk file for Android and same for iOS." - at least for Android that's not true. APK is nothing else as a zip archive and can contain any data. No need to put this data into your executable. I'm quite sure for iOS it is similar.

[Slv007]

@ambershark
Split big data file into smaller does not itself solves the problem because resource file (*.qrc) whic contains list of resources is translated in one big *.cpp file and cause the same error. I was have to make a lot of *.qrc files each of each contains two 5MB parts of initial 60MB encrypted resources file and this helps.

<RCC>
    <qresource prefix="/">
        <file>local_data.p8</file>
        <file>local_data.p9</file>
	    </qresource>
</RCC>

Compiling of each *.qrc file takes about 1 minute. I think something goes absolutely wrong on how Qt manages resources.
Also I looked into *.apk and found that embedding user resources comes included as static library and loaded at startup into RAM. By the way there is a lot template *.qml files for default controls used by Qt core which are packed resources must be used in *.apk not as attached library.
Seems this is sort of bug and it has to be reported.

[ambershark]

Sorry I should have been more clear. When I said to split it up I meant to bypass the qrc and do it manually. You turn your binary file into cpp files. You could use a utility like xxd or write a small piece of code to do it yourself.

Anyway, you would then have to put those pieces back together anyway which Qt would not do for you.

Sorry I wasn't more clear on that.

From what @jsulm said you should be able to include the file in your apk without including it as a resource. That would be the appropriate path in my opinion. A file that big should not be a resource on any system.

[Slv007]

Here is right solution.
Just add "CONFIG += resources_big" into *.pro file.

[ambershark]

@Slv007 I had no idea that was a thing. Learn something new every day. :) I don't use qmake any more though, moved on to cmake long ago so maybe that is new since I last used qmake.

[supaiku_]

@Slv007

Sorry for necroposting, but I'm fascinated by the fact that it's nowhere documented? This magical flag just reduced my compile times from minutes to mere seconds and I don't run occasionally OOM anymore. How does it work?

[tekojo]

@supaiku_
I'll raise this as a documentation bug. The problem seems to come up regularly, and the solution is really simple.

[vineesh_tp]

@Slv007 : Thank you. This should be an acceptable answer.