Resources text file unable to be opened in Windows, but works fine in Linux

tranxxenolab

Hello everyone, I have a strange issue. I am in the advanced stages of a project and have run into a showstopping problem. I am using PySide 6.9.2.

I have some somewhat large JSON files (around 150k) that I want to include with my program and read from the resources. I have successfully added the files to the resources and the program works just fine under Linux (Ubuntu 24.04)--the resource files can be loaded, the program runs just fine.

In Windows, however, the resources cannot be loaded. I get a generic "Input/Output" error.

This is the relevant code in question:

        file = QFile(":/json/resources/json/stars_magnitudes.json")
        if not file.open(QFile.ReadOnly | QFile.Text):
            print(file.errorString())
            return
        lines = []
        textStream = QTextStream(file)
        while not textStream.atEnd():
            lines.append(textStream.readLine())

        data = "\n".join(lines)
        starsByMagnitude = json.loads(data)

I am stumped why I am getting this error, and searching through the forums didn't give me any relevant results.

In case this might help solve things: I develop entirely within a Linux virtual machine and commit to a gitlab repository. On Windows for testing I simply pull from that repository into a virtual environment within Anaconda and run the code from the command-line. I don't compile the resources file on Windows but compile it on Linux, push it to the repository, and then use that same resources file on Windows. To date that hasn't caused any problems with any other resource file in my project (dozens of icons, other kinds of text files, etc).

Are there any suggestions of what could be going wrong here?

Thanks in advance.

tranxxenolab

As a further update, I've found that the issue only arises with large-ish JSON text files (over 100k) in the resources; my smaller JSON file, of only around 20k, works just fine on Windows.

Is there a maximum size for text files added to the resources on Windows?

JonB

@tranxxenolab
I do not use Windows, but do not see any mentioned limit on resource files (text or not, though obviously 2/4Gb may be a limit but not, say, >64k).

Start by removing the QFile.Text so it is opened as binary. Get rid of QTextStream and readLine() and use readAll() to slurp the whole content (as a Python bytes). Any difference? Use seek()/size()/bytesAvailable() while open via QFile to see what you have, and compare against the len(data) you send to JSON. Also try QFileInfo.size() on the resource file path to see what that reports, as that looks at the "file system" of the resource.

As an aside: since JSON format does not care about actual "lines" but only about "whitespace" you might permanently forget about opening this as a text file, reading lines, putting them in an array and joining to form a string, you can simply pass the whole blob from readAll(). Up to you.

I don't compile the resources file on Windows but compile it on Linux, push it to the repository, and then use that same resources file on Windows.

I don't know what the implications are. TBH I have not looked it up but I don't know how one does Qt resources from Python: from C++ they are compiled/linked into the final executable, but there is no such for Python scripts. You are also apparently supplying the same actual byte content for Linux and Windows. For a text file Linux stores just \n for end of line but under Windows it should be \r\n. Qt's QFile.Text/QTextStream may accept \n instead of \r\n under Windows, I don't know, but other applications like Notepad do not fair well.

Finally, just in case, you might like to temporarily move the identical resource content from resource to external file and verify this particular content goes through your code and JSON correctly.

Christian Ehrlicher

@JonB said in Resources text file unable to be opened in Windows, but works fine in Linux:

I don't compile the resources file on Windows but compile it on Linux, push it to the repository, and then use that same resources file on Windows.

Are you sure you push them as binary and not text?

tranxxenolab

@JonB Unfortunately trying to seek or get the size doesn't work, as the open command fails.

QFileInfo.size() returns 0 for one file, and -1 for another. It returns the correct size on the smaller JSON file.

On Linux the correct size is returned for all of the files.

Right now I am indeed doing the workaround of having the file outside of resources, but it's strange that it would be doing this and I'd like to figure out why :)

@Christian-Ehrlicher I enabled binary transfer using a .gitattributes and ensured that the file was changed and committed, and pulled. Nothing changed.

Both files have the same md5sum and are around 12 megabytes.

I'm stuck.

I'll try in the next day or two to see if I can make a MRE.

Thanks for the suggestions!

tranxxenolab

Hello everyone, I've created a minimum example with my actual data.

This is a simple example that opens a window and tries to read from three different JSON files in the resources. On Ubuntu 24.04 with Pyside 6.9.2, everything runs properly. On Windows and MacOS, reading fails with the larger 2 JSON files. Any help is appreciated!

The archive for the example can be downloaded from my Nextcloud (as I don't seem to have the ability to upload files here): https://nebula.tranxxenolab.net/s/Cob9ttPkZ9NmZzJ

JoeCFD

@tranxxenolab You have confirmed no issues when smaller json files are loaded. Can not you split your big json files into smaller ones? Then merge them after they are loaded. You may need to take a look at the source code and see how the loader works.

hskoglund

Yeah it might be something with the PySide resource system, I created a C++ Qt 6.9.2 Widget test app and got all 3 resource files to read in correctly both on Ubuntu 24.04 and Windows 11.

friedemannkleint

We had some issues with ARM ( QTBUG-140794 ), but plain Windows should work. The issue might be related though.

tranxxenolab

@hskoglund Yes, I've just confirmed that it does seem to be a difference between the way resources are created for PySide6 on Linux versus Windows. Running rcc -g python resource.qrc -o rc_resource.py (for the correct resource file) on Linux creates a very different file from what is created on Windows. In the particular example here, the files differ by almost 190kb between Windows (the larger file) and Linux. The number of lines is vastly different between the two files as well, by over a factor of 2 (as counted by wc, so it doesn't seem to just be a question of line endings).

But indeed, when I run rcc on Windows (which creates the larger file), the JSON files in the resources are read correctly on Windows. And, even more strangely, the resource file created on Windows can be properly read on Linux, but not the other way around.

I have no idea why this might be happening. I've submitted a bug report as it seems like extremely inconsistent behaviour: PYSIDE-3251

Christian Ehrlicher

I would guess a different compression is used which is not available on Windows

tranxxenolab

@Christian-Ehrlicher fair enough, but I haven't seen this documented anywhere, which it should be at least.

Christian Ehrlicher

You normally don't share resources between different operating systems as they are embedded in an executable. Now the problem is that python is somehwat special wrt this so...

tranxxenolab

@Christian-Ehrlicher Sure, although there are certainly use cases for compiling resources on one platform, checking into git, and then using those resources, especially during testing, on another platform. (@friedemannkleint talked about a similar use case.) To have resource files not be cross-platform compatible, but only in certain cases, should be something that is documented.

tranxxenolab

For anyone else who is having issues similar to mine: I have confirmed that by running pyside6-rcc with --compress-algo zlib allows for the creation of a resources file that works cross-platform, at the cost of a larger resources file size. So, if you need cross-platform compatibility for your resources files that are generated on Linux but pulled on other platforms, be sure to use zlib as the compression algorithm, rather than the default of zstd (which isn't even available on Windows).

I'm going to mark this as solved, but will be going through the bug report I mentioned above to hopefully get some documentation written about this inconsistency.