Improve rendering performance while integrating libmpv with QtQuick (+Python)
-
Hi all,
tl;dr: By adhering to the libmpv examples, we can seamlessly render videos into the default application's framebuffer. However, this may result in the UI's frame rate being limited to that of the video or necessitate manual synchronization of video and audio.
Together with a friend, I have been building a small application using Python + libmpv: mpvQC. Our primary challenge lies in enhancing video rendering performance.
After following these examples:
- https://github.com/mpv-player/mpv-examples/tree/master/libmpv/qml
- https://gitlab.com/robozman/python-mpv-qml-example
we managed to get libmpv working with PySide2 & PySide6: https://github.com/trin94/qtquick-mpv
The underlying concept revolves around configuring QtQuick and libmpv to facilitate the utilization of QQuickFramebufferObject. This approach mandates the implementation of a QQuickFramebufferObject::Renderer, featuring a render() method that invokes libmpv's render function to render into the default framebuffer. It's been working effectively, achieving audio-video synchronization without excessive complexity from a developer's perspective.
However, there's a drawback: it uses the default framebuffer, which means libmpv synchronizes video frames with audio, causing the entire UI's frame rate (including animations and scrolling) to be capped at the video's frame rate, typically 24 or 30 fps. This results in a less-than-ideal user experience. This issue has been discussed before, as highlighted in this YouTube video.
Various alternative approaches can be explored. One option is to instruct mpv not to sync video frames with audio, which would improve the frame rate but require addressing audio drifts and other low-level challenges. Another suggestion from the libmpv example is to render into a different framebuffer and manually draw it.
We've been on the lookout for examples demonstrating this approach but haven't found a suitable one. We also examined the information provided in Making Movies with QML but couldn't adapt it to our specific case.
We've created a minimal working example to illustrate the problem: https://github.com/trin94/qtquick-rendering-into-framebuffers
Do any of you have insights on how to enhance rendering performance, such as rendering into an offscreen screen buffer on a separate thread, and could provide guidance on its implementation?
We would greatly appreciate any assistance!