QtMultimedia (GStreamer backend) memory leak with hardware V4L2 decoder (v4l2h264dec) on Raspberry Pi 3 / Wayland — RSS grows continuously until crash

[The Qt Mayssa]

Environment

• Qt 5.15.17 (QtMultimedia, GStreamer backend)
• Raspberry Pi 3, Raspberry Pi OS Trixie (Debian 13-based), 64-bit
• Wayland / labwc compositor (no X11)
• GStreamer 1.26.2
• Hardware H.264 decode via v4l2h264dec (V4L2 M2M stateful decoder, /dev/video10)
• Video playback done through QML import QtMultimedia 5.0, Video { } element (standard MediaPlayer + VideoOutput, no custom pipeline)

Summary

On Raspberry Pi 3 (Trixie), our application plays a rotation of short video clips through QtMultimedia's QML Video element. Whenever a new clip starts (including looping the same clip), our QML code resets and reassigns the source property to (re)start playback, which is standard practice with this API. Each of these reassignments causes QtMultimedia's GStreamer backend to tear down the previous internal pipeline and construct a new one.

When hardware decoding is active (v4l2h264dec), the process's resident memory (VmRSS) grows continuously and monotonically across these teardown/rebuild cycles, with no plateau, until the process eventually crashes (memory exhaustion). When hardware decoding is disabled (forcing software decode via avdec_h264), memory stays flat indefinitely under the exact same playback pattern.

This is fully reproducible and isolated with a clean A/B test on a single device, changing only one variable.

Test methodology

Same QML widget code, same video file, same playback pattern (looping / rotating clips, each triggering a source reassignment and full pipeline reconstruction) in both cases below.

Test A — hardware decode forced:

export GST_PLUGIN_FEATURE_RANK="v4l2h264dec:MAX"

Result: VmRSS (/proc/<pid>/status) increases continuously over time, with no stabilization, until the process is killed by memory exhaustion.

Test B — hardware decode disabled:

export GST_PLUGIN_FEATURE_RANK="v4l2h264dec:NONE"

This forces decodebin to fall back to the software decoder (avdec_h264). Result: VmRSS remains flat and stable over the same playback duration and number of pipeline rebuild cycles.

Cross-check on different hardware: the identical QML widget code, unmodified, run on a Raspberry Pi 5 (also Trixie/Wayland) shows no RSS growth at all. The Pi 5's SoC (BCM2712) has no dedicated hardware H.264 decode path, so it always uses software decode there — consistent with Test B above.

Factors ruled out

• File descriptor leak: monitored /proc/<pid>/fd for handles to /dev/video10 throughout multiple playback/loop cycles. Count stayed constant (4 fds, all opened once, never growing), so this is not a V4L2 device-handle leak.
• Thermal throttling: monitored vcgencmd measure_temp / vcgencmd get_throttled during the RSS growth. Temperature stayed in a normal range (~71°C) and throttled=0x0 throughout (no throttling now or historically), so the RSS growth is not a side effect of thermal frequency scaling.
• Application/QML logic bug in isolation: the same widget code, with the same source reassignment pattern, does not leak at all when hardware decode is not in play (Test B, and Pi 5 cross-check). So the QML-side reassignment pattern is only the trigger (it's what causes pipeline teardown/rebuild to happen repeatedly), not the underlying defect.

Possible causes (not yet pinpointed to a specific line, looking for input)

Given the leak only manifests when v4l2h264dec output buffers are involved, and does not correlate with descriptor count, our current working hypotheses are:

• Incomplete unref/cleanup of GstBuffer/GstMemory objects backed by DMABUF (the V4L2 CAPTURE queue buffers exported as DMABUF for zero-copy) when QtMultimedia's GStreamer video renderer tears down the pipeline (state change to NULL) as part of rebuilding it for a new source.
• Incomplete release of GL/EGL resources (EGLImage import from DMABUF, GL textures) used internally by Qt's video sink/QSGVideoNode texture upload path, tied to the DMABUF pipeline specifically (software-decoded frames go through a plain memory copy path that doesn't have this).
• A reference-counting issue in the V4L2 buffer pool (GstV4l2BufferPool or equivalent) when the OUTPUT/CAPTURE queues are repeatedly renegotiated across multiple pipeline construct/destroy cycles, without the underlying device being closed (consistent with the fd count staying flat — the pool/buffers may not be released even though the device stays open).
• Possibly specific to the Trixie (Debian 13) GStreamer 1.26.2 / kernel / Mesa V3D driver stack, since this is a V4L2 M2M + DMABUF + GL interop path that has changed significantly from the legacy OMX driver used on older Raspberry Pi OS releases.

Question

Has anyone else observed memory growth specific to v4l2h264dec (or V4L2 M2M decoders in general) combined with QtMultimedia's GStreamer backend when repeatedly tearing down/rebuilding the pipeline (e.g., via source changes on a QML Video/MediaPlayer)? Is this a known issue in the Qt5 Multimedia GStreamer plugin's handling of DMABUF-backed buffers, or would this be better reported against GStreamer's v4l2h264dec/v4l2bufferpool itself?

Happy to run additional diagnostics (GST_DEBUG traces, GST_TRACER=leaks, valgrind) if that would help narrow this down — will update this thread with results.

[Nils Petter Skalerud]

Hello, just a friendly reminder that Qt 5 is end-of-life, so we won't be able to provide any bugfixes for this issue from the Multimedia team. If you want our support, this issue will have to reproduced in a more modern Qt version (6.8 and 6.11 are good candidates). I'll try asking one of our GStreamer-experts when I have the chance whether this issue is fixed in recent Qt versions.

Beyond that, let's hope someone from community has some insight.

[The Qt Mayssa]

@Nils-Petter-Skalerud i'm aware of this ! that's why i needed to know if there's a known fix or an export maybe that resolve this without touching the qt code or at least my qml code to make it work.
I'll be waiting for your respond ! i hope you'll not forget to ask the expert, this can really help me :') thank you !!