Processing Every Frame in a Video File

PetrichorShark

Hi! I'm working on an application that processes video, which I am performing using video filters for a VideoOutput. The source for this VideoOutput is a MediaPlayer, which has a video file (e.g., an .mp4) as input. That is, my QML basically looks like this:

MediaPlayer {
    id: playVid
    source: "someFilePath/someVideo.mp4"
    autoPlay: true
}

VideoOutput {
    id: videoOutput
    source: playVid
    filters: [ someVidFilter ]
    ...
}

SomeVidFilter {
    id: someVidFilter
}

However, with this setup, if the filter takes "n" seconds to process a frame, the next frame it receives will be the one from "n" seconds further into the video, skipping all of the frames in-between. This behaviour makes sense, but it's not what I require. I need to process every frame in the video, in order.

Is there a way to easily do this with Qt? After a fair bit of reading through the documentation and similar questions online, it looks like the only option is something like this StackOverflow post, where setPosition() is used to select the desired frame each time.

Is this the best/only way to achieve this in Qt? I'm a bit hesitant to implement the above because I'm not very familiar with Qt/QML and so I don't know exactly where setPosition()/play() would need to be called to get the expected result. E.g., would it be acceptable to call them at the very end of my QVideoFilterRunnable's run() implementation? Or would the MediaPlayer consider time to have passed since that setPosition() call, giving me a frame further into the video than what I want?

SGaist

Hi and welcome to devnet,

I would recommend switching to PySide6 and Qt6. The QtMultimedia module has seen a big overhaul that might help in your situation.

PetrichorShark

Thanks! I've given the new QtMultimedia module a look, but I did not see anything new that would address this problem. I could receive video frames in a video sink as opposed to a filter or abstract video surface, but I don't think that would change which video frames Qt gives me; it'd just be a different way to handle the same frames. Is this reasoning correct, or am I missing a key feature of the overhaul?

SGaist

If memory serves well, the video sink should give you each and every frame of the video for you to process in a more consistent manner.

That said, how long does your processing take ?

PetrichorShark

@SGaist Ah, in that case, I'll give it a try. Would that also be the case with QAbstractVideoSurface, or would that one receive further-in frames if more time passes?

Processing takes about 250ms per frame on Android right now. I'm working on ways to speed that up, but the processing fps is going to be worse than the video file's fps for some time yet.