Rendering FHD images as video

NguyenTuan

So I have lists of QImage that I want to render sequentially (like video), with the following requirements:

Each list may have around 1000 ~ 3000 images.
Each image is in FHD resolution (1080p).
Video should be played in decent framerate (30 FPS minimum, 60 FPS preferred).
Realtime image manipulation like white-balance, contrast, and gamma correction.

After doing some research and tinkering I found that the most promising solution is using OpenGL via QOpenGLWidget. But there is a problem, converting QImage to QOpenGLTexture is simply to slow and unstable (From my measurement, for each draw call, time needed to create texture can vary from 10ms to 60ms). This is kind of problem because I can only give 16ms max to each frame for achieving 60 FPS. Can anyone give me guidance on how to fix this? Or maybe, an alternative solution?

Here how my draw call were implemented:

void paintGL() override 
{
   glClearColor(0.1, 0.1, 0.1, 1.0);
   glClear(GL_COLOR_BUFFER_BIT);

   auto t1 = std::chrono::high_resolution_clock::now();

   QOpenGLTexture tex(*this->GetCurrentImageFrame()); // GetCurrentImageFrame return a pointer to current active QImage
   tex.setMagnificationFilter(QOpenGLTexture::Filter::Linear);
   tex.setMinificationFilter(QOpenGLTexture::Filter::Linear);
   tex.bind(1);

   auto t2 = std::chrono::high_resolution_clock::now();
   printf("Time to create texture: %ld ms\n", std::chrono::duration_cast<std::chrono::milliseconds>(t2-t1).count());

   m_program->bind();
   m_program->setUniformValue(m_program->uniformLocation("uTex"), 1);

   m_vao->bind(); // Setup rect for rendering the texture
   glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, 0);

   tex.release();
}

Also, my device is Lenovo ThinkPad x250, i5-5300U 2.30GHz, 8 GB RAM.

Thanks in advance.

jsulm

@NguyenTuan Did you see:
"Note: image is automatically converted to QImage::Format_RGBA8888 which may have performance implications for large images with a different format." (https://doc.qt.io/qt-6/qopengltexture.html#QOpenGLTexture-1). What format do you use for your images?

NguyenTuan

Actually I haven't read that part yet. My default image is in ARGB32 and by changing it to RGBA8888 I can reduce the loading time to 5 ~ 50 ms but it still doesn't solve the instability issue.

For now during normal usage (updating image using timer) my draw function fluctuate between 5 ~ 30 ms, but when dragging the window it can become 10 ~ 40ms and when resizing the window its fluctuate wildly between 6 ~ 50ms.

I can live with stopping the video when dragging and resizing as work around. But I still need to find a way to fix the fluctuation during normal usage, since currently the fluctuation cause the video to become kinda sluggish.

SGaist

Hi,

How are you applying your effects ?
If you want that kind of speed you basically have to optimize loading your data in GPU RAM and then do as much as possible there as well.

JoeCFD

@NguyenTuan
You can encode all images into video with a frame rate and a format you choose. And then play it with qt gui + qml sink or qt multimedia.
https://ffmpeg.org/doxygen/7.0/encode_video_8c-example.html

Gstreamer may have similar functions.

NguyenTuan

@SGaist

My plan is to use fragment shader for the effect and multiple uniform to control it. From my initial testing it's should run just fine. The only problem is time instability when converting QImage into QOpenGLTexture

Here is my shader code (for now it's only manipulate the color, but I will implement more later on):

#version 330 core
layout (location = 0) in vec3 aPos;
layout (location = 1) in vec3 aColor;
layout (location = 2) in vec2 aTexCoord;

out vec3 vColor;
out vec2 vTexCoord;

void main() {
	gl_Position = vec4(aPos, 1.0);
	vColor = aColor;
	vTexCoord = vec2(aTexCoord.x, aTexCoord.y);
}

#version 330 core
out vec4 FragColor;

in vec3 vColor;
in vec2 vTexCoord;

uniform sampler2D uTex;

void main() {
	FragColor = mix(texture(uTex, vTexCoord), vec4(vColor, 1.0), 0.4);
}

NguyenTuan

@JoeCFD

Unfortunately using a video isn't an option, since the list of QImage is crucial for other part of the application. I know I can extract a single frame from the video, but I don't know how performant it would be, especially when I need to do a lot of it.

Moreover, by using list of QImage it enable me to lazy load the images, which give the ability to instantly change the images list. I assume by using video I would need to load all the images first then encode it before it can be used for the application.

SGaist

Another question: why are you recreating the texture every time ? You can create it once and then update its content.
Also, do you need the mipmaps to be generated ?

NguyenTuan

@SGaist said in Rendering FHD images as video:

You can create it once and then update its content.

I've done that. Something kinda like this:

void initializeGL() override {
	initializeOpenGLFunctions();
	m_tex = new QOpenGLTexture(QOpenGLTexture::Target::Target2D);
	m_tex->create();
	m_tex->setSize(4000, 4000); // Maximum possible frame resolution
	m_tex->setFormat(QOpenGLTexture::RGBA8_UNorm);
	m_tex->allocateStorage(QOpenGLTexture::RGBA, QOpenGLTexture::UInt8);
	m_tex->bind(1);
}

void paintGL() override {
	QImage *img = this->GetCurrentImageFrame();
	m_tex->setData(0, 0, 0, img->width(), img->height(), 1, 0, QOpenGLTexture::RGBA, QOpenGLTexture::UInt8, img->constBits());
}

But there are no performance improvement.

I guess 30 FPS is the maximum my machine can do, so I would just move on with it. Maybe I will implement a toggle so user with more powerful machine can run it in 60 FPS.

I will now close the thread, thanks for everyone helping.