QWidget Flickering for frame by frame streaming of video

SilverSurfer

Is there any way I can avoid loadFromData() call, which I assume is doing a copy somewhere ?

To rephrase my question:
Can't I draw an Image from the existing buffer without a need for the copy? If yes , How?

SGaist

Hi,

How are you getting the data in the first place ?

SilverSurfer

@SGaist
By making a call to a server to fill the passed buffer.

//time take by this depends size of buffer to be fetched
//which may be more than 34 ms depending on size
fetchBitmap(handle, cameraId,(uchar*)buffer , x_width, y_height);

emit bufferAvailable(buffer,size); //calls getImageData(..) slot in the UI thread

Thanks!

SGaist

What is that server ?

What does it provide as API to retrieve images ?

SilverSurfer

@SGaist
It is a Video Management Server(ExacqVision Server to be precise).

Yes, the API( fetchBitmap(...)) is provided by the server.
So I don't think the data retrieval time can be reduced.

What I would Ideally like to do is avoid any copying for this data.
Which happens when I do pixmap->loadFromData(...) .

SGaist

Is their API documentation accessible somewhere ?

If you pass the buffer to the fetchBimap method, then you should be able to pass a QImage buffer pointer and in this case, the data will be copied directly on the QImage that you can then manipulate.

SilverSurfer

@SGaist
No the Documentation is not available publicly.

It take a uint8_t* as buffer argument.

//registered with server
Callback(event)
{
    if(event == LIVE_FRAME_AVAILABLE)
{
    frameSize = fetchBitmap(NULL); 
    // Size fetched everytime as this can change with every frame if image is resized
    vector.resize(framesize)
    fetchBitmap(&vector[0]);
    emit bufferAvailable(&vector[0],frameSize);
}
}

I am not sure how can I use QImage pointer here.

SGaist

QImage::bits.

Make the QImage the correct format and size.

SilverSurfer

@SGaist
Tried using
QImage::QImage(uchar * data, int width, int height, Format format, QImageCleanupFunction cleanupFunction = 0, void * cleanupInfo = 0)

data = pixel data , updated by fetchBitmap(..) function above.
format = RGB888
where as per documentation it does not do a deep copy. So this helps.

But my image has R and B planes swapped.
Can't seem to find the appropriate format .

P.S: I am on a Windows machine.

Thanks,
Kshitij Mayekar

SGaist

See QImage::rgbSwapped

SilverSurfer

@SGaist
QImage::rgbSwapped results in creating new QImage with is equivalent to deep copy. So It does not help. Is there any other way to dislaying the RGB planes correctly ?

Note: Whenever I write the contents of the buffer to a file with appropriate headers I could see the Image correctly when I open the Image through windows photo viewer.

Original Problem Statement : Display Byte Buffer(uchar*) to a QWidget without the need to deep Copy .

SGaist

Where are these frames coming from ?

SilverSurfer

@SGaist

These frames are received from a server.

//Api that talks to server
fetchBitmap(&vector[0]);

I do not have access to server code.

P.S: Server Machine is Linux and Client is Windows . Could this be the reason for RGB /BGR mismatch ?

wrosecrans

Is there any chance of the paint event being triggered by the system at a moment when the pixmap isn't fully populated? If that is possible, you would get flickering regardless of whether the load function is faster or slower than real time.

mrjj

@SilverSurfer said in QWidget Flickering for frame by frame streaming of video:

Could this be the reason for RGB /BGR mismatch ?

Well there is always Endianness but if using same sort of CPU in both ends, its likely not related to that.

SilverSurfer

@mrjj
Both machines are little endian.

mrjj

@SilverSurfer
Ok. so most likely they
are just using openCV and send as BGR888

So i dont see any way to convert to supported format without copy.

wrosecrans

@mrjj If you use a shader in a QOpenGLWidget, you can swizzle the r/g/b around arbitrarily when you do the drawing into the frame buffer, which would avoid an extra copy step to normalize the channel order.

SilverSurfer

@wrosecrans said in QWidget Flickering for frame by frame streaming of video:

you can swizzle the r/g/b around arbitrarily when you do the drawing into the frame buffer

Could you provide sample code for the same ?

wrosecrans

@SilverSurfer I don't have the time handy to do a full working example, but this is the technique in glsl:
https://www.khronos.org/opengl/wiki/Data_Type_(GLSL)#Swizzling

(The vec4 can hold "rgba" values if that's what you put in it - the example there just calls it "xyzw" because the data type generically handles any 4 floats.)

Basically, start from an example that draws an image with a simple glsl shader as a texture. Then hack on the shader until it flips around the colors like you want when it draws the texture. Then use your code that gets the image over the network to upload the image as the OpenGL texture every frame.