QThreads and libraries

[drbroly]

Hi,

I'm trying to implement parallel processing for my camera streams. The plan is to spawn multiple threads, each handling a camera. Using OpenCV, the frame is taken and my custom library will perform various image processing techniques on it. When the process is done, a signal is emitted to inform the GUI to display the image.

The process works fine when a single QThread is handling a camera. But as I introduce a 2nd thread, a memory error is thrown from my library. It looks like they're sharing the same resource and conflicting even though each camera object contains its own copy of the library. If I increase the timer to something high (4000ms +), they no longer conflict but the result is not functionally practical. I cannot pin it down and don't know how to tackle this.

--Appended .h files--

#include <QObject>
#include <QTimer>
#include <opencv2/core/mat.hpp>
#include "TISAlib.h"

class Camera : public QObject
{
	Q_OBJECT

public:
	Camera(int index, int slot);
	~Camera();
	int cameraIndex;
	int assignedSlot;

private:
	QTimer timer;
	TISAlib library;
	cv::VideoCapture* capture;
	QImage cvMatToQImage(const cv::Mat& inptMat);

	cv::Mat frame;
	QImage img;
	std::vector<cv::Rect>objectLocations;
	std::vector<cv::Mat> objList;

private slots:
	void run();

	signals:
	void imageReady(int, QImage, cv::Mat, std::vector<cv::Mat>);


};

My camera object

Camera::Camera(int index, int slot)
{
	cameraIndex = index;
	assignedSlot = slot;
	capture = new cv::VideoCapture(cameraIndex);

	procLib= new ImageProclib();
	connect(&timer, SIGNAL(timeout()), this, SLOT(run()));
	timer.start(1);
}

void Camera::run()
{
	qWarning() << "Timer TICKED from: " << QThread::currentThreadId();

	QElapsedTimer timer;
	timer.start();

	(*capture) >> frame;

	objectLocations= procLib->getObjLocations(frame);

	if (objectLocations.size() > 0)
	{
		objList = procLib->getAllObjImages(objectLocations, frame);
		frame = procLib->highlightObjsInFrame(frame, objectLocations);
	}
	img = cvMatToQImage(frame);
	qDebug() << "This operation took" << timer.elapsed() << "milliseconds";
	emit imageReady(assignedSlot, img, frame, faceList);
}

How i'm creating the camera threads and initiating them.
Controler.h

#include <QObject>
#include "Camera.h"
#include "QThread"

class Controller : public QObject
{
	Q_OBJECT

public:
	Controller(QObject *parent);
	~Controller();

private:
	QThread* thread;
	QThread* thread2;

public slots:
	void onImageReady(int, QImage, cv::Mat, std::vector<cv::Mat>);

	signals:
	void imageReady(int, QImage);
};

controller.cpp

        thread = new QThread;
	Camera* cam = new Camera(0,0);
	qRegisterMetaType< cv::Mat >("cv::Mat");
	qRegisterMetaType< std::vector<cv::Mat> >("std::vector<cv::Mat>");
	connect(cam, SIGNAL(imageReady(int, QImage, cv::Mat, std::vector<cv::Mat>)), this, SLOT(onImageReady(int, QImage, cv::Mat, std::vector<cv::Mat>)));
	cam->moveToThread(thread);
	thread->start();

/* Having both causes memory errors
	thread2 = new QThread;
	Camera* cam2 = new Camera(1,1);
	connect(cam2, SIGNAL(imageReady(int, QImage, cv::Mat, std::vector<cv::Mat>)), this, SLOT(onImageReady(int, QImage, cv::Mat, std::vector<cv::Mat>)));
	cam2->moveToThread(thread2);
	thread2->start();*/

Any insight as to what i'm doing wrong would be greatly appreciated.

[A Former User]

Please show more code. What is frame?

[kshegunov]

And also a stack trace, please.

[drbroly]

Thanks for getting back to me.

@Wieland - Frame is a cv::Mat object. It's the captured image from the cameraI've appended the .h for greater context.

@kshegunov

 	KernelBase.dll!00007ffb4b797788()	Unknown
 	msvcr120d.dll!00007ffafe4ec366()	Unknown
 	opencv_core320d.dll!00007ffaff143278()	Unknown
 	opencv_core320d.dll!00007ffaff143407()	Unknown
 	opencv_core320d.dll!00007ffafeed2c49()	Unknown
 	opencv_core320d.dll!00007ffafeed0d08()	Unknown
 	opencv_core320d.dll!00007ffaff1eaab7()	Unknown
 	opencv_cudaobjdetect320d.dll!00007ffafbcb1858()	Unknown
 	opencv_cudaobjdetect320d.dll!00007ffafbcbad0e()	Unknown
>	TISAlib.dll!objDetector::getobjectLocations(cv::Mat frame) Line 83 C++
 	TISAlib.dll!TISAlib::getObjLocations(cv::Mat frame) Line 74 C++
 	Project TISA.exe!Camera::run() Line 30 C++
 	Project TISA.exe!Camera::qt_static_metacall(QObject * _o, QMetaObject::Call _c, int _id, void * * _a) Line 85 C++
 	Qt5Cored.dll!QMetaCallEvent::placeMetaCall(QObject * object) Line 503 C++
 	Qt5Cored.dll!QObject::event(QEvent * e) Line 1263 C++
 	Qt5Widgetsd.dll!QApplicationPrivate::notify_helper(QObject * receiver, QEvent * e) Line 3745 C++
 	Qt5Widgetsd.dll!QApplication::notify(QObject * receiver, QEvent * e) Line 3105 C++
 	Qt5Cored.dll!QCoreApplication::notifyInternal2(QObject * receiver, QEvent * event) Line 988 C++
 	Qt5Cored.dll!QCoreApplication::sendEvent(QObject * receiver, QEvent * event) Line 231 C++
 	Qt5Cored.dll!QCoreApplicationPrivate::sendPostedEvents(QObject * receiver, int event_type, QThreadData * data) Line 1648 C++
 	Qt5Cored.dll!QEventDispatcherWin32::sendPostedEvents() Line 1059 C++
 	Qt5Cored.dll!qt_internal_proc(HWND__ * hwnd, unsigned int message, unsigned __int64 wp, __int64 lp) Line 223 C++
 	[External Code]	
 	Qt5Cored.dll!QEventDispatcherWin32::processEvents(QFlags<enum QEventLoop::ProcessEventsFlag> flags) Line 613	C++
 	Qt5Cored.dll!QEventLoop::processEvents(QFlags<enum QEventLoop::ProcessEventsFlag> flags) Line 135 C++
 	Qt5Cored.dll!QEventLoop::exec(QFlags<enum QEventLoop::ProcessEventsFlag> flags) Line 212 C++
 	Qt5Cored.dll!QThread::exec() Line 507	C++
 	Qt5Cored.dll!QThread::run() Line 575	C++
 	Qt5Cored.dll!QThreadPrivate::start(void * arg) Line 380	C++
 	[External Code]	

in case it's of use, this is the method it landed on:

std::vector<cv::Rect> objDetector::getobjectLocations(cv::Mat frame)
{
	std::vector<cv::Rect> objects;
	cv::Mat frame_cpu, gray_cpu, resized_cpu, frameDisp, image;

	cv::cuda::GpuMat frame_gpu, gray_gpu, resized_gpu, objBuf_gpu;

	(image.empty() ? frame : image).copyTo(frame_cpu);
	frame_gpu.upload(image.empty() ? frame : image);

	convertAndResize(frame_gpu, gray_gpu, resized_gpu, scaleFactor);
	convertAndResizeCPU(frame_cpu, gray_cpu, resized_cpu, scaleFactor);

	ballCascadeClassifier_GPU->setFindLargestObject(findLargestObject);
	ballCascadeClassifier_GPU->setScaleFactor(1.2);
	ballCascadeClassifier_GPU->setMinNeighbors((filterRects || findLargestObject) ? 4 : 0);

	ballCascadeClassifier_GPU->detectMultiScale(resized_gpu, objBuf_gpu);
	ballCascadeClassifier_GPU->convert(facesBuf_gpu, faces);

	return objects;
}

Where the only external variable is: cv::Ptrcv::cuda::CascadeClassifier ballCascadeClassifier_GPU - is private in .h

[J.Hilk]

@drbroly

Ah, I run into the same problem, when I made my face recognition software using opencv.

each thread needs its own cascade.xml files, those are read during runtime and can and will conflict if you access them from different threads simultaniously.

[drbroly]

@J.Hilk

Thanks for your input! I have moved my method locally to each thread, with the resources they need but I still get the same result.

In your situation, were you using the GPU to use the cascade, or the CPU? I'm beginning to suspect the GPU implementation is a resource that can't handle requests from multiple threads.

[J.Hilk]

@drbroly

It's been a while since I used openCV, but if I remember correctly, Iused the default settings for it and that ought to be CPU-computation.

That work was for an android app, in a time, when QCamera did not work properly, QML-Videoframes were not handled correctly in cpp-code and openCV-Camera driver were horribly outdated. Let's say I had happier moments in my career. Those I remember better ...