Test suite concept

ck_ee

Hi,

what would be the best strategy for managing a lot of unit tests for many different libaries? I'd like to have a testsuite which can selectively run tests or simply run every test there is. My approach would be to build a single testsuite application which has a a single testmanager object which holds the unit tests objects with some kind of id/name system. This testsuite would be accesed via command line with many different options like -all or -lib "mylib" and so on. This application would of course depend on all libs that need to be tested and would include all the sources of the testcases.

https://marcoarena.wordpress.com/2012/06/23/increase-your-qtest-productivity/ comes close to what i have in mind.

What would be the downside to a strategy like that and are there better alternatives? I would use the QT Test framework.

VRonin

Th QtTest module is not that flexible. To make it work properly you'll need a single project for every class to want to test and that will produce a binary file for each of them. You are then free to include a batch file that runs each test app based on argument passed but I'd do it outside Qt altogether

ck_ee

@VRonin
you can do a QTest::qExec(Qbject*) on your TestClasses in a simple console application

int main() {
    int ret1 = QTest::qExec(pMyTest1);
    int ret2 = QTest::qExec(pMyTest2);
    int ret3 = QTest::qExec(pMyTest3);
}

The test classes need to have the QObject macro and the testfunctions.

VRonin

@ck_ee said:

@VRonin
you can do a QTest::qExec(Qbject*) on your TestClasses

Unfortunately while the overall pass/fail result works that way, if you try to export the results (in xUnit format for example) you'll only get those from the last test, pMyTest3 in that example. That's why I recommend to separate them as meaningful output is the most important thing in a test imho

ck_ee

This post is deleted!

ck_ee

you could redirect the console output to a file - for example:

myTest.exe -xml > result.xml

ck_ee

@VRonin

i found another approach:
http://stackoverflow.com/questions/24800912/how-to-compose-multiple-unit-test-result-in-a-single-txt-file?rq=1

QList<QObject *> objects;
objects << new TestSpinBox << new TestComboBox;

QString result;
foreach (QObject *o, objects) {
    QTemporaryFile f;
    f.open();
    QStringList args = app.arguments();
    args << "-o" << f.fileName();
    QTest::qExec(o, args);
    result += "\r\n" + f.readAll();
}
qDeleteAll(objects);

you collect test object as i proposed but then set the output file in the argument list on a temporary file and concate that in a QString which then could be wrote anywhere you want.

VRonin

That would still create an invalid xml file if you use xUnit output.
I use jenkins so the results should be readable from it too. Agree you can handle the xml file manually to make it valid but it looks like a lot of useless work compared to a single line in (windows) batch if you split the tests

for /f %%f in ('dir /b "Tests\*.exe"') do Tests\%%f -o %%f.xml,xml