Error in counting skipped tests in test center stats
-
Hello,
In our suite we have some test where we use the method test.skip(). As you can see in the picture it is well displayed but here we find an issue in the passed pourcentage.
The success of passed takes into account the total of failed + skipped + passed test cases. For us it's a problem because we use test.skip because it is important that those test cases are skipped and completely ignore by the test center.
Is there a way to update this?
Thank you
-
Hello,
In our suite we have some test where we use the method test.skip(). As you can see in the picture it is well displayed but here we find an issue in the passed pourcentage.
The success of passed takes into account the total of failed + skipped + passed test cases. For us it's a problem because we use test.skip because it is important that those test cases are skipped and completely ignore by the test center.
Is there a way to update this?
Thank you
@SylvainA We could potentially make this more configurable in the future, but I don't think it could be the default, as how test.skip() is used will vary from customer to customer and some might want to see how many tests have been skipped. It might for example include temporarily excluded tests that you might not want to forget about.
The skips already don't aggregate up to the top level. So at the suite level you should not see them. That the previous results don't show them is probably a bug.
I would suggest to not execute the tests at all instead of using test.skip() for now.
-
Hello,
In our suite we have some test where we use the method test.skip(). As you can see in the picture it is well displayed but here we find an issue in the passed pourcentage.
The success of passed takes into account the total of failed + skipped + passed test cases. For us it's a problem because we use test.skip because it is important that those test cases are skipped and completely ignore by the test center.
Is there a way to update this?
Thank you
-
Hello,
In our suite we have some test where we use the method test.skip(). As you can see in the picture it is well displayed but here we find an issue in the passed pourcentage.
The success of passed takes into account the total of failed + skipped + passed test cases. For us it's a problem because we use test.skip because it is important that those test cases are skipped and completely ignore by the test center.
Is there a way to update this?
Thank you
@SylvainA We could potentially make this more configurable in the future, but I don't think it could be the default, as how test.skip() is used will vary from customer to customer and some might want to see how many tests have been skipped. It might for example include temporarily excluded tests that you might not want to forget about.
The skips already don't aggregate up to the top level. So at the suite level you should not see them. That the previous results don't show them is probably a bug.
I would suggest to not execute the tests at all instead of using test.skip() for now.
-
@SylvainA We could potentially make this more configurable in the future, but I don't think it could be the default, as how test.skip() is used will vary from customer to customer and some might want to see how many tests have been skipped. It might for example include temporarily excluded tests that you might not want to forget about.
The skips already don't aggregate up to the top level. So at the suite level you should not see them. That the previous results don't show them is probably a bug.
I would suggest to not execute the tests at all instead of using test.skip() for now.
@Nico-Kaufmann
I understand. We launch test case on 2 different configurations (A and B).
If we detect config A some TC should not run so we have a simple condition like this:if config == A:
test.skip()
Maybe there is a better solution but because we have a lot of cases we can't just put on the cmd which test cases should be ignore.
Maybe if we play with tags it can be better. -
S SylvainA has marked this topic as solved on
-
@SylvainA
The older result also doesn't take the skipped status. It showes passed instead of skipped.
