Bug in processing surefire/failsafe groups

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug in processing surefire/failsafe groups

Christopher
Found a bug in maven-failsafe-plugin (
https://github.com/apache/accumulo/issues/411) when using groups (JUnit
categories):
http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#groups
http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#excludedGroups

The problem is when the class corresponding to a configured group is not on
the class path. This is particularly problematic in multi-module builds, if
a group is specified which exists in one module but not another one.

It appears the bug occurs because the configured group/category is loaded
from the class path, and then compared to the classes to find matching
tests. However, this seems backwards. If the tests were examined first,
then the name of the groups could be matched against the configuration
without loading to load the configured groups/category. Even if this is not
possible due to some limitation on how these groups are passed to JUnit or
some other restriction, a ClassNotFoundException (CNFE) should be handled
more sanely: For example, in the case of a CNFE for the "groups" parameter,
then it's clearly not possible for any tests to be included, so no tests
should run. In the case of a CNFE for "excludedGroups", then it's clearly
not possible for any tests to match, so none should be excluded.

This bug has got me thinking about a related feature I'd like to see in
maven-surefire-plugin and maven-failsafe-plugin: I'd like to see groups /
excludedGroups specified by patterns, similar to `-Dtest=MyTest` instead of
`-Dtest=org.apache.project.package.subpackage.testing.MyTest`, it would be
nice to set `-Dgroups=MyGroup` rather than
`-Dgroups=org.apache.project.package.subpackage.testing.categories.MyGroup`.
If groups could be specified as patterns, it would be more obvious that a
CNFE could not occur... if a pattern didn't match any classes, then it
wouldn't result in a CNFE... it would simply return an empty set.

Thanks,

Christopher
Reply | Threaded
Open this post in threaded view
|

Re: Bug in processing surefire/failsafe groups

Christopher
Enrico: Thanks for the info. I'm using the version inherited from the ASF
parent POM 19, which turns out to be 2.20.1. Version 2.21.0 seems to fix
the problem. Thanks!

Tibor: No, I didn't get previous email. The Maven lists are too spammy for
me to subscribe. That could be the reason. Please CC me if follow-up is
needed. Thanks.

On Sat, Mar 31, 2018 at 10:16 AM Enrico Olivelli <[hidden email]>
wrote:

> This problem has been fixed in 2.21.0, what version are you running?
>
> Enrico
>
>
> Il sab 31 mar 2018, 14:54 Tibor Digana <[hidden email]> ha
> scritto:
>
>> Christopher,
>>
>> Did you receive my previous answer?
>> There is some problem with email delivery.
>> Tibor
>>
>> On Fri, Mar 30, 2018 at 8:58 PM, Christopher <[hidden email]> wrote:
>>
>> > Found a bug in maven-failsafe-plugin (
>> > https://github.com/apache/accumulo/issues/411) when using groups (JUnit
>> > categories):
>> > http://maven.apache.org/surefire/maven-failsafe-
>> > plugin/integration-test-mojo.html#groups
>> > http://maven.apache.org/surefire/maven-failsafe-
>> > plugin/integration-test-mojo.html#excludedGroups
>> >
>> > The problem is when the class corresponding to a configured group is
>> not on
>> > the class path. This is particularly problematic in multi-module
>> builds, if
>> > a group is specified which exists in one module but not another one.
>> >
>> > It appears the bug occurs because the configured group/category is
>> loaded
>> > from the class path, and then compared to the classes to find matching
>> > tests. However, this seems backwards. If the tests were examined first,
>> > then the name of the groups could be matched against the configuration
>> > without loading to load the configured groups/category. Even if this is
>> not
>> > possible due to some limitation on how these groups are passed to JUnit
>> or
>> > some other restriction, a ClassNotFoundException (CNFE) should be
>> handled
>> > more sanely: For example, in the case of a CNFE for the "groups"
>> parameter,
>> > then it's clearly not possible for any tests to be included, so no tests
>> > should run. In the case of a CNFE for "excludedGroups", then it's
>> clearly
>> > not possible for any tests to match, so none should be excluded.
>> >
>> > This bug has got me thinking about a related feature I'd like to see in
>> > maven-surefire-plugin and maven-failsafe-plugin: I'd like to see groups
>> /
>> > excludedGroups specified by patterns, similar to `-Dtest=MyTest`
>> instead of
>> > `-Dtest=org.apache.project.package.subpackage.testing.MyTest`, it would
>> be
>> > nice to set `-Dgroups=MyGroup` rather than
>> > `-Dgroups=org.apache.project.package.subpackage.testing.
>> > categories.MyGroup`.
>> > If groups could be specified as patterns, it would be more obvious that
>> a
>> > CNFE could not occur... if a pattern didn't match any classes, then it
>> > wouldn't result in a CNFE... it would simply return an empty set.
>> >
>> > Thanks,
>> >
>> > Christopher
>> >
>>
> --
>
>
> -- Enrico Olivelli
>