Surefire 3.0.0-M4 not failing build on errors

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Surefire 3.0.0-M4 not failing build on errors

Jeff Jensen
I looked for this issue in JIRA but haven't found anything yet.  Anyone
know if this has been reported and/or fixed?

Our scenario is a failure occurs at startup of tests but Surefire doesn't
fail the build.  Specifically, Spring controller tests aren't failing when
there is a Spring configuration problem.

Spring top-level thrown exception is:
  java.lang.IllegalStateException: Failed to load ApplicationContext

but Surefire doesn't report fail.  Surefire output results is:
  Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,

Running the tests in an IDE correctly fail for the setup problem.
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Enrico Olivelli
Jeff

Il Sab 13 Giu 2020, 00:15 Jeff Jensen <[hidden email]>
ha scritto:

> I looked for this issue in JIRA but haven't found anything yet.  Anyone
> know if this has been reported and/or fixed?
>
> Our scenario is a failure occurs at startup of tests but Surefire doesn't
> fail the build.  Specifically, Spring controller tests aren't failing when
> there is a Spring configuration problem.
>
> Spring top-level thrown exception is:
>   java.lang.IllegalStateException: Failed to load ApplicationContext
>
> but Surefire doesn't report fail.  Surefire output results is:
>   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
>
> Running the tests in an IDE correctly fail for the setup problem.
>

Which IDE?

Did  you try other versions of surefire? Like the latest from 2.x release
line?

Regards
Enrico
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Jeff Jensen-2
Hi Enrico,
Thanks for the reply.

Surefire 2.22.2 correctly fails the build.  This is with Java 8 and JUnit 5.


On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <[hidden email]>
wrote:

> Jeff
>
> Il Sab 13 Giu 2020, 00:15 Jeff Jensen <[hidden email]>
> ha scritto:
>
> > I looked for this issue in JIRA but haven't found anything yet.  Anyone
> > know if this has been reported and/or fixed?
> >
> > Our scenario is a failure occurs at startup of tests but Surefire doesn't
> > fail the build.  Specifically, Spring controller tests aren't failing
> when
> > there is a Spring configuration problem.
> >
> > Spring top-level thrown exception is:
> >   java.lang.IllegalStateException: Failed to load ApplicationContext
> >
> > but Surefire doesn't report fail.  Surefire output results is:
> >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> >
> > Running the tests in an IDE correctly fail for the setup problem.
> >
>
> Which IDE?
>
> Did  you try other versions of surefire? Like the latest from 2.x release
> line?
>
> Regards
> Enrico
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Tibor Digana
In reply to this post by Jeff Jensen
Can you upload the project, dump file and log to the gist?
I am sure it has nothing to do with the spring.
run with:
mvn test -X -e

On Sat, Jun 13, 2020 at 12:15 AM Jeff Jensen <
[hidden email]> wrote:

> I looked for this issue in JIRA but haven't found anything yet.  Anyone
> know if this has been reported and/or fixed?
>
> Our scenario is a failure occurs at startup of tests but Surefire doesn't
> fail the build.  Specifically, Spring controller tests aren't failing when
> there is a Spring configuration problem.
>
> Spring top-level thrown exception is:
>   java.lang.IllegalStateException: Failed to load ApplicationContext
>
> but Surefire doesn't report fail.  Surefire output results is:
>   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
>
> Running the tests in an IDE correctly fail for the setup problem.
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Enrico Olivelli
In reply to this post by Jeff Jensen-2
Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha scritto:

> Hi Enrico,
> Thanks for the reply.
>
> Surefire 2.22.2 correctly fails the build.  This is with Java 8 and JUnit
> 5.
>


This sounds interesting.

It looks like a big bug to me and it is worth a JIRA

It would help a lot if you could attach a simple reproducer for the problem.
If you know surefire codebase it would also help a lot to create the
reproducer as an integration test

Thanks

Enrico


>
> On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <[hidden email]>
> wrote:
>
> > Jeff
> >
> > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <[hidden email]
> >
> > ha scritto:
> >
> > > I looked for this issue in JIRA but haven't found anything yet.  Anyone
> > > know if this has been reported and/or fixed?
> > >
> > > Our scenario is a failure occurs at startup of tests but Surefire
> doesn't
> > > fail the build.  Specifically, Spring controller tests aren't failing
> > when
> > > there is a Spring configuration problem.
> > >
> > > Spring top-level thrown exception is:
> > >   java.lang.IllegalStateException: Failed to load ApplicationContext
> > >
> > > but Surefire doesn't report fail.  Surefire output results is:
> > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > >
> > > Running the tests in an IDE correctly fail for the setup problem.
> > >
> >
> > Which IDE?
> >
> > Did  you try other versions of surefire? Like the latest from 2.x release
> > line?
> >
> > Regards
> > Enrico
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Jeff Jensen-2
Thank you Enrico and Tibor for verifying it is a probable new issue.

I cannot upload this project (customer product/private repo) but will make
a project that reproduces it and reply again.


On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <[hidden email]> wrote:

> Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha scritto:
>
> > Hi Enrico,
> > Thanks for the reply.
> >
> > Surefire 2.22.2 correctly fails the build.  This is with Java 8 and JUnit
> > 5.
> >
>
>
> This sounds interesting.
>
> It looks like a big bug to me and it is worth a JIRA
>
> It would help a lot if you could attach a simple reproducer for the
> problem.
> If you know surefire codebase it would also help a lot to create the
> reproducer as an integration test
>
> Thanks
>
> Enrico
>
>
> >
> > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <[hidden email]>
> > wrote:
> >
> > > Jeff
> > >
> > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> [hidden email]
> > >
> > > ha scritto:
> > >
> > > > I looked for this issue in JIRA but haven't found anything yet.
> Anyone
> > > > know if this has been reported and/or fixed?
> > > >
> > > > Our scenario is a failure occurs at startup of tests but Surefire
> > doesn't
> > > > fail the build.  Specifically, Spring controller tests aren't failing
> > > when
> > > > there is a Spring configuration problem.
> > > >
> > > > Spring top-level thrown exception is:
> > > >   java.lang.IllegalStateException: Failed to load ApplicationContext
> > > >
> > > > but Surefire doesn't report fail.  Surefire output results is:
> > > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > >
> > > > Running the tests in an IDE correctly fail for the setup problem.
> > > >
> > >
> > > Which IDE?
> > >
> > > Did  you try other versions of surefire? Like the latest from 2.x
> release
> > > line?
> > >
> > > Regards
> > > Enrico
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Tibor Digana-3
yes, the reproducible project would be great to have.
Thx

On Sat, Jun 13, 2020 at 4:04 PM Jeff Jensen <[hidden email]> wrote:

> Thank you Enrico and Tibor for verifying it is a probable new issue.
>
> I cannot upload this project (customer product/private repo) but will make
> a project that reproduces it and reply again.
>
>
> On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <[hidden email]>
> wrote:
>
> > Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha scritto:
> >
> > > Hi Enrico,
> > > Thanks for the reply.
> > >
> > > Surefire 2.22.2 correctly fails the build.  This is with Java 8 and
> JUnit
> > > 5.
> > >
> >
> >
> > This sounds interesting.
> >
> > It looks like a big bug to me and it is worth a JIRA
> >
> > It would help a lot if you could attach a simple reproducer for the
> > problem.
> > If you know surefire codebase it would also help a lot to create the
> > reproducer as an integration test
> >
> > Thanks
> >
> > Enrico
> >
> >
> > >
> > > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <[hidden email]>
> > > wrote:
> > >
> > > > Jeff
> > > >
> > > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> > [hidden email]
> > > >
> > > > ha scritto:
> > > >
> > > > > I looked for this issue in JIRA but haven't found anything yet.
> > Anyone
> > > > > know if this has been reported and/or fixed?
> > > > >
> > > > > Our scenario is a failure occurs at startup of tests but Surefire
> > > doesn't
> > > > > fail the build.  Specifically, Spring controller tests aren't
> failing
> > > > when
> > > > > there is a Spring configuration problem.
> > > > >
> > > > > Spring top-level thrown exception is:
> > > > >   java.lang.IllegalStateException: Failed to load
> ApplicationContext
> > > > >
> > > > > but Surefire doesn't report fail.  Surefire output results is:
> > > > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > > >
> > > > > Running the tests in an IDE correctly fail for the setup problem.
> > > > >
> > > >
> > > > Which IDE?
> > > >
> > > > Did  you try other versions of surefire? Like the latest from 2.x
> > release
> > > > line?
> > > >
> > > > Regards
> > > > Enrico
> > > >
> > >
> >
>


--
Cheers
Tibor
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Jeff Jensen-2
This reproduces the issue:
https://github.com/jeffjensen/maven-surefire-3.0.0-m4-no-build-fail

Please let me know if I can help further.


On Sat, Jun 13, 2020 at 9:09 AM Tibor Digaňa
<[hidden email]> wrote:

> yes, the reproducible project would be great to have.
> Thx
>
> On Sat, Jun 13, 2020 at 4:04 PM Jeff Jensen <[hidden email]> wrote:
>
> > Thank you Enrico and Tibor for verifying it is a probable new issue.
> >
> > I cannot upload this project (customer product/private repo) but will
> make
> > a project that reproduces it and reply again.
> >
> >
> > On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <[hidden email]>
> > wrote:
> >
> > > Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha scritto:
> > >
> > > > Hi Enrico,
> > > > Thanks for the reply.
> > > >
> > > > Surefire 2.22.2 correctly fails the build.  This is with Java 8 and
> > JUnit
> > > > 5.
> > > >
> > >
> > >
> > > This sounds interesting.
> > >
> > > It looks like a big bug to me and it is worth a JIRA
> > >
> > > It would help a lot if you could attach a simple reproducer for the
> > > problem.
> > > If you know surefire codebase it would also help a lot to create the
> > > reproducer as an integration test
> > >
> > > Thanks
> > >
> > > Enrico
> > >
> > >
> > > >
> > > > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Jeff
> > > > >
> > > > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> > > [hidden email]
> > > > >
> > > > > ha scritto:
> > > > >
> > > > > > I looked for this issue in JIRA but haven't found anything yet.
> > > Anyone
> > > > > > know if this has been reported and/or fixed?
> > > > > >
> > > > > > Our scenario is a failure occurs at startup of tests but Surefire
> > > > doesn't
> > > > > > fail the build.  Specifically, Spring controller tests aren't
> > failing
> > > > > when
> > > > > > there is a Spring configuration problem.
> > > > > >
> > > > > > Spring top-level thrown exception is:
> > > > > >   java.lang.IllegalStateException: Failed to load
> > ApplicationContext
> > > > > >
> > > > > > but Surefire doesn't report fail.  Surefire output results is:
> > > > > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > > > >
> > > > > > Running the tests in an IDE correctly fail for the setup problem.
> > > > > >
> > > > >
> > > > > Which IDE?
> > > > >
> > > > > Did  you try other versions of surefire? Like the latest from 2.x
> > > release
> > > > > line?
> > > > >
> > > > > Regards
> > > > > Enrico
> > > > >
> > > >
> > >
> >
>
>
> --
> Cheers
> Tibor
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Tibor Digana
We have changed the resolution of junit5 engines 2 times.
Now I have got the result with 3.0.0-M5. I think it is what you expect:

[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @
surefire-no-build-fail ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.jeffjensen.surefire.ApplicationTest
19:43:18.789 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating CacheAwareContextLoaderDelegate from class
[org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
19:43:18.821 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating BootstrapContext using constructor [public
org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
19:43:18.910 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating TestContextBootstrapper for test class
[com.jeffjensen.surefire.ApplicationTest] from class
[org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
19:43:18.944 [main] INFO
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
- Neither @ContextConfiguration nor @ContextHierarchy found for test class
[com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
19:43:18.951 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
19:43:18.952 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
19:43:18.952 [main] INFO
org.springframework.test.context.support.AbstractContextLoader - Could not
detect default resource locations for test class
[com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
{-context.xml, Context.groovy}.
19:43:18.953 [main] INFO
org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
- Could not detect default configuration classes for test class
[com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
any static, non-private, non-final, nested classes annotated with
@Configuration.
19:43:19.024 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating CacheAwareContextLoaderDelegate from class
[org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
19:43:19.024 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating BootstrapContext using constructor [public
org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
19:43:19.025 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating TestContextBootstrapper for test class
[com.jeffjensen.surefire.ApplicationTest] from class
[org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
19:43:19.026 [main] INFO
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
- Neither @ContextConfiguration nor @ContextHierarchy found for test class
[com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
19:43:19.027 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
19:43:19.028 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
19:43:19.029 [main] INFO
org.springframework.test.context.support.AbstractContextLoader - Could not
detect default resource locations for test class
[com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
{-context.xml, Context.groovy}.
19:43:19.029 [main] INFO
org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
- Could not detect default configuration classes for test class
[com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
any static, non-private, non-final, nested classes annotated with
@Configuration.
19:43:19.035 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating CacheAwareContextLoaderDelegate from class
[org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
19:43:19.035 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating BootstrapContext using constructor [public
org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
19:43:19.037 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating TestContextBootstrapper for test class
[com.jeffjensen.surefire.ApplicationTest] from class
[org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
19:43:19.038 [main] INFO
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
- Neither @ContextConfiguration nor @ContextHierarchy found for test class
[com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
19:43:19.040 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
19:43:19.040 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
19:43:19.040 [main] INFO
org.springframework.test.context.support.AbstractContextLoader - Could not
detect default resource locations for test class
[com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
{-context.xml, Context.groovy}.
19:43:19.041 [main] INFO
org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
- Could not detect default configuration classes for test class
[com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
any static, non-private, non-final, nested classes annotated with
@Configuration.
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
0.417 s <<< FAILURE! - in com.jeffjensen.surefire.ApplicationTest
[ERROR] com.jeffjensen.surefire.ApplicationTest  Time elapsed: 0.417 s  <<<
ERROR!
java.lang.NoClassDefFoundError:
org/springframework/web/bind/annotation/ControllerAdvice
Caused by: java.lang.ClassNotFoundException:
org.springframework.web.bind.annotation.ControllerAdvice

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ApplicationTest » NoClassDefFound
org/springframework/web/bind/annotation/Cont...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time:  6.723 s
[INFO] Finished at: 2020-06-13T19:43:19+02:00
[INFO]
------------------------------------------------------------------------



On Sat, Jun 13, 2020 at 6:56 PM Jeff Jensen <[hidden email]> wrote:

> This reproduces the issue:
> https://github.com/jeffjensen/maven-surefire-3.0.0-m4-no-build-fail
>
> Please let me know if I can help further.
>
>
> On Sat, Jun 13, 2020 at 9:09 AM Tibor Digaňa
> <[hidden email]> wrote:
>
> > yes, the reproducible project would be great to have.
> > Thx
> >
> > On Sat, Jun 13, 2020 at 4:04 PM Jeff Jensen <[hidden email]> wrote:
> >
> > > Thank you Enrico and Tibor for verifying it is a probable new issue.
> > >
> > > I cannot upload this project (customer product/private repo) but will
> > make
> > > a project that reproduces it and reply again.
> > >
> > >
> > > On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <[hidden email]>
> > > wrote:
> > >
> > > > Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha
> scritto:
> > > >
> > > > > Hi Enrico,
> > > > > Thanks for the reply.
> > > > >
> > > > > Surefire 2.22.2 correctly fails the build.  This is with Java 8 and
> > > JUnit
> > > > > 5.
> > > > >
> > > >
> > > >
> > > > This sounds interesting.
> > > >
> > > > It looks like a big bug to me and it is worth a JIRA
> > > >
> > > > It would help a lot if you could attach a simple reproducer for the
> > > > problem.
> > > > If you know surefire codebase it would also help a lot to create the
> > > > reproducer as an integration test
> > > >
> > > > Thanks
> > > >
> > > > Enrico
> > > >
> > > >
> > > > >
> > > > > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Jeff
> > > > > >
> > > > > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> > > > [hidden email]
> > > > > >
> > > > > > ha scritto:
> > > > > >
> > > > > > > I looked for this issue in JIRA but haven't found anything yet.
> > > > Anyone
> > > > > > > know if this has been reported and/or fixed?
> > > > > > >
> > > > > > > Our scenario is a failure occurs at startup of tests but
> Surefire
> > > > > doesn't
> > > > > > > fail the build.  Specifically, Spring controller tests aren't
> > > failing
> > > > > > when
> > > > > > > there is a Spring configuration problem.
> > > > > > >
> > > > > > > Spring top-level thrown exception is:
> > > > > > >   java.lang.IllegalStateException: Failed to load
> > > ApplicationContext
> > > > > > >
> > > > > > > but Surefire doesn't report fail.  Surefire output results is:
> > > > > > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > > > > >
> > > > > > > Running the tests in an IDE correctly fail for the setup
> problem.
> > > > > > >
> > > > > >
> > > > > > Which IDE?
> > > > > >
> > > > > > Did  you try other versions of surefire? Like the latest from 2.x
> > > > release
> > > > > > line?
> > > > > >
> > > > > > Regards
> > > > > > Enrico
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Cheers
> > Tibor
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Jeff Jensen-2
Great news that it fails!  Thank you Tibor.

I note it emits a different exception than with 2.22.2 or running in IDE.
2.22.2 emits an IllegalStateException with root
NoSuchBeanDefinitionException but per below M5 emits NoClassDefFoundError.
Is this ok because you changed something locally or is this a concern?


On Sat, Jun 13, 2020 at 12:47 PM Tibor Digana <[hidden email]>
wrote:

> We have changed the resolution of junit5 engines 2 times.
> Now I have got the result with 3.0.0-M5. I think it is what you expect:
>
> [INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @
> surefire-no-build-fail ---
> [INFO]
> [INFO] -------------------------------------------------------
> [INFO]  T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running com.jeffjensen.surefire.ApplicationTest
> 19:43:18.789 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating CacheAwareContextLoaderDelegate from class
>
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> 19:43:18.821 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating BootstrapContext using constructor [public
>
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> 19:43:18.910 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating TestContextBootstrapper for test class
> [com.jeffjensen.surefire.ApplicationTest] from class
>
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> 19:43:18.944 [main] INFO
>
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> - Neither @ContextConfiguration nor @ContextHierarchy found for test class
> [com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
> 19:43:18.951 [main] DEBUG
> org.springframework.test.context.support.AbstractContextLoader - Did not
> detect default resource location for test class
> [com.jeffjensen.surefire.ApplicationTest]: class path resource
> [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> 19:43:18.952 [main] DEBUG
> org.springframework.test.context.support.AbstractContextLoader - Did not
> detect default resource location for test class
> [com.jeffjensen.surefire.ApplicationTest]: class path resource
> [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> 19:43:18.952 [main] INFO
> org.springframework.test.context.support.AbstractContextLoader - Could not
> detect default resource locations for test class
> [com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
> {-context.xml, Context.groovy}.
> 19:43:18.953 [main] INFO
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> - Could not detect default configuration classes for test class
> [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
> any static, non-private, non-final, nested classes annotated with
> @Configuration.
> 19:43:19.024 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating CacheAwareContextLoaderDelegate from class
>
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> 19:43:19.024 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating BootstrapContext using constructor [public
>
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> 19:43:19.025 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating TestContextBootstrapper for test class
> [com.jeffjensen.surefire.ApplicationTest] from class
>
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> 19:43:19.026 [main] INFO
>
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> - Neither @ContextConfiguration nor @ContextHierarchy found for test class
> [com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
> 19:43:19.027 [main] DEBUG
> org.springframework.test.context.support.AbstractContextLoader - Did not
> detect default resource location for test class
> [com.jeffjensen.surefire.ApplicationTest]: class path resource
> [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> 19:43:19.028 [main] DEBUG
> org.springframework.test.context.support.AbstractContextLoader - Did not
> detect default resource location for test class
> [com.jeffjensen.surefire.ApplicationTest]: class path resource
> [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> 19:43:19.029 [main] INFO
> org.springframework.test.context.support.AbstractContextLoader - Could not
> detect default resource locations for test class
> [com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
> {-context.xml, Context.groovy}.
> 19:43:19.029 [main] INFO
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> - Could not detect default configuration classes for test class
> [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
> any static, non-private, non-final, nested classes annotated with
> @Configuration.
> 19:43:19.035 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating CacheAwareContextLoaderDelegate from class
>
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> 19:43:19.035 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating BootstrapContext using constructor [public
>
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> 19:43:19.037 [main] DEBUG org.springframework.test.context.BootstrapUtils -
> Instantiating TestContextBootstrapper for test class
> [com.jeffjensen.surefire.ApplicationTest] from class
>
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> 19:43:19.038 [main] INFO
>
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> - Neither @ContextConfiguration nor @ContextHierarchy found for test class
> [com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
> 19:43:19.040 [main] DEBUG
> org.springframework.test.context.support.AbstractContextLoader - Did not
> detect default resource location for test class
> [com.jeffjensen.surefire.ApplicationTest]: class path resource
> [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> 19:43:19.040 [main] DEBUG
> org.springframework.test.context.support.AbstractContextLoader - Did not
> detect default resource location for test class
> [com.jeffjensen.surefire.ApplicationTest]: class path resource
> [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> 19:43:19.040 [main] INFO
> org.springframework.test.context.support.AbstractContextLoader - Could not
> detect default resource locations for test class
> [com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
> {-context.xml, Context.groovy}.
> 19:43:19.041 [main] INFO
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> - Could not detect default configuration classes for test class
> [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
> any static, non-private, non-final, nested classes annotated with
> @Configuration.
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 0.417 s <<< FAILURE! - in com.jeffjensen.surefire.ApplicationTest
> [ERROR] com.jeffjensen.surefire.ApplicationTest  Time elapsed: 0.417 s  <<<
> ERROR!
> java.lang.NoClassDefFoundError:
> org/springframework/web/bind/annotation/ControllerAdvice
> Caused by: java.lang.ClassNotFoundException:
> org.springframework.web.bind.annotation.ControllerAdvice
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR]   ApplicationTest » NoClassDefFound
> org/springframework/web/bind/annotation/Cont...
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time:  6.723 s
> [INFO] Finished at: 2020-06-13T19:43:19+02:00
> [INFO]
> ------------------------------------------------------------------------
>
>
>
> On Sat, Jun 13, 2020 at 6:56 PM Jeff Jensen <[hidden email]> wrote:
>
> > This reproduces the issue:
> > https://github.com/jeffjensen/maven-surefire-3.0.0-m4-no-build-fail
> >
> > Please let me know if I can help further.
> >
> >
> > On Sat, Jun 13, 2020 at 9:09 AM Tibor Digaňa
> > <[hidden email]> wrote:
> >
> > > yes, the reproducible project would be great to have.
> > > Thx
> > >
> > > On Sat, Jun 13, 2020 at 4:04 PM Jeff Jensen <[hidden email]>
> wrote:
> > >
> > > > Thank you Enrico and Tibor for verifying it is a probable new issue.
> > > >
> > > > I cannot upload this project (customer product/private repo) but will
> > > make
> > > > a project that reproduces it and reply again.
> > > >
> > > >
> > > > On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha
> > scritto:
> > > > >
> > > > > > Hi Enrico,
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > Surefire 2.22.2 correctly fails the build.  This is with Java 8
> and
> > > > JUnit
> > > > > > 5.
> > > > > >
> > > > >
> > > > >
> > > > > This sounds interesting.
> > > > >
> > > > > It looks like a big bug to me and it is worth a JIRA
> > > > >
> > > > > It would help a lot if you could attach a simple reproducer for the
> > > > > problem.
> > > > > If you know surefire codebase it would also help a lot to create
> the
> > > > > reproducer as an integration test
> > > > >
> > > > > Thanks
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > > >
> > > > > > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Jeff
> > > > > > >
> > > > > > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> > > > > [hidden email]
> > > > > > >
> > > > > > > ha scritto:
> > > > > > >
> > > > > > > > I looked for this issue in JIRA but haven't found anything
> yet.
> > > > > Anyone
> > > > > > > > know if this has been reported and/or fixed?
> > > > > > > >
> > > > > > > > Our scenario is a failure occurs at startup of tests but
> > Surefire
> > > > > > doesn't
> > > > > > > > fail the build.  Specifically, Spring controller tests aren't
> > > > failing
> > > > > > > when
> > > > > > > > there is a Spring configuration problem.
> > > > > > > >
> > > > > > > > Spring top-level thrown exception is:
> > > > > > > >   java.lang.IllegalStateException: Failed to load
> > > > ApplicationContext
> > > > > > > >
> > > > > > > > but Surefire doesn't report fail.  Surefire output results
> is:
> > > > > > > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > > > > > >
> > > > > > > > Running the tests in an IDE correctly fail for the setup
> > problem.
> > > > > > > >
> > > > > > >
> > > > > > > Which IDE?
> > > > > > >
> > > > > > > Did  you try other versions of surefire? Like the latest from
> 2.x
> > > > > release
> > > > > > > line?
> > > > > > >
> > > > > > > Regards
> > > > > > > Enrico
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Cheers
> > > Tibor
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Tibor Digana
Now we are able to handle the errors on the class level.
Before in 2.22.x and in M4 we handled the test errors only and ignored the
class level errors.

I would trust NoClassDefFoundError because ControllerAdvice really could
not be found in IntelliJ IDEA.
So the IDE is telling me the same thing, and this is perhaps the root cause
for NoSuchBeanDefinitionException. You know your application better than
me.

Even if i use the IntelliJ IDEA and run the tests, i get the same error on
the console:

Test ignored.

*java.lang.NoClassDefFoundError:
org/springframework/web/bind/annotation/ControllerAdvice*
at
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTypeExcludeFilter.<clinit>(WebMvcTypeExcludeFilter.java:59)
...
Suppressed: java.lang.NoClassDefFoundError: Could not initialize class
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTypeExcludeFilter
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
...
Caused by: java.lang.ClassNotFoundException:
org.springframework.web.bind.annotation.ControllerAdvice
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:604)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 61 more








On Sat, Jun 13, 2020 at 10:51 PM Jeff Jensen <[hidden email]> wrote:

> Great news that it fails!  Thank you Tibor.
>
> I note it emits a different exception than with 2.22.2 or running in IDE.
> 2.22.2 emits an IllegalStateException with root
> NoSuchBeanDefinitionException but per below M5 emits NoClassDefFoundError.
> Is this ok because you changed something locally or is this a concern?
>
>
> On Sat, Jun 13, 2020 at 12:47 PM Tibor Digana <[hidden email]>
> wrote:
>
> > We have changed the resolution of junit5 engines 2 times.
> > Now I have got the result with 3.0.0-M5. I think it is what you expect:
> >
> > [INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @
> > surefire-no-build-fail ---
> > [INFO]
> > [INFO] -------------------------------------------------------
> > [INFO]  T E S T S
> > [INFO] -------------------------------------------------------
> > [INFO] Running com.jeffjensen.surefire.ApplicationTest
> > 19:43:18.789 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating CacheAwareContextLoaderDelegate from class
> >
> >
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> > 19:43:18.821 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating BootstrapContext using constructor [public
> >
> >
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> > 19:43:18.910 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating TestContextBootstrapper for test class
> > [com.jeffjensen.surefire.ApplicationTest] from class
> >
> >
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> > 19:43:18.944 [main] INFO
> >
> >
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> > - Neither @ContextConfiguration nor @ContextHierarchy found for test
> class
> > [com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
> > 19:43:18.951 [main] DEBUG
> > org.springframework.test.context.support.AbstractContextLoader - Did not
> > detect default resource location for test class
> > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> > 19:43:18.952 [main] DEBUG
> > org.springframework.test.context.support.AbstractContextLoader - Did not
> > detect default resource location for test class
> > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> > 19:43:18.952 [main] INFO
> > org.springframework.test.context.support.AbstractContextLoader - Could
> not
> > detect default resource locations for test class
> > [com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
> > {-context.xml, Context.groovy}.
> > 19:43:18.953 [main] INFO
> >
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> > - Could not detect default configuration classes for test class
> > [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not
> declare
> > any static, non-private, non-final, nested classes annotated with
> > @Configuration.
> > 19:43:19.024 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating CacheAwareContextLoaderDelegate from class
> >
> >
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> > 19:43:19.024 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating BootstrapContext using constructor [public
> >
> >
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> > 19:43:19.025 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating TestContextBootstrapper for test class
> > [com.jeffjensen.surefire.ApplicationTest] from class
> >
> >
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> > 19:43:19.026 [main] INFO
> >
> >
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> > - Neither @ContextConfiguration nor @ContextHierarchy found for test
> class
> > [com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
> > 19:43:19.027 [main] DEBUG
> > org.springframework.test.context.support.AbstractContextLoader - Did not
> > detect default resource location for test class
> > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> > 19:43:19.028 [main] DEBUG
> > org.springframework.test.context.support.AbstractContextLoader - Did not
> > detect default resource location for test class
> > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> > 19:43:19.029 [main] INFO
> > org.springframework.test.context.support.AbstractContextLoader - Could
> not
> > detect default resource locations for test class
> > [com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
> > {-context.xml, Context.groovy}.
> > 19:43:19.029 [main] INFO
> >
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> > - Could not detect default configuration classes for test class
> > [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not
> declare
> > any static, non-private, non-final, nested classes annotated with
> > @Configuration.
> > 19:43:19.035 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating CacheAwareContextLoaderDelegate from class
> >
> >
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> > 19:43:19.035 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating BootstrapContext using constructor [public
> >
> >
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> > 19:43:19.037 [main] DEBUG
> org.springframework.test.context.BootstrapUtils -
> > Instantiating TestContextBootstrapper for test class
> > [com.jeffjensen.surefire.ApplicationTest] from class
> >
> >
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> > 19:43:19.038 [main] INFO
> >
> >
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> > - Neither @ContextConfiguration nor @ContextHierarchy found for test
> class
> > [com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
> > 19:43:19.040 [main] DEBUG
> > org.springframework.test.context.support.AbstractContextLoader - Did not
> > detect default resource location for test class
> > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> > 19:43:19.040 [main] DEBUG
> > org.springframework.test.context.support.AbstractContextLoader - Did not
> > detect default resource location for test class
> > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> > 19:43:19.040 [main] INFO
> > org.springframework.test.context.support.AbstractContextLoader - Could
> not
> > detect default resource locations for test class
> > [com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
> > {-context.xml, Context.groovy}.
> > 19:43:19.041 [main] INFO
> >
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> > - Could not detect default configuration classes for test class
> > [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not
> declare
> > any static, non-private, non-final, nested classes annotated with
> > @Configuration.
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 0.417 s <<< FAILURE! - in com.jeffjensen.surefire.ApplicationTest
> > [ERROR] com.jeffjensen.surefire.ApplicationTest  Time elapsed: 0.417 s
> <<<
> > ERROR!
> > java.lang.NoClassDefFoundError:
> > org/springframework/web/bind/annotation/ControllerAdvice
> > Caused by: java.lang.ClassNotFoundException:
> > org.springframework.web.bind.annotation.ControllerAdvice
> >
> > [INFO]
> > [INFO] Results:
> > [INFO]
> > [ERROR] Errors:
> > [ERROR]   ApplicationTest » NoClassDefFound
> > org/springframework/web/bind/annotation/Cont...
> > [INFO]
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> > [INFO]
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD FAILURE
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Total time:  6.723 s
> > [INFO] Finished at: 2020-06-13T19:43:19+02:00
> > [INFO]
> > ------------------------------------------------------------------------
> >
> >
> >
> > On Sat, Jun 13, 2020 at 6:56 PM Jeff Jensen <[hidden email]> wrote:
> >
> > > This reproduces the issue:
> > > https://github.com/jeffjensen/maven-surefire-3.0.0-m4-no-build-fail
> > >
> > > Please let me know if I can help further.
> > >
> > >
> > > On Sat, Jun 13, 2020 at 9:09 AM Tibor Digaňa
> > > <[hidden email]> wrote:
> > >
> > > > yes, the reproducible project would be great to have.
> > > > Thx
> > > >
> > > > On Sat, Jun 13, 2020 at 4:04 PM Jeff Jensen <[hidden email]>
> > wrote:
> > > >
> > > > > Thank you Enrico and Tibor for verifying it is a probable new
> issue.
> > > > >
> > > > > I cannot upload this project (customer product/private repo) but
> will
> > > > make
> > > > > a project that reproduces it and reply again.
> > > > >
> > > > >
> > > > > On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha
> > > scritto:
> > > > > >
> > > > > > > Hi Enrico,
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > Surefire 2.22.2 correctly fails the build.  This is with Java 8
> > and
> > > > > JUnit
> > > > > > > 5.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > This sounds interesting.
> > > > > >
> > > > > > It looks like a big bug to me and it is worth a JIRA
> > > > > >
> > > > > > It would help a lot if you could attach a simple reproducer for
> the
> > > > > > problem.
> > > > > > If you know surefire codebase it would also help a lot to create
> > the
> > > > > > reproducer as an integration test
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Jeff
> > > > > > > >
> > > > > > > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > ha scritto:
> > > > > > > >
> > > > > > > > > I looked for this issue in JIRA but haven't found anything
> > yet.
> > > > > > Anyone
> > > > > > > > > know if this has been reported and/or fixed?
> > > > > > > > >
> > > > > > > > > Our scenario is a failure occurs at startup of tests but
> > > Surefire
> > > > > > > doesn't
> > > > > > > > > fail the build.  Specifically, Spring controller tests
> aren't
> > > > > failing
> > > > > > > > when
> > > > > > > > > there is a Spring configuration problem.
> > > > > > > > >
> > > > > > > > > Spring top-level thrown exception is:
> > > > > > > > >   java.lang.IllegalStateException: Failed to load
> > > > > ApplicationContext
> > > > > > > > >
> > > > > > > > > but Surefire doesn't report fail.  Surefire output results
> > is:
> > > > > > > > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > > > > > > >
> > > > > > > > > Running the tests in an IDE correctly fail for the setup
> > > problem.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Which IDE?
> > > > > > > >
> > > > > > > > Did  you try other versions of surefire? Like the latest from
> > 2.x
> > > > > > release
> > > > > > > > line?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Enrico
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Tibor
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Surefire 3.0.0-M4 not failing build on errors

Jeff Jensen-2
Thank you for explaining.  Yes, turns out a dependency is missing in my
reproducer and looks like M5 correctly detects that error; that's great
news.

Interestingly, if I correct that by adding the dep, M4 correctly works with
this reproducer too, so it's not a reproducer of the intended error!  That
is not the problem in the customer app though - both 2.22.2 and IDEA detect
the NoSuchBeanDefinitionException in it.

Thanks again for the help; looking forward to M5 release.


On Sun, Jun 14, 2020 at 3:55 AM Tibor Digana <[hidden email]> wrote:

> Now we are able to handle the errors on the class level.
> Before in 2.22.x and in M4 we handled the test errors only and ignored the
> class level errors.
>
> I would trust NoClassDefFoundError because ControllerAdvice really could
> not be found in IntelliJ IDEA.
> So the IDE is telling me the same thing, and this is perhaps the root cause
> for NoSuchBeanDefinitionException. You know your application better than
> me.
>
> Even if i use the IntelliJ IDEA and run the tests, i get the same error on
> the console:
>
> Test ignored.
>
> *java.lang.NoClassDefFoundError:
> org/springframework/web/bind/annotation/ControllerAdvice*
> at
>
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTypeExcludeFilter.<clinit>(WebMvcTypeExcludeFilter.java:59)
> ...
> Suppressed: java.lang.NoClassDefFoundError: Could not initialize class
>
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTypeExcludeFilter
> at
>
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
>
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.springframework.web.bind.annotation.ControllerAdvice
> at
>
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:604)
> at
>
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> ... 61 more
>
>
>
>
>
>
>
>
> On Sat, Jun 13, 2020 at 10:51 PM Jeff Jensen <[hidden email]> wrote:
>
> > Great news that it fails!  Thank you Tibor.
> >
> > I note it emits a different exception than with 2.22.2 or running in IDE.
> > 2.22.2 emits an IllegalStateException with root
> > NoSuchBeanDefinitionException but per below M5 emits
> NoClassDefFoundError.
> > Is this ok because you changed something locally or is this a concern?
> >
> >
> > On Sat, Jun 13, 2020 at 12:47 PM Tibor Digana <[hidden email]>
> > wrote:
> >
> > > We have changed the resolution of junit5 engines 2 times.
> > > Now I have got the result with 3.0.0-M5. I think it is what you expect:
> > >
> > > [INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @
> > > surefire-no-build-fail ---
> > > [INFO]
> > > [INFO] -------------------------------------------------------
> > > [INFO]  T E S T S
> > > [INFO] -------------------------------------------------------
> > > [INFO] Running com.jeffjensen.surefire.ApplicationTest
> > > 19:43:18.789 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating CacheAwareContextLoaderDelegate from class
> > >
> > >
> >
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> > > 19:43:18.821 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating BootstrapContext using constructor [public
> > >
> > >
> >
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> > > 19:43:18.910 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating TestContextBootstrapper for test class
> > > [com.jeffjensen.surefire.ApplicationTest] from class
> > >
> > >
> >
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> > > 19:43:18.944 [main] INFO
> > >
> > >
> >
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> > > - Neither @ContextConfiguration nor @ContextHierarchy found for test
> > class
> > > [com.jeffjensen.surefire.ApplicationTest], using
> SpringBootContextLoader
> > > 19:43:18.951 [main] DEBUG
> > > org.springframework.test.context.support.AbstractContextLoader - Did
> not
> > > detect default resource location for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > > [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> > > 19:43:18.952 [main] DEBUG
> > > org.springframework.test.context.support.AbstractContextLoader - Did
> not
> > > detect default resource location for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > > [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> > > 19:43:18.952 [main] INFO
> > > org.springframework.test.context.support.AbstractContextLoader - Could
> > not
> > > detect default resource locations for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: no resource found for
> suffixes
> > > {-context.xml, Context.groovy}.
> > > 19:43:18.953 [main] INFO
> > >
> >
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> > > - Could not detect default configuration classes for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not
> > declare
> > > any static, non-private, non-final, nested classes annotated with
> > > @Configuration.
> > > 19:43:19.024 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating CacheAwareContextLoaderDelegate from class
> > >
> > >
> >
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> > > 19:43:19.024 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating BootstrapContext using constructor [public
> > >
> > >
> >
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> > > 19:43:19.025 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating TestContextBootstrapper for test class
> > > [com.jeffjensen.surefire.ApplicationTest] from class
> > >
> > >
> >
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> > > 19:43:19.026 [main] INFO
> > >
> > >
> >
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> > > - Neither @ContextConfiguration nor @ContextHierarchy found for test
> > class
> > > [com.jeffjensen.surefire.ApplicationTest], using
> SpringBootContextLoader
> > > 19:43:19.027 [main] DEBUG
> > > org.springframework.test.context.support.AbstractContextLoader - Did
> not
> > > detect default resource location for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > > [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> > > 19:43:19.028 [main] DEBUG
> > > org.springframework.test.context.support.AbstractContextLoader - Did
> not
> > > detect default resource location for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > > [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> > > 19:43:19.029 [main] INFO
> > > org.springframework.test.context.support.AbstractContextLoader - Could
> > not
> > > detect default resource locations for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: no resource found for
> suffixes
> > > {-context.xml, Context.groovy}.
> > > 19:43:19.029 [main] INFO
> > >
> >
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> > > - Could not detect default configuration classes for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not
> > declare
> > > any static, non-private, non-final, nested classes annotated with
> > > @Configuration.
> > > 19:43:19.035 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating CacheAwareContextLoaderDelegate from class
> > >
> > >
> >
> [org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
> > > 19:43:19.035 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating BootstrapContext using constructor [public
> > >
> > >
> >
> org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
> > > 19:43:19.037 [main] DEBUG
> > org.springframework.test.context.BootstrapUtils -
> > > Instantiating TestContextBootstrapper for test class
> > > [com.jeffjensen.surefire.ApplicationTest] from class
> > >
> > >
> >
> [org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
> > > 19:43:19.038 [main] INFO
> > >
> > >
> >
> org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
> > > - Neither @ContextConfiguration nor @ContextHierarchy found for test
> > class
> > > [com.jeffjensen.surefire.ApplicationTest], using
> SpringBootContextLoader
> > > 19:43:19.040 [main] DEBUG
> > > org.springframework.test.context.support.AbstractContextLoader - Did
> not
> > > detect default resource location for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > > [com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
> > > 19:43:19.040 [main] DEBUG
> > > org.springframework.test.context.support.AbstractContextLoader - Did
> not
> > > detect default resource location for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: class path resource
> > > [com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
> > > 19:43:19.040 [main] INFO
> > > org.springframework.test.context.support.AbstractContextLoader - Could
> > not
> > > detect default resource locations for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: no resource found for
> suffixes
> > > {-context.xml, Context.groovy}.
> > > 19:43:19.041 [main] INFO
> > >
> >
> org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
> > > - Could not detect default configuration classes for test class
> > > [com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not
> > declare
> > > any static, non-private, non-final, nested classes annotated with
> > > @Configuration.
> > > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > > 0.417 s <<< FAILURE! - in com.jeffjensen.surefire.ApplicationTest
> > > [ERROR] com.jeffjensen.surefire.ApplicationTest  Time elapsed: 0.417 s
> > <<<
> > > ERROR!
> > > java.lang.NoClassDefFoundError:
> > > org/springframework/web/bind/annotation/ControllerAdvice
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.springframework.web.bind.annotation.ControllerAdvice
> > >
> > > [INFO]
> > > [INFO] Results:
> > > [INFO]
> > > [ERROR] Errors:
> > > [ERROR]   ApplicationTest » NoClassDefFound
> > > org/springframework/web/bind/annotation/Cont...
> > > [INFO]
> > > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> > > [INFO]
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] BUILD FAILURE
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > > [INFO] Total time:  6.723 s
> > > [INFO] Finished at: 2020-06-13T19:43:19+02:00
> > > [INFO]
> > >
> ------------------------------------------------------------------------
> > >
> > >
> > >
> > > On Sat, Jun 13, 2020 at 6:56 PM Jeff Jensen <[hidden email]>
> wrote:
> > >
> > > > This reproduces the issue:
> > > > https://github.com/jeffjensen/maven-surefire-3.0.0-m4-no-build-fail
> > > >
> > > > Please let me know if I can help further.
> > > >
> > > >
> > > > On Sat, Jun 13, 2020 at 9:09 AM Tibor Digaňa
> > > > <[hidden email]> wrote:
> > > >
> > > > > yes, the reproducible project would be great to have.
> > > > > Thx
> > > > >
> > > > > On Sat, Jun 13, 2020 at 4:04 PM Jeff Jensen <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Thank you Enrico and Tibor for verifying it is a probable new
> > issue.
> > > > > >
> > > > > > I cannot upload this project (customer product/private repo) but
> > will
> > > > > make
> > > > > > a project that reproduces it and reply again.
> > > > > >
> > > > > >
> > > > > > On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Il Sab 13 Giu 2020, 15:07 Jeff Jensen <[hidden email]> ha
> > > > scritto:
> > > > > > >
> > > > > > > > Hi Enrico,
> > > > > > > > Thanks for the reply.
> > > > > > > >
> > > > > > > > Surefire 2.22.2 correctly fails the build.  This is with
> Java 8
> > > and
> > > > > > JUnit
> > > > > > > > 5.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This sounds interesting.
> > > > > > >
> > > > > > > It looks like a big bug to me and it is worth a JIRA
> > > > > > >
> > > > > > > It would help a lot if you could attach a simple reproducer for
> > the
> > > > > > > problem.
> > > > > > > If you know surefire codebase it would also help a lot to
> create
> > > the
> > > > > > > reproducer as an integration test
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <
> > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Jeff
> > > > > > > > >
> > > > > > > > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > ha scritto:
> > > > > > > > >
> > > > > > > > > > I looked for this issue in JIRA but haven't found
> anything
> > > yet.
> > > > > > > Anyone
> > > > > > > > > > know if this has been reported and/or fixed?
> > > > > > > > > >
> > > > > > > > > > Our scenario is a failure occurs at startup of tests but
> > > > Surefire
> > > > > > > > doesn't
> > > > > > > > > > fail the build.  Specifically, Spring controller tests
> > aren't
> > > > > > failing
> > > > > > > > > when
> > > > > > > > > > there is a Spring configuration problem.
> > > > > > > > > >
> > > > > > > > > > Spring top-level thrown exception is:
> > > > > > > > > >   java.lang.IllegalStateException: Failed to load
> > > > > > ApplicationContext
> > > > > > > > > >
> > > > > > > > > > but Surefire doesn't report fail.  Surefire output
> results
> > > is:
> > > > > > > > > >   Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > > > > > > > >
> > > > > > > > > > Running the tests in an IDE correctly fail for the setup
> > > > problem.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Which IDE?
> > > > > > > > >
> > > > > > > > > Did  you try other versions of surefire? Like the latest
> from
> > > 2.x
> > > > > > > release
> > > > > > > > > line?
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Enrico
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers
> > > > > Tibor
> > > > >
> > > >
> > >
> >
>