Surefire - Forking - Identifying Tests that run in a particular fork

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Surefire - Forking - Identifying Tests that run in a particular fork

Martin Lambert
Hello,

 

I'm currently using the Surefire plugin with Maven 3.6.3 and would like to
know if there is any way to identify what tests were run in a given fork? I
haven't seen any option for it in the current documentation. I'm running
around ~10K test cases hence using the fork option to speed things up. The
problem I'm trying to solve is that a single test intermittently fails,
which, when run individually succeeds; I suspect this is due to a prior test
case not cleaning up properly and would therefore like to be able to
identify at a minimum what test cases are in a fork group so that I can take
a divide and conquer approach to resolving the issue, even better would be
to know which test cases ran before the failing test case.

 

Thanks for your help,

 

Martin.

 

Reply | Threaded
Open this post in threaded view
|

Re: Surefire - Forking - Identifying Tests that run in a particular fork

Falko Modler
Hi Martin,

OOTB you'll probably have to resort to -X (which logs way too much) and
even then I am not 100% sure you'll get what you need.

Did you have a look at |${surefire.forkNumber}| (see
https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html)?

I've been using this as a system property in my projects to:

- write one logfile per fork (e.g. via Logback)
- prefix each logged line with the fork number (also via Logback,
especially useful for Jenins console log)
- log each test method in a JUnit4 listener together with the log file
(certainly also possible with JUnit5 etc.)
- separate working directories, generate tracing ids etc.

I can really recommend to take these extra measures, otherwise analyzing
test failures with multiple forks and/or parallel modules (-T) will be a
real pain in the long run.

Cheers,

Falko

Am 02.07.2020 um 00:12 schrieb Martin Lambert:

> Hello,
>
>
>
> I'm currently using the Surefire plugin with Maven 3.6.3 and would like to
> know if there is any way to identify what tests were run in a given fork? I
> haven't seen any option for it in the current documentation. I'm running
> around ~10K test cases hence using the fork option to speed things up. The
> problem I'm trying to solve is that a single test intermittently fails,
> which, when run individually succeeds; I suspect this is due to a prior test
> case not cleaning up properly and would therefore like to be able to
> identify at a minimum what test cases are in a fork group so that I can take
> a divide and conquer approach to resolving the issue, even better would be
> to know which test cases ran before the failing test case.
>
>
>
> Thanks for your help,
>
>
>
> Martin.
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

RE: Surefire - Forking - Identifying Tests that run in a particular fork

Martin Lambert
Hi Falko,

Thanks for the suggestion. In the end I created a simple Junit TestExecutionListener class that overrode executionStarted() and executionFinished(); both methods have a TestIdentifier parameter, if this is actually an instance of MethodSource (https://junit.org/junit5/docs/current/api/org.junit.platform.engine/org/junit/platform/engine/support/descriptor/MethodSource.html) you can downcast it, this allows you can get hold of the test class' name using getClassName() on the same object; somewhere I was able to call an isTest() method to filter out noise - I can't remember where off the top of my head. To make the whole thing work you have to create a ServiceLoader file as mentioned in 6.1.4 here: https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom (it really does just involve creating the specific file in the location mentioned, containing only the fully qualified name of your TestExecutionListener class 😊 ).

Cheers,

Martin.

-----Original Message-----
From: Falko Modler <[hidden email]>
Sent: 01 July 2020 23:48
To: [hidden email]
Subject: Re: Surefire - Forking - Identifying Tests that run in a particular fork

Hi Martin,

OOTB you'll probably have to resort to -X (which logs way too much) and even then I am not 100% sure you'll get what you need.

Did you have a look at |${surefire.forkNumber}| (see https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html)?

I've been using this as a system property in my projects to:

- write one logfile per fork (e.g. via Logback)
- prefix each logged line with the fork number (also via Logback, especially useful for Jenins console log)
- log each test method in a JUnit4 listener together with the log file (certainly also possible with JUnit5 etc.)
- separate working directories, generate tracing ids etc.

I can really recommend to take these extra measures, otherwise analyzing test failures with multiple forks and/or parallel modules (-T) will be a real pain in the long run.

Cheers,

Falko

Am 02.07.2020 um 00:12 schrieb Martin Lambert:

> Hello,
>
>
>
> I'm currently using the Surefire plugin with Maven 3.6.3 and would
> like to know if there is any way to identify what tests were run in a
> given fork? I haven't seen any option for it in the current
> documentation. I'm running around ~10K test cases hence using the fork
> option to speed things up. The problem I'm trying to solve is that a
> single test intermittently fails, which, when run individually
> succeeds; I suspect this is due to a prior test case not cleaning up
> properly and would therefore like to be able to identify at a minimum
> what test cases are in a fork group so that I can take a divide and
> conquer approach to resolving the issue, even better would be to know which test cases ran before the failing test case.
>
>
>
> Thanks for your help,
>
>
>
> Martin.
>
>
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Surefire - Forking - Identifying Tests that run in a particular fork

Tibor Digana
Hi Martin,

Can you be more concrete with your business expectations?
What logger, what kind of notations about forked JVM id you want to express
in the logs or somewhere else?

Maybe you are not aware of new features in Surefire and Failsafe, but we
introduced Extensions API in 3.0.0-M4.
This way you have a chance to implement the Extensions API as you like.
It means that the features of loggers and wrapped configuration parameters
can be customized by adding new ones.
See the documentation
https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName
It is maybe easier for the user due to now it is quite simple for the
contributor to concentrate on the abstraction which easily leads you to the
(default) implementation.

The extensions concerns anything, not only junit5.

Cheers
T

On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert
<[hidden email]> wrote:

> Hi Falko,
>
> Thanks for the suggestion. In the end I created a simple Junit
> TestExecutionListener class that overrode executionStarted() and
> executionFinished(); both methods have a TestIdentifier parameter, if this
> is actually an instance of MethodSource (
> https://junit.org/junit5/docs/current/api/org.junit.platform.engine/org/junit/platform/engine/support/descriptor/MethodSource.html)
> you can downcast it, this allows you can get hold of the test class' name
> using getClassName() on the same object; somewhere I was able to call an
> isTest() method to filter out noise - I can't remember where off the top of
> my head. To make the whole thing work you have to create a ServiceLoader
> file as mentioned in 6.1.4 here:
> https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom
> (it really does just involve creating the specific file in the location
> mentioned, containing only the fully qualified name of your
> TestExecutionListener class 😊 ).
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Falko Modler <[hidden email]>
> Sent: 01 July 2020 23:48
> To: [hidden email]
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> OOTB you'll probably have to resort to -X (which logs way too much) and
> even then I am not 100% sure you'll get what you need.
>
> Did you have a look at |${surefire.forkNumber}| (see
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
> )?
>
> I've been using this as a system property in my projects to:
>
> - write one logfile per fork (e.g. via Logback)
> - prefix each logged line with the fork number (also via Logback,
> especially useful for Jenins console log)
> - log each test method in a JUnit4 listener together with the log file
> (certainly also possible with JUnit5 etc.)
> - separate working directories, generate tracing ids etc.
>
> I can really recommend to take these extra measures, otherwise analyzing
> test failures with multiple forks and/or parallel modules (-T) will be a
> real pain in the long run.
>
> Cheers,
>
> Falko
>
> Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > Hello,
> >
> >
> >
> > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > like to know if there is any way to identify what tests were run in a
> > given fork? I haven't seen any option for it in the current
> > documentation. I'm running around ~10K test cases hence using the fork
> > option to speed things up. The problem I'm trying to solve is that a
> > single test intermittently fails, which, when run individually
> > succeeds; I suspect this is due to a prior test case not cleaning up
> > properly and would therefore like to be able to identify at a minimum
> > what test cases are in a fork group so that I can take a divide and
> > conquer approach to resolving the issue, even better would be to know
> which test cases ran before the failing test case.
> >
> >
> >
> > Thanks for your help,
> >
> >
> >
> > Martin.
> >
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Surefire - Forking - Identifying Tests that run in a particular fork

Martin Lambert
Hi Tibor,

Sorry it's taken a little while to get back to you.

To set the scene, as basic requirements when running n tests serially I'd like to know...

* When all pass: I'd mainly be interested in the fact that all tests passed, what tests ran and if any were skipped
* If one or more tests failed: it seems reasonable to additionally be told which test(s) failed
* If one or more tests failed and re-running a failed test individually did not reproduce a failure: it would be exceedingly helpful to also be able to see what tests ran before a failing test, including their output, in order to identify if prior tests were not cleaning up after themselves

The reason for posting my original question was that when Surefire is configured to use forking, e.g. <forkCount>8</count>, then as far as I've been able to find out only the first 2 points above hold. To help visualise this, you could imagine getting console output something like:

Running org.apache.maven.MyTest1
Running org.apache.maven.MyTest32
Running org.apache.maven.MyTest4
Running org.apache.maven.MyTest77
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
...
Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2 skipped: 0
~~~~~~ Output from test unknown
~~~~~~ Output from test unknown
Running org.apache.maven.MyTest2
...

What I'm looking for is to be able to untangle this obfuscated log i.e. only look at the output for a single fork, and if necessary thread, thereby getting back to a similar place as running a set of tests serially. As an example, given the output below one could easily filter the output based on "[Fork 2]":

[Fork 1] Running org.apache.maven.MyTest1
[Fork 2] Running org.apache.maven.MyTest32
[Fork 3] Running org.apache.maven.MyTest4
[Fork 4] Running org.apache.maven.MyTest77
[Fork 2] ~~~~~~ Output from test unknown
[Fork 2] ~~~~~~ Output from test unknown
[Fork 1] ~~~~~~ Output from test unknown
[Fork 3] ~~~~~~ Output from test unknown
[Fork 2] ~~~~~~ Output from test unknown
...
[Fork 3]  Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2 skipped: 0
[Fork 4] ~~~~~~ Output from test unknown
[Fork 4] ~~~~~~ Output from test unknown
[Fork 3]  Running org.apache.maven.MyTest2
...

My current work around achieves something similar to the last example and uses:
* The JUnit TestExecutionListener mechanism to supplement the existing JUnit / Surefire log lines with test start and end messages that include a given fork JVM's process id, this could easily be expanded to include the thread if needed
* Changed my test logging code to also include a given fork JVM's process id; again thread could be included if needed

I did look at the examples page, however, on reflection it seems that if the fork functionality doesn't provide a mechanism for this out of the box then it would be highly desirable for it to do so based on the reasons outlined; of course, if the fork mechanism already supports doing this then it would be great to hear how to configure Surefire in order to get this behaviour.

Cheers,

Martin.

-----Original Message-----
From: Tibor Digana <[hidden email]>
Sent: 05 July 2020 11:59
To: [hidden email]
Cc: Maven Users List <[hidden email]>
Subject: Re: Surefire - Forking - Identifying Tests that run in a particular fork

Hi Martin,

Can you be more concrete with your business expectations?
What logger, what kind of notations about forked JVM id you want to express in the logs or somewhere else?

Maybe you are not aware of new features in Surefire and Failsafe, but we introduced Extensions API in 3.0.0-M4.
This way you have a chance to implement the Extensions API as you like.
It means that the features of loggers and wrapped configuration parameters can be customized by adding new ones.
See the documentation
https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName
It is maybe easier for the user due to now it is quite simple for the contributor to concentrate on the abstraction which easily leads you to the
(default) implementation.

The extensions concerns anything, not only junit5.

Cheers
T

On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert <[hidden email]> wrote:

> Hi Falko,
>
> Thanks for the suggestion. In the end I created a simple Junit
> TestExecutionListener class that overrode executionStarted() and
> executionFinished(); both methods have a TestIdentifier parameter, if
> this is actually an instance of MethodSource (
> https://junit.org/junit5/docs/current/api/org.junit.platform.engine/or
> g/junit/platform/engine/support/descriptor/MethodSource.html)
> you can downcast it, this allows you can get hold of the test class'
> name using getClassName() on the same object; somewhere I was able to
> call an
> isTest() method to filter out noise - I can't remember where off the
> top of my head. To make the whole thing work you have to create a
> ServiceLoader file as mentioned in 6.1.4 here:
> https://junit.org/junit5/docs/current/user-guide/#launcher-api-listene
> rs-custom (it really does just involve creating the specific file in
> the location mentioned, containing only the fully qualified name of
> your TestExecutionListener class 😊 ).
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Falko Modler <[hidden email]>
> Sent: 01 July 2020 23:48
> To: [hidden email]
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> OOTB you'll probably have to resort to -X (which logs way too much)
> and even then I am not 100% sure you'll get what you need.
>
> Did you have a look at |${surefire.forkNumber}| (see
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-
> options-and-parallel-execution.html
> )?
>
> I've been using this as a system property in my projects to:
>
> - write one logfile per fork (e.g. via Logback)
> - prefix each logged line with the fork number (also via Logback,
> especially useful for Jenins console log)
> - log each test method in a JUnit4 listener together with the log file
> (certainly also possible with JUnit5 etc.)
> - separate working directories, generate tracing ids etc.
>
> I can really recommend to take these extra measures, otherwise
> analyzing test failures with multiple forks and/or parallel modules
> (-T) will be a real pain in the long run.
>
> Cheers,
>
> Falko
>
> Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > Hello,
> >
> >
> >
> > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > like to know if there is any way to identify what tests were run in
> > a given fork? I haven't seen any option for it in the current
> > documentation. I'm running around ~10K test cases hence using the
> > fork option to speed things up. The problem I'm trying to solve is
> > that a single test intermittently fails, which, when run
> > individually succeeds; I suspect this is due to a prior test case
> > not cleaning up properly and would therefore like to be able to
> > identify at a minimum what test cases are in a fork group so that I
> > can take a divide and conquer approach to resolving the issue, even
> > better would be to know
> which test cases ran before the failing test case.
> >
> >
> >
> > Thanks for your help,
> >
> >
> >
> > Martin.
> >
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Surefire - Forking - Identifying Tests that run in a particular fork

Tibor Digana
Thank you Martin for your ideas.
I have expected this goal in the console logs.

Can you support us with implementing this feature?
To make it more simple for contributors we have the abstraction in a
separate module.
Can you upgrade the class SurefireConsoleOutputReporter and open a
pullrequest on GitHub?
There should be a boolean parameter which specifically enables this feature
so that it stays disabled by default as we keep the backwards compatibility
with the previous behavior.
The code has all data available, so this is a sandbox for you and it is
quite doable i would say.

Cheers
Tibor

On Wed, Jul 8, 2020 at 11:43 PM Martin Lambert
<[hidden email]> wrote:

> Hi Tibor,
>
> Sorry it's taken a little while to get back to you.
>
> To set the scene, as basic requirements when running n tests serially I'd
> like to know...
>
> * When all pass: I'd mainly be interested in the fact that all tests
> passed, what tests ran and if any were skipped
> * If one or more tests failed: it seems reasonable to additionally be told
> which test(s) failed
> * If one or more tests failed and re-running a failed test individually
> did not reproduce a failure: it would be exceedingly helpful to also be
> able to see what tests ran before a failing test, including their output,
> in order to identify if prior tests were not cleaning up after themselves
>
> The reason for posting my original question was that when Surefire is
> configured to use forking, e.g. <forkCount>8</count>, then as far as I've
> been able to find out only the first 2 points above hold. To help visualise
> this, you could imagine getting console output something like:
>
> Running org.apache.maven.MyTest1
> Running org.apache.maven.MyTest32
> Running org.apache.maven.MyTest4
> Running org.apache.maven.MyTest77
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ...
> Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2
> skipped: 0
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> Running org.apache.maven.MyTest2
> ...
>
> What I'm looking for is to be able to untangle this obfuscated log i.e.
> only look at the output for a single fork, and if necessary thread, thereby
> getting back to a similar place as running a set of tests serially. As an
> example, given the output below one could easily filter the output based on
> "[Fork 2]":
>
> [Fork 1] Running org.apache.maven.MyTest1
> [Fork 2] Running org.apache.maven.MyTest32
> [Fork 3] Running org.apache.maven.MyTest4
> [Fork 4] Running org.apache.maven.MyTest77
> [Fork 2] ~~~~~~ Output from test unknown
> [Fork 2] ~~~~~~ Output from test unknown
> [Fork 1] ~~~~~~ Output from test unknown
> [Fork 3] ~~~~~~ Output from test unknown
> [Fork 2] ~~~~~~ Output from test unknown
> ...
> [Fork 3]  Test finished: org.apache.maven.MyTest4 total: 6 success: 4
> failure: 2 skipped: 0
> [Fork 4] ~~~~~~ Output from test unknown
> [Fork 4] ~~~~~~ Output from test unknown
> [Fork 3]  Running org.apache.maven.MyTest2
> ...
>
> My current work around achieves something similar to the last example and
> uses:
> * The JUnit TestExecutionListener mechanism to supplement the existing
> JUnit / Surefire log lines with test start and end messages that include a
> given fork JVM's process id, this could easily be expanded to include the
> thread if needed
> * Changed my test logging code to also include a given fork JVM's process
> id; again thread could be included if needed
>
> I did look at the examples page, however, on reflection it seems that if
> the fork functionality doesn't provide a mechanism for this out of the box
> then it would be highly desirable for it to do so based on the reasons
> outlined; of course, if the fork mechanism already supports doing this then
> it would be great to hear how to configure Surefire in order to get this
> behaviour.
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Tibor Digana <[hidden email]>
> Sent: 05 July 2020 11:59
> To: [hidden email]
> Cc: Maven Users List <[hidden email]>
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> Can you be more concrete with your business expectations?
> What logger, what kind of notations about forked JVM id you want to
> express in the logs or somewhere else?
>
> Maybe you are not aware of new features in Surefire and Failsafe, but we
> introduced Extensions API in 3.0.0-M4.
> This way you have a chance to implement the Extensions API as you like.
> It means that the features of loggers and wrapped configuration parameters
> can be customized by adding new ones.
> See the documentation
>
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40DisplayName
> It is maybe easier for the user due to now it is quite simple for the
> contributor to concentrate on the abstraction which easily leads you to the
> (default) implementation.
>
> The extensions concerns anything, not only junit5.
>
> Cheers
> T
>
> On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert <
> [hidden email]> wrote:
>
> > Hi Falko,
> >
> > Thanks for the suggestion. In the end I created a simple Junit
> > TestExecutionListener class that overrode executionStarted() and
> > executionFinished(); both methods have a TestIdentifier parameter, if
> > this is actually an instance of MethodSource (
> > https://junit.org/junit5/docs/current/api/org.junit.platform.engine/or
> > g/junit/platform/engine/support/descriptor/MethodSource.html)
> > you can downcast it, this allows you can get hold of the test class'
> > name using getClassName() on the same object; somewhere I was able to
> > call an
> > isTest() method to filter out noise - I can't remember where off the
> > top of my head. To make the whole thing work you have to create a
> > ServiceLoader file as mentioned in 6.1.4 here:
> > https://junit.org/junit5/docs/current/user-guide/#launcher-api-listene
> > rs-custom (it really does just involve creating the specific file in
> > the location mentioned, containing only the fully qualified name of
> > your TestExecutionListener class 😊 ).
> >
> > Cheers,
> >
> > Martin.
> >
> > -----Original Message-----
> > From: Falko Modler <[hidden email]>
> > Sent: 01 July 2020 23:48
> > To: [hidden email]
> > Subject: Re: Surefire - Forking - Identifying Tests that run in a
> > particular fork
> >
> > Hi Martin,
> >
> > OOTB you'll probably have to resort to -X (which logs way too much)
> > and even then I am not 100% sure you'll get what you need.
> >
> > Did you have a look at |${surefire.forkNumber}| (see
> > https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-
> > options-and-parallel-execution.html
> > )?
> >
> > I've been using this as a system property in my projects to:
> >
> > - write one logfile per fork (e.g. via Logback)
> > - prefix each logged line with the fork number (also via Logback,
> > especially useful for Jenins console log)
> > - log each test method in a JUnit4 listener together with the log file
> > (certainly also possible with JUnit5 etc.)
> > - separate working directories, generate tracing ids etc.
> >
> > I can really recommend to take these extra measures, otherwise
> > analyzing test failures with multiple forks and/or parallel modules
> > (-T) will be a real pain in the long run.
> >
> > Cheers,
> >
> > Falko
> >
> > Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > > Hello,
> > >
> > >
> > >
> > > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > > like to know if there is any way to identify what tests were run in
> > > a given fork? I haven't seen any option for it in the current
> > > documentation. I'm running around ~10K test cases hence using the
> > > fork option to speed things up. The problem I'm trying to solve is
> > > that a single test intermittently fails, which, when run
> > > individually succeeds; I suspect this is due to a prior test case
> > > not cleaning up properly and would therefore like to be able to
> > > identify at a minimum what test cases are in a fork group so that I
> > > can take a divide and conquer approach to resolving the issue, even
> > > better would be to know
> > which test cases ran before the failing test case.
> > >
> > >
> > >
> > > Thanks for your help,
> > >
> > >
> > >
> > > Martin.
> > >
> > >
> > >
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Surefire - Forking - Identifying Tests that run in a particular fork

Martin Lambert
Hi Tibor,

I'm happy to help. Please see here for my current progress https://github.com/apache/maven-surefire/compare/master...mlambert:master.

Prior to raising a pull request I was planning to:
* Add more test cases
* Test these changes with a local maven project - I'm hoping it will be as simple as "mvn clean install" and then update the Surefire dependency in the project being used as a test harness
* Add exporting of the current fork number, possibly as a System property, to allow tests to include the fork number in their output

If you prefer I can raise a request based on what I have so far.

Kind Regards,

Martin.

-----Original Message-----
From: Tibor Digana <[hidden email]>
Sent: 09 July 2020 12:05
To: Maven Users List <[hidden email]>
Subject: Re: Surefire - Forking - Identifying Tests that run in a particular fork

Thank you Martin for your ideas.
I have expected this goal in the console logs.

Can you support us with implementing this feature?
To make it more simple for contributors we have the abstraction in a separate module.
Can you upgrade the class SurefireConsoleOutputReporter and open a pullrequest on GitHub?
There should be a boolean parameter which specifically enables this feature so that it stays disabled by default as we keep the backwards compatibility with the previous behavior.
The code has all data available, so this is a sandbox for you and it is quite doable i would say.

Cheers
Tibor

On Wed, Jul 8, 2020 at 11:43 PM Martin Lambert <[hidden email]> wrote:

> Hi Tibor,
>
> Sorry it's taken a little while to get back to you.
>
> To set the scene, as basic requirements when running n tests serially
> I'd like to know...
>
> * When all pass: I'd mainly be interested in the fact that all tests
> passed, what tests ran and if any were skipped
> * If one or more tests failed: it seems reasonable to additionally be
> told which test(s) failed
> * If one or more tests failed and re-running a failed test
> individually did not reproduce a failure: it would be exceedingly
> helpful to also be able to see what tests ran before a failing test,
> including their output, in order to identify if prior tests were not
> cleaning up after themselves
>
> The reason for posting my original question was that when Surefire is
> configured to use forking, e.g. <forkCount>8</count>, then as far as
> I've been able to find out only the first 2 points above hold. To help
> visualise this, you could imagine getting console output something like:
>
> Running org.apache.maven.MyTest1
> Running org.apache.maven.MyTest32
> Running org.apache.maven.MyTest4
> Running org.apache.maven.MyTest77
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> ...
> Test finished: org.apache.maven.MyTest4 total: 6 success: 4 failure: 2
> skipped: 0
> ~~~~~~ Output from test unknown
> ~~~~~~ Output from test unknown
> Running org.apache.maven.MyTest2
> ...
>
> What I'm looking for is to be able to untangle this obfuscated log i.e.
> only look at the output for a single fork, and if necessary thread,
> thereby getting back to a similar place as running a set of tests
> serially. As an example, given the output below one could easily
> filter the output based on "[Fork 2]":
>
> [Fork 1] Running org.apache.maven.MyTest1 [Fork 2] Running
> org.apache.maven.MyTest32 [Fork 3] Running org.apache.maven.MyTest4
> [Fork 4] Running org.apache.maven.MyTest77 [Fork 2] ~~~~~~ Output from
> test unknown [Fork 2] ~~~~~~ Output from test unknown [Fork 1] ~~~~~~
> Output from test unknown [Fork 3] ~~~~~~ Output from test unknown
> [Fork 2] ~~~~~~ Output from test unknown ...
> [Fork 3]  Test finished: org.apache.maven.MyTest4 total: 6 success: 4
> failure: 2 skipped: 0
> [Fork 4] ~~~~~~ Output from test unknown [Fork 4] ~~~~~~ Output from
> test unknown [Fork 3]  Running org.apache.maven.MyTest2 ...
>
> My current work around achieves something similar to the last example
> and
> uses:
> * The JUnit TestExecutionListener mechanism to supplement the existing
> JUnit / Surefire log lines with test start and end messages that
> include a given fork JVM's process id, this could easily be expanded
> to include the thread if needed
> * Changed my test logging code to also include a given fork JVM's
> process id; again thread could be included if needed
>
> I did look at the examples page, however, on reflection it seems that
> if the fork functionality doesn't provide a mechanism for this out of
> the box then it would be highly desirable for it to do so based on the
> reasons outlined; of course, if the fork mechanism already supports
> doing this then it would be great to hear how to configure Surefire in
> order to get this behaviour.
>
> Cheers,
>
> Martin.
>
> -----Original Message-----
> From: Tibor Digana <[hidden email]>
> Sent: 05 July 2020 11:59
> To: [hidden email]
> Cc: Maven Users List <[hidden email]>
> Subject: Re: Surefire - Forking - Identifying Tests that run in a
> particular fork
>
> Hi Martin,
>
> Can you be more concrete with your business expectations?
> What logger, what kind of notations about forked JVM id you want to
> express in the logs or somewhere else?
>
> Maybe you are not aware of new features in Surefire and Failsafe, but
> we introduced Extensions API in 3.0.0-M4.
> This way you have a chance to implement the Extensions API as you like.
> It means that the features of loggers and wrapped configuration
> parameters can be customized by adding new ones.
> See the documentation
>
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit
> -platform.html#Surefire_Extensions_and_Reports_Configuration_for_.40Di
> splayName It is maybe easier for the user due to now it is quite
> simple for the contributor to concentrate on the abstraction which
> easily leads you to the
> (default) implementation.
>
> The extensions concerns anything, not only junit5.
>
> Cheers
> T
>
> On Fri, Jul 3, 2020 at 9:25 AM Martin Lambert <
> [hidden email]> wrote:
>
> > Hi Falko,
> >
> > Thanks for the suggestion. In the end I created a simple Junit
> > TestExecutionListener class that overrode executionStarted() and
> > executionFinished(); both methods have a TestIdentifier parameter,
> > if this is actually an instance of MethodSource (
> > https://junit.org/junit5/docs/current/api/org.junit.platform.engine/
> > or
> > g/junit/platform/engine/support/descriptor/MethodSource.html)
> > you can downcast it, this allows you can get hold of the test class'
> > name using getClassName() on the same object; somewhere I was able
> > to call an
> > isTest() method to filter out noise - I can't remember where off the
> > top of my head. To make the whole thing work you have to create a
> > ServiceLoader file as mentioned in 6.1.4 here:
> > https://junit.org/junit5/docs/current/user-guide/#launcher-api-liste
> > ne rs-custom (it really does just involve creating the specific file
> > in the location mentioned, containing only the fully qualified name
> > of your TestExecutionListener class 😊 ).
> >
> > Cheers,
> >
> > Martin.
> >
> > -----Original Message-----
> > From: Falko Modler <[hidden email]>
> > Sent: 01 July 2020 23:48
> > To: [hidden email]
> > Subject: Re: Surefire - Forking - Identifying Tests that run in a
> > particular fork
> >
> > Hi Martin,
> >
> > OOTB you'll probably have to resort to -X (which logs way too much)
> > and even then I am not 100% sure you'll get what you need.
> >
> > Did you have a look at |${surefire.forkNumber}| (see
> > https://maven.apache.org/surefire/maven-surefire-plugin/examples/for
> > k-
> > options-and-parallel-execution.html
> > )?
> >
> > I've been using this as a system property in my projects to:
> >
> > - write one logfile per fork (e.g. via Logback)
> > - prefix each logged line with the fork number (also via Logback,
> > especially useful for Jenins console log)
> > - log each test method in a JUnit4 listener together with the log
> > file (certainly also possible with JUnit5 etc.)
> > - separate working directories, generate tracing ids etc.
> >
> > I can really recommend to take these extra measures, otherwise
> > analyzing test failures with multiple forks and/or parallel modules
> > (-T) will be a real pain in the long run.
> >
> > Cheers,
> >
> > Falko
> >
> > Am 02.07.2020 um 00:12 schrieb Martin Lambert:
> > > Hello,
> > >
> > >
> > >
> > > I'm currently using the Surefire plugin with Maven 3.6.3 and would
> > > like to know if there is any way to identify what tests were run
> > > in a given fork? I haven't seen any option for it in the current
> > > documentation. I'm running around ~10K test cases hence using the
> > > fork option to speed things up. The problem I'm trying to solve is
> > > that a single test intermittently fails, which, when run
> > > individually succeeds; I suspect this is due to a prior test case
> > > not cleaning up properly and would therefore like to be able to
> > > identify at a minimum what test cases are in a fork group so that
> > > I can take a divide and conquer approach to resolving the issue,
> > > even better would be to know
> > which test cases ran before the failing test case.
> > >
> > >
> > >
> > > Thanks for your help,
> > >
> > >
> > >
> > > Martin.
> > >
> > >
> > >
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]