Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core

Elliotte Rusty Harold
This investigation and resolution of this mystery has been quite
informative. I suspect there are general principles and lessons here
about separating infrastructure for different projects that extend
well beyond Maven itself and probably beyond Jenkins. Containers might
play a role.

Perhaps someone would be interested in more formally publishing a full
writeup somewhere? Or perhaps delivering a lightning talk on this at
ApacheCon? This feels quite valuable for general DevOps.

On Sat, Aug 3, 2019 at 3:12 PM Robert Scholte <[hidden email]> wrote:

>
> Here's a very short summary and conclusion about what happened the last
> period.
>
> Half July some integration tests of Maven started to fail, but not always
> (it good happen that a certain JDK would succeed again with the next run).
> At that time we only noticed:
> - it failed only on the Linux nodes of Jenkins.
> - the failing tests were using MAVEN_OPTS
>
> It took quite a while, but in the end Hervé discovered the root cause:
> - On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
> that set the value of MAVEN_OPTS
>
> INFRA discovered that it was the incubator-shardingsphere that created
> this file when building there project.
>
> We've provided a PR for this project together with an explanation. This PR
> has been accepted and INFRA has removed the .mavenrc files.
>
> Over the past weeks a lot of changes has been done to the code for
> investigation.
> All changes that were done to change the behavior related to environments
> variables and mavenrc file should and will be reverted.
> Mavens master branch was stable, so to ensure the right things are still
> tested the need to be restored (some of them did indeed test the
> MAVEN_OPTS explicitly as issued in its JIRA).
> In general we don't change ITs, we only extend then and provide a range to
> control for which Maven versions the test is valid.
>
> Details and discussions can be found on the slack channel.
>
> thanks,
> Robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


--
Elliotte Rusty Harold
[hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core

Tibor Digana
Hi devs,

I hope you have spent a nice weekend ;-)

Integrated new Surefire feature [1] in the core-its branch [2] which fixed
the first issue (see point 1.) and the build is successful [3].

After making an analysis, with a help of the logs, we can see that there
are 3 known issues at least:

1. Surefire forked JVM should not be spied by Agent of Jenkins. This Agent
is mandatory in Maven build process.
   Using unnecessary agent in Surefire JVM would mean that the tests fail
with well known JUnit4 assertion failures [4].
   Removing JAVA_TOOL_OPTIONS cut the agent off in the tests.
   [proof by Stephen Connolly] Ensure any forked Maven executions are not
spied on by Jenkins
2. a JVM warning is printed in standard output and caused by the commit [5]
where the tests expected clean stream, see [6].
   The build [7] contains the commit 690841e9 and fails. This was found and
proved by bisect of comits.
3. MAVEN_OPTS
   There are several variants.
   Herve's investigation which is fine, and
   mine [8] which is an improvement taking developer's mavenOpts from his
build into Surefire's forked JVM.
   Additionally, I skipped [9] global env.MAVEN_OPTS from '.mavenrc' in
order to make sure that the influence of the system (hardware) is avoided
and I can fully consider concentrate only on software issues.
   So this helps me to separate the influence and avoid a mix of influences
in my investigation process.

[1]:
https://github.com/apache/maven-surefire/commit/065001b2e6417126fd80d5f1a5523a3469e3dd5a
[2]:
https://github.com/apache/maven-integration-testing/commit/5be07fd52727ef09dcc29539e80fb6aff534ad3b#diff-600376dffeb79835ede4a0b285078036R157
[3]:
https://builds.apache.org/job/maven-box/job/maven/job/removed-java-tools-2/2/
[4]: expected:<[]> but was:<[Picked up JAVA_TOOL_OPTIONS: ...]>
[5]: 7/24/2019, 690841e9, [MNG-6697]
https://github.com/apache/maven/pull/261#issuecomment-517999286
[6]: expected:<[]> but was:<[WARNING: An illegal reflective access
operation has occurred ... ]>.
     For more information see [5].
[7]:
https://builds.apache.org/job/maven-box/job/maven/job/removed-java-tools-3/
[8]:
https://github.com/apache/maven-integration-testing/commit/5a40bb177b5ad3851cabf20c7983998d76258860#diff-089d3294860484e2addcd4d75bf8596fR352

https://github.com/apache/maven/commit/ad58ca489038829d5c9d54d7201ddf3eac49ad20
[9]:
https://github.com/apache/maven/commit/e0a67951a8c5258546c48cd45a74a31a71b74424

Cheers
Tibor17

On Sat, Aug 3, 2019 at 9:12 PM Robert Scholte <[hidden email]> wrote:

> Here's a very short summary and conclusion about what happened the last
> period.
>
> Half July some integration tests of Maven started to fail, but not always
> (it good happen that a certain JDK would succeed again with the next
> run).
> At that time we only noticed:
> - it failed only on the Linux nodes of Jenkins.
> - the failing tests were using MAVEN_OPTS
>
> It took quite a while, but in the end Hervé discovered the root cause:
> - On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
> that set the value of MAVEN_OPTS
>
> INFRA discovered that it was the incubator-shardingsphere that created
> this file when building there project.
>
> We've provided a PR for this project together with an explanation. This
> PR
> has been accepted and INFRA has removed the .mavenrc files.
>
> Over the past weeks a lot of changes has been done to the code for
> investigation.
> All changes that were done to change the behavior related to environments
> variables and mavenrc file should and will be reverted.
> Mavens master branch was stable, so to ensure the right things are still
> tested the need to be restored (some of them did indeed test the
> MAVEN_OPTS explicitly as issued in its JIRA).
> In general we don't change ITs, we only extend then and provide a range
> to
> control for which Maven versions the test is valid.
>
> Details and discussions can be found on the slack channel.
>
> thanks,
> Robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>