3 ITs permanently fail on Linux / Maven Core

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

3 ITs permanently fail on Linux / Maven Core

Tibor Digana
Heads up!,

I am investigating the build errors on Maven Core.
So I created three branches moved the HEAD backwards (i.e. git reset --hard
HEAD~12) and observed the outcome.

I have investigated 29 commits. Not sure how far to go...

`maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11, 12
(16 ITs)
`maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
ITs)
`maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)

Always the ITs 0553, 4590, 4747 fail on several nodes.
Always related to Linux.

See the list of errors and branches:

https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-reset-head-12/1/#showFailuresLink
https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-reset-head-14/1/
https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3-reset-head-29/1/



From the MavenITmng0553SettingsAuthzEncryptionTest:

[ERROR] Error executing Maven.
org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml (No
such file or directory)
    at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
(SecUtil.java:69)
    at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

[ERROR] Failed to execute goal on project test: Could not resolve
dependencies for project
org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to collect
dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT:
Failed to read artifact descriptor for
org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not transfer
artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to test
(http://localhost:32917/): Not authorized



From the MavenITmng4590ImportedPomUsesSystemPropertiesTest:

expected:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-head-29/test/core-it-suite/target/test-classes/mng-4590/pom.xml>
but was:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-head-29/test/core-it-suite/target/test-classes/mng-4590/${test.file}>



From the MavenITmng4747JavaAgentUsedByPluginTest:

junit.framework.AssertionFailedError
        at org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit(MavenITmng4747JavaAgentUsedByPluginTest.java:63)
Reply | Threaded
Open this post in threaded view
|

Re: 3 ITs permanently fail on Linux / Maven Core

Hervé BOUTEMY
I took time to launch the build many times and got to following conclusion:

- the only issue is about MAVEN_OPTS not being honoured in forked JVM
- it's not related to a JDK version but mode Jenkins node:
-> failing nodes: H24, H27, H28, H29, H33, H37
-> working nodes: H31, H32, H34, H44


to me, Maven core master branch is currently stable: the 4 ITs failing we see
are caused by ASF Jenkins issues
If someone wants to launch a release, I'm fully supportive

Of course, if anybody has an idea on the root cause to the MAVEN_OPTS issue,
I'm all ears open...

Regards,

Hervé

Le samedi 20 juillet 2019, 19:53:05 CEST Tibor Digana a écrit :

> Hello Herve,
>
> no problem. I am investigating in
> https://builds.apache.org/job/maven-box/job/maven/job/EOL/
> the build is named after end-of-line because I have expected isues like
> that. Only Jenkins Linux build fail, pls my last two commits in the branch
> EOL.
>
> My commits correspond to ".gitattributes" file and Jenkinsfile.
> In ".gitattributes" I set eol=LF and I am archiving the folder
> "core-it-suite" in Jenkinsfile.
>
> It is strange that only Linux builds fail and thus there's is a kind of
> platform dependency as it seems in this error.
> I hope the archived ZIP file tells us more, e.g. logs at least.
>
>
> Definitely something is wrong on Jenkins/Linux or in Jenkinsfile because
> the build #22 was successful in
> https://builds.apache.org/job/maven-box/job/maven/job/MNG-6294/ one month
> ago, but I manually triggered #23 and I have got the same set of failed ITs
> as we have in master today.
>
> Cheers
> Tibor
>
> On Sat, Jul 20, 2019 at 7:25 PM Hervé BOUTEMY <[hidden email]> wrote:
> > I don't get where you want to go: there is no disagreement on reverting
> > resolver from 1.4.0 to 1.3.3
> >
> > the question is, once this revert has been done, about the 4 ITs failing
> > [2]:
> > master #242 linux-jdk7 / Run ITs Linux Java 7
> > 1. org.apache.maven.it
> > .MavenITmng0553SettingsAuthzEncryptionTest.testitEncryption
> > 2. org.apache.maven.it
> > .MavenITmng0553SettingsAuthzEncryptionTest.testitBasic
> > 3. org.apache.maven.it
> > .MavenITmng4590ImportedPomUsesSystemPropertiesTest.testit
> > 4. org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit
> >
> > I investigated these ITs and found one common fact: they try to define
> > MAVEN_OPTS (and are the only ones during that)
> >
> >         verifier.setEnvironmentVariable( "MAVEN_OPTS", "...." );
> >
> > and this does not seem to be transferred to the verifier execution
> >
> > I don't know yet why this does not work on some Jenkins nodes, trying to
> > add a few additional output to see if I can get some hints...
> >
> > any idea welcome.
> >
> > Regards,
> >
> > Hervé
> >
> > [2]
> > https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/
> > master/242/testReport/>
> > Le samedi 20 juillet 2019, 12:48:27 CEST Tibor Digana a écrit :
> > > I wrote in our chat that Maven Core fails randomly on random JDK
> > > versions
> > > due to maven-resolver:1.4.0. It was proved by:
> > >
> > > + Jenkins on Maven Core
> > > + Jira MNG-6714
> > > + logical analysis of code in resolver.
> > >
> > > I wrote to Harald Wellmann that we should make code review together.
> > > In my profession and company, the multithreading was my Java Advanced
> > > expertize, so I am able to evaluate code which is thread safe and which
> >
> > is
> >
> > > not. Additionally, resolver:1.4.0 has logical problem even in single
> >
> > thread
> >
> > > which was proved by the branch where 11 unit tests failed on single
> >
> > thread
> >
> > > which of course must never happen.
> > >
> > > These things prove me to say that MRESOLVER-7 is not applicable in
> > > production and we have to make code review together!
> > >
> > > Sorry to say, I can teach few devs with JSR-133 and thread safety
> >
> > regarding
> >
> > > Java Memory Model, and then the devs would probably understand why I
> >
> > found
> >
> > > issues in the code. Nevertheless still there are two problems with the
> > > algorithm itself for whatever number of threads are executed:
> > >
> > > + one object is concurrently modified across multiple threads and the
> > > object unexpectedly changes the state of the object
> > > + there is one loop which resolves version but the last iteration wins,
> >
> > so
> >
> > > it loop looks to me ilogical and causes unnecessary object state
> > > modifications
> > >
> > >
> > > To be unbiased, all these things have to be clarified in PR on GitHub,
> > > dicsussed, and changes have to be made in order to provide STABLE
> >
> > resolver.
> >
> > > That's all I can say in this issue.
> > > Cheers
> > > Tibor17
> > >
> > >
> > >
> > > On Sat, Jul 20, 2019 at 12:25 PM Hervé BOUTEMY <[hidden email]>
> > >
> > > wrote:
> > > > little update on master branch runs in ASF Jenkins [1]:
> > > > build #240: stupidly failed on a Git issue on a node
> > > >
> > > > then I launched build #241 *which worked fully*!!!
> > > >
> > > > to be sure of the stability, I relaunched the build, and build #242
> >
> > failed
> >
> > > > on
> > > > the 4  ITs on Linux JDK 7: really strange. FYI, I'm personally on
> > > > Linux
> > > > with
> > > > Java 7 and I can't reproduce the issue. And the fact that Maven 3.6.0
> > > > branch
> > > > fails on the same ITs, but sometimes on other configurations, makes me
> > > > think
> > > > there is an issue on some slave nodes on ASF Jenkins
> > > >
> > > > I integrated one little commit and build #243 still fails on the same
> > > > 4
> > > > ITs
> > > >
> > > >
> > > > I suspect it's an issue on ASF Jenkins, but I can't prove anything: I
> > > > can't
> > > > even tell which slave node run successfully for build #241 but which
> >
> > ones
> >
> > > > failed on next builds...
> > > >
> > > > Any idea on how to investigate?
> > > > Should this postpone the 3.6.2 release or not? To me, MNG-6712 fixed
> >
> > the
> >
> > > > real
> > > > issue, that was causing infinite loops during artifacts resolution.
> > > >
> > > >
> > > > As a side note, I started by doing a lot of cleanup on old merged
> > > > branches: it
> > > > would be nice if everybody did its own cleanup when merging
> > > >
> > > > Regards,
> > > >
> > > > Hervé
> > > >
> > > >
> > > > [1]
> >
> > https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job/
> >
> > > > master/
> > > > <
> >
> > https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven/job
> >
> > > > /master/>>
> > > >
> > > > Le mardi 16 juillet 2019, 23:22:22 CEST Tibor Digana a écrit :
> > > > > Heads up!,
> > > > >
> > > > > I am investigating the build errors on Maven Core.
> > > > > So I created three branches moved the HEAD backwards (i.e. git reset
> > > >
> > > > --hard
> > > >
> > > > > HEAD~12) and observed the outcome.
> > > > >
> > > > > I have investigated 29 commits. Not sure how far to go...
> > > > >
> > > > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8,
> >
> > 11,
> >
> > > > 12
> > > >
> > > > > (16 ITs)
> > > > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and
> >
> > 8 (8
> >
> > > > > ITs)
> > > > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > > > >
> > > > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > > > Always related to Linux.
> >
> > > > > See the list of errors and branches:
> > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> >
> > > > -r>
> > > >
> > > > > eset-head-12/1/#showFailuresLink
> >
> > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> >
> > > > -> reset-head-14/1/
> >
> > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> >
> > > > -> reset-head-29/1/
> > > >
> > > > > From the MavenITmng0553SettingsAuthzEncryptionTest:
> > > > >
> > > > > [ERROR] Error executing Maven.
> > > > > org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException
> > > > > :
> >
> > > > > java.io.FileNotFoundException:
> > /home/jenkins/.m2/settings-security.xml
> >
> > > > (No
> > > >
> > > > > such file or directory)
> > > > >
> > > > >     at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> > > > >
> > > > > (SecUtil.java:69)
> > > > >
> > > > >     at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> > > > >     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> > > > >     at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
> > > > >
> > > > > [ERROR] Failed to execute goal on project test: Could not resolve
> > > > > dependencies for project
> > > > > org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to
> > > > > collect
> > > > > dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT:
> > > > > Failed to read artifact descriptor for
> > > > > org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not transfer
> > > > > artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to
> > > > > test
> > > > > (http://localhost:32917/): Not authorized
> >
> > > > > From the MavenITmng4590ImportedPomUsesSystemPropertiesTest:
> > expected:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-h
> >
> > > > ea>
> > > >
> > > > > d-29/test/core-it-suite/target/test-classes/mng-4590/pom.xml> but
> >
> > was:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-head-2
> >
> > > > 9
> > > >
> > > > > /test/core-it-suite/target/test-classes/mng-4590/${test.file}>
> > > > >
> > > > >
> > > > >
> > > > > From the MavenITmng4747JavaAgentUsedByPluginTest:
> > > > >
> > > > > junit.framework.AssertionFailedError
> > > > >
> > > > >       at
> > > > >
> > > > > org.apache.maven.it
> > > >
> > > > .MavenITmng4747JavaAgentUsedByPluginTest.testit(MavenITm
> > > >
> > > > > ng4747JavaAgentUsedByPluginTest.java:63)
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [hidden email]
> > > > For additional commands, e-mail: [hidden email]
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: 3 ITs permanently fail on Linux / Maven Core

Hervé BOUTEMY
In reply to this post by Tibor Digana
INFRA-18812 created

https://issues.apache.org/jira/browse/INFRA-18812

Le mardi 30 juillet 2019, 07:39:49 CEST Hervé BOUTEMY a écrit :

> I found the root cause of failing IT on some Linux nodes!!!
>
>
> activating debug mode for mvn shell script, we get:
> + [ -f /home/jenkins/.mavenrc ]
> + . /home/jenkins/.mavenrc
> + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
>
> there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
> instead of appending: I did not yet report to INFRA, need to check if this
> variable is defined with Puppet and provide a PR
>
> FYI, core ITs log files have finally always been accessible: just need to
> look inside
> org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT-
> tests.jar artifact
>
>
> This debugging session has been hard, but I'm happy to say that it is now
> under control: no more wild guesses about strange root causes, but pure
> simple logic.
>
> Regards,
>
> Hervé
>
> Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > We discuss this issue in Slack
> > We know for certain that there is a problem with environment variables.
> > So there must be a bug in our code then. I have analysed the code I think
> > the problem should be with spaces in the value of environment variable
> > MAVEN_OPTS. There is no quotation marks around such environemnt variable
> > when we have such rough element in String[] when we call
> > Runtime.execute(cli, String[]).
> >
> > Here is the list of comments from Slack and the way how we got there:
> >
> > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> > the IT but I got to `execute()` method and I was interested in `String[]
> > environment = getEnvironmentVariables();` which finally is passed to Java
> >
> > running forked process. Please tell me if this code is okay for you:
> >     ```public String[] getEnvironmentVariables()
> >     {
> >    
> >         addSystemEnvironment();
> >         String[] environmentVars = new String[envVars.size()];
> >         int i = 0;
> >         for ( String name : envVars.keySet() )
> >         {
> >        
> >             String value = envVars.get( name );
> >             environmentVars[i] = name + "=" + value;
> >             i++;
> >        
> >         }
> >         return environmentVars;
> >    
> >     }```
> >
> > I do not see quotation marks. This is important if you use two or more
> > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> > Next issue is that the env vars of the parent process are merged with
> > given
> > vars for the child process in `addSystemEnvironment();`, Line 239.
> >
> > that's why maybe platform Linux does not understand the space and maybe
> > Windows does not have a problem with it.
> >
> > merging env vars is strange because of JAVA_HOME, there are at least two.
> > One in CLI and second in env vars.
> >
> > I think we should use latest `maven-shared-utils` and check the code in
> > shared utils. Maybe to fix shared utils and meanwhile use snapshot
> > version,
> > maybe.
> >
> >
> >
> >
> > On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana <[hidden email]>
> >
> > wrote:
> > > Heads up!,
> > >
> > > I am investigating the build errors on Maven Core.
> > > So I created three branches moved the HEAD backwards (i.e. git reset
> > > --hard HEAD~12) and observed the outcome.
> > >
> > > I have investigated 29 commits. Not sure how far to go...
> > >
> > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8, 11,
> > > 12
> > > (16 ITs)
> > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8 (8
> > > ITs)
> > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > >
> > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > Always related to Linux.
> > >
> > > See the list of errors and branches:
> > >
> > >
> > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3
> > > .3
> > > -reset-head-12/1/#showFailuresLink
> > >
> > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3
> > > .3
> > > -reset-head-14/1/
> > >
> > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3
> > > .3
> > > -reset-head-29/1/
> > >
> > >
> > >
> > > From the MavenITmng0553SettingsAuthzEncryptionTest:
> > >
> > > [ERROR] Error executing Maven.
> > > org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> > > java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml
> > > (No
> > > such file or directory)
> > >
> > >     at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> > >
> > > (SecUtil.java:69)
> > >
> > >     at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> > >     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> > >     at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
> > >
> > > [ERROR] Failed to execute goal on project test: Could not resolve
> > > dependencies for project
> > > org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to collect
> > > dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Failed
> > > to read artifact descriptor for
> > > org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not transfer
> > > artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to test
> > > (http://localhost:32917/): Not authorized
> > >
> > >
> > >
> > > From the MavenITmng4590ImportedPomUsesSystemPropertiesTest:
> > >
> > > expected:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset
> > > -h
> > > ead-29/test/core-it-suite/target/test-classes/mng-4590/pom.xml> but
> > > was:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-head
> > > -
> > > 29/test/core-it-suite/target/test-classes/mng-4590/${test.file}>
> > >
> > >
> > >
> > > From the MavenITmng4747JavaAgentUsedByPluginTest:
> > >
> > > junit.framework.AssertionFailedError
> > >
> > > at
> > >
org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit(Mave
> > > n
> > > ITmng4747JavaAgentUsedByPluginTest.java:63)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: 3 ITs permanently fail on Linux / Maven Core

Tibor Digana
In reply to this post by Tibor Digana
Essentially you want to say that global environment variable overrides
local MAVEN_OPTS?
This way I have undertood your email.
but I would say that it cannot be!

Download the Maven dist and open the scripts $MVN_HOME/bin/mvn and there we
will see this.
This script in our distcorresponds to what I see in your email.
So I think all we have to do is to export MAVEN_SKIP_RC to 1 in our ITs and
prevent from merging global and local environment variables.

if [ -z "$MAVEN_SKIP_RC" ] ; then
  if [ -f /etc/mavenrc ] ; then
    . /etc/mavenrc
  fi
  if [ -f "$HOME/.mavenrc" ] ; then
    . "$HOME/.mavenrc"
  fi
fi
...
MAVEN_OPTS="`concat_lines "$MAVEN_PROJECTBASEDIR/.mvn/jvm.config"`
$MAVEN_OPTS"

We have a lots of troubles with Jenkins.
Solving this does not mean anything becase we ca see in the logs that
"apache/maven-archetype" randomly lost files.
We have problem with networ connectivity to Maven Central, every second day
and more ofter in the night of European time.

Cheers
Tibor17



On Tue, Jul 30, 2019 at 7:40 AM Hervé BOUTEMY <[hidden email]> wrote:

> I found the root cause of failing IT on some Linux nodes!!!
>
>
> activating debug mode for mvn shell script, we get:
> + [ -f /home/jenkins/.mavenrc ]
> + . /home/jenkins/.mavenrc
> + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
>
> there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
> instead of appending: I did not yet report to INFRA, need to check if this
> variable is defined with Puppet and provide a PR
>
> FYI, core ITs log files have finally always been accessible: just need to
> look inside
> org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT-tests.jar
> artifact
>
>
> This debugging session has been hard, but I'm happy to say that it is now
> under control: no more wild guesses about strange root causes, but pure
> simple logic.
>
> Regards,
>
> Hervé
>
> Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > We discuss this issue in Slack
> > We know for certain that there is a problem with environment variables.
> > So there must be a bug in our code then. I have analysed the code I think
> > the problem should be with spaces in the value of environment variable
> > MAVEN_OPTS. There is no quotation marks around such environemnt variable
> > when we have such rough element in String[] when we call
> > Runtime.execute(cli, String[]).
> >
> > Here is the list of comments from Slack and the way how we got there:
> >
> > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> > the IT but I got to `execute()` method and I was interested in `String[]
> > environment = getEnvironmentVariables();` which finally is passed to Java
> > running forked process. Please tell me if this code is okay for you:
> >     ```public String[] getEnvironmentVariables()
> >     {
> >         addSystemEnvironment();
> >         String[] environmentVars = new String[envVars.size()];
> >         int i = 0;
> >         for ( String name : envVars.keySet() )
> >         {
> >             String value = envVars.get( name );
> >             environmentVars[i] = name + "=" + value;
> >             i++;
> >         }
> >         return environmentVars;
> >     }```
> > I do not see quotation marks. This is important if you use two or more
> > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> > Next issue is that the env vars of the parent process are merged with
> given
> > vars for the child process in `addSystemEnvironment();`, Line 239.
> >
> > that's why maybe platform Linux does not understand the space and maybe
> > Windows does not have a problem with it.
> >
> > merging env vars is strange because of JAVA_HOME, there are at least two.
> > One in CLI and second in env vars.
> >
> > I think we should use latest `maven-shared-utils` and check the code in
> > shared utils. Maybe to fix shared utils and meanwhile use snapshot
> version,
> > maybe.
> >
> >
> >
> >
> > On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana <[hidden email]>
> >
> > wrote:
> > > Heads up!,
> > >
> > > I am investigating the build errors on Maven Core.
> > > So I created three branches moved the HEAD backwards (i.e. git reset
> > > --hard HEAD~12) and observed the outcome.
> > >
> > > I have investigated 29 commits. Not sure how far to go...
> > >
> > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8,
> 11, 12
> > > (16 ITs)
> > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8
> (8
> > > ITs)
> > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > >
> > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > Always related to Linux.
> > >
> > > See the list of errors and branches:
> > >
> > >
> > >
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> > > -reset-head-12/1/#showFailuresLink
> > >
> > >
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> > > -reset-head-14/1/
> > >
> > >
> https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1.3.3
> > > -reset-head-29/1/
> > >
> > >
> > >
> > > From the MavenITmng0553SettingsAuthzEncryptionTest:
> > >
> > > [ERROR] Error executing Maven.
> > > org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> > > java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml
> (No
> > > such file or directory)
> > >
> > >     at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> > >
> > > (SecUtil.java:69)
> > >
> > >     at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> > >     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> > >     at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
> > >
> > > [ERROR] Failed to execute goal on project test: Could not resolve
> > > dependencies for project
> > > org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to collect
> > > dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Failed
> > > to read artifact descriptor for
> > > org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not transfer
> > > artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to test
> > > (http://localhost:32917/): Not authorized
> > >
> > >
> > >
> > > From the MavenITmng4590ImportedPomUsesSystemPropertiesTest:
> > >
> > >
> expected:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-h
> > > ead-29/test/core-it-suite/target/test-classes/mng-4590/pom.xml> but
> > >
> was:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-head-
> > > 29/test/core-it-suite/target/test-classes/mng-4590/${test.file}>
> > >
> > >
> > >
> > > From the MavenITmng4747JavaAgentUsedByPluginTest:
> > >
> > > junit.framework.AssertionFailedError
> > >
> > >     at
> > >     org.apache.maven.it
> .MavenITmng4747JavaAgentUsedByPluginTest.testit(Maven
> > >     ITmng4747JavaAgentUsedByPluginTest.java:63)
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: 3 ITs permanently fail on Linux / Maven Core

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
here it is: issue fixed

as you can see by reading the INFRA-18812 Jira issue [1]:
- .mavenrc file polluting Jenkins slaves was created by Shardingshpere project
- Robert proposed a PR to the project
- INFRA removed unwanted .mavenrc files

now the ITs depending on MAVEN_OPTS are back working
https://builds.apache.org/job/maven-box/job/maven/job/MAVEN_OPTS/

time to fix other issues (and drop workarounds prepared while investigating)

Regards,

Hervé

[1] https://issues.apache.org/jira/browse/INFRA-18812

Le mardi 30 juillet 2019, 07:57:27 CEST Hervé BOUTEMY a écrit :

> INFRA-18812 created
>
> https://issues.apache.org/jira/browse/INFRA-18812
>
> Le mardi 30 juillet 2019, 07:39:49 CEST Hervé BOUTEMY a écrit :
> > I found the root cause of failing IT on some Linux nodes!!!
> >
> >
> > activating debug mode for mvn shell script, we get:
> > + [ -f /home/jenkins/.mavenrc ]
> > + . /home/jenkins/.mavenrc
> > + MAVEN_OPTS=-Xmx1024m -XX:MaxPermSize=256m
> >
> > there is a .mavenrc script on some Linux nodes that overrides MAVEN_OPTS
> > instead of appending: I did not yet report to INFRA, need to check if this
> > variable is defined with Puppet and provide a PR
> >
> > FYI, core ITs log files have finally always been accessible: just need to
> > look inside
> > org\apache\maven\its\core-it-suite\2.1-SNAPSHOT\core-it-suite-2.1-SNAPSHOT
> > -
> > tests.jar artifact
> >
> >
> > This debugging session has been hard, but I'm happy to say that it is now
> > under control: no more wild guesses about strange root causes, but pure
> > simple logic.
> >
> > Regards,
> >
> > Hervé
> >
> > Le dimanche 28 juillet 2019, 16:20:59 CEST Tibor Digana a écrit :
> > > We discuss this issue in Slack
> > > We know for certain that there is a problem with environment variables.
> > > So there must be a bug in our code then. I have analysed the code I
> > > think
> > > the problem should be with spaces in the value of environment variable
> > > MAVEN_OPTS. There is no quotation marks around such environemnt variable
> > > when we have such rough element in String[] when we call
> > > Runtime.execute(cli, String[]).
> > >
> > > Here is the list of comments from Slack and the way how we got there:
> > >
> > > the `forkMode=auto` in profile `embedded` so it's okay. But I analyzed
> > > `MavenITmng4747JavaAgentUsedByPluginTest` and Line  53 `newVerifier(
> > > testDir.getAbsolutePath() )` and I got to `CommandLine` class from
> > > `maven-shared-utils:0.9`. Although forking is used if we use env vars in
> > > the IT but I got to `execute()` method and I was interested in `String[]
> > > environment = getEnvironmentVariables();` which finally is passed to
> > > Java
> > >
> > > running forked process. Please tell me if this code is okay for you:
> > >     ```public String[] getEnvironmentVariables()
> > >     {
> > >    
> > >         addSystemEnvironment();
> > >         String[] environmentVars = new String[envVars.size()];
> > >         int i = 0;
> > >         for ( String name : envVars.keySet() )
> > >         {
> > >        
> > >             String value = envVars.get( name );
> > >             environmentVars[i] = name + "=" + value;
> > >             i++;
> > >        
> > >         }
> > >         return environmentVars;
> > >    
> > >     }```
> > >
> > > I do not see quotation marks. This is important if you use two or more
> > > systm properties in `MAVEN_OPTS=-Dk1=v1 -Dk2=v2` - there is a space!
> > > Next issue is that the env vars of the parent process are merged with
> > > given
> > > vars for the child process in `addSystemEnvironment();`, Line 239.
> > >
> > > that's why maybe platform Linux does not understand the space and maybe
> > > Windows does not have a problem with it.
> > >
> > > merging env vars is strange because of JAVA_HOME, there are at least
> > > two.
> > > One in CLI and second in env vars.
> > >
> > > I think we should use latest `maven-shared-utils` and check the code in
> > > shared utils. Maybe to fix shared utils and meanwhile use snapshot
> > > version,
> > > maybe.
> > >
> > >
> > >
> > >
> > > On Tue, Jul 16, 2019 at 11:22 PM Tibor Digana <[hidden email]>
> > >
> > > wrote:
> > > > Heads up!,
> > > >
> > > > I am investigating the build errors on Maven Core.
> > > > So I created three branches moved the HEAD backwards (i.e. git reset
> > > > --hard HEAD~12) and observed the outcome.
> > > >
> > > > I have investigated 29 commits. Not sure how far to go...
> > > >
> > > > `maven-resolver-1.3.3-reset-head-12` crashed with Linux + JDK 7, 8,
> > > > 11,
> > > > 12
> > > > (16 ITs)
> > > > `maven-resolver-1.3.3-reset-head-14` crashed with Linux + JDK 7 and 8
> > > > (8
> > > > ITs)
> > > > `maven-resolver-1.3.3-reset-head-29` crashed Linux JDK 8 (4 ITs)
> > > >
> > > > Always the ITs 0553, 4590, 4747 fail on several nodes.
> > > > Always related to Linux.
> > > >
> > > > See the list of errors and branches:
> > > >
> > > >
> > > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1
> > > > .3
> > > > .3
> > > > -reset-head-12/1/#showFailuresLink
> > > >
> > > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1
> > > > .3
> > > > .3
> > > > -reset-head-14/1/
> > > >
> > > > https://builds.apache.org/job/maven-box/job/maven/job/maven-resolver-1
> > > > .3
> > > > .3
> > > > -reset-head-29/1/
> > > >
> > > >
> > > >
> > > > From the MavenITmng0553SettingsAuthzEncryptionTest:
> > > >
> > > > [ERROR] Error executing Maven.
> > > > org.sonatype.plexus.components.sec.dispatcher.SecDispatcherException:
> > > > java.io.FileNotFoundException: /home/jenkins/.m2/settings-security.xml
> > > > (No
> > > > such file or directory)
> > > >
> > > >     at org.sonatype.plexus.components.sec.dispatcher.SecUtil.read
> > > >
> > > > (SecUtil.java:69)
> > > >
> > > >     at org.apache.maven.cli.MavenCli.encryption (MavenCli.java:920)
> > > >     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:286)
> > > >     at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
> > > >
> > > > [ERROR] Failed to execute goal on project test: Could not resolve
> > > > dependencies for project
> > > > org.apache.maven.its.mng0553:test:jar:1.0-SNAPSHOT: Failed to collect
> > > > dependencies at org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT:
> > > > Failed
> > > > to read artifact descriptor for
> > > > org.apache.maven.its.mng0553:a:jar:0.1-SNAPSHOT: Could not transfer
> > > > artifact org.apache.maven.its.mng0553:a:pom:0.1-SNAPSHOT from/to test
> > > > (http://localhost:32917/): Not authorized
> > > >
> > > >
> > > >
> > > > From the MavenITmng4590ImportedPomUsesSystemPropertiesTest:
> > > >
> > > > expected:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-res
> > > > et
> > > > -h
> > > > ead-29/test/core-it-suite/target/test-classes/mng-4590/pom.xml> but
> > > > was:</home/jenkins/jenkins-slave/workspace/ven-resolver-1.3.3-reset-he
> > > > ad
> > > > -
> > > > 29/test/core-it-suite/target/test-classes/mng-4590/${test.file}>
> > > >
> > > >
> > > >
> > > > From the MavenITmng4747JavaAgentUsedByPluginTest:
> > > >
> > > > junit.framework.AssertionFailedError
> > > >
> > > > at
>
> org.apache.maven.it.MavenITmng4747JavaAgentUsedByPluginTest.testit(Mave
>
> > > > n
> > > > ITmng4747JavaAgentUsedByPluginTest.java:63)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[SUMMARY] 3 ITs permanently fail on Linux / Maven Core

Robert Scholte-8
In reply to this post by Tibor Digana
Here's a very short summary and conclusion about what happened the last  
period.

Half July some integration tests of Maven started to fail, but not always  
(it good happen that a certain JDK would succeed again with the next run).  
At that time we only noticed:
- it failed only on the Linux nodes of Jenkins.
- the failing tests were using MAVEN_OPTS

It took quite a while, but in the end Hervé discovered the root cause:
- On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file  
that set the value of MAVEN_OPTS

INFRA discovered that it was the incubator-shardingsphere that created  
this file when building there project.

We've provided a PR for this project together with an explanation. This PR  
has been accepted and INFRA has removed the .mavenrc files.

Over the past weeks a lot of changes has been done to the code for  
investigation.
All changes that were done to change the behavior related to environments  
variables and mavenrc file should and will be reverted.
Mavens master branch was stable, so to ensure the right things are still  
tested the need to be restored (some of them did indeed test the  
MAVEN_OPTS explicitly as issued in its JIRA).
In general we don't change ITs, we only extend then and provide a range to  
control for which Maven versions the test is valid.

Details and discussions can be found on the slack channel.

thanks,
Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core

Robert Scholte-8
No, there's no need for that and it would over-complicate the scripts.
mavenrc can be handy, but only at INFRA level.

As said, we were able to fix it on the project-level. The project simply  
wasn't aware of
- side effects and impact of creating this file on Jenkins
- the existence of .mvn/jvm.config

Robert

On Sun, 04 Aug 2019 18:51:52 +0200, Bernd Eckenfels  
<[hidden email]> wrote:

> Do we need a Maven Option to turn of processing of implicit  
> configurations, just like you can turn of shellrc Files?
>
>
> --
> http://bernd.eckenfels.net
>
> ________________________________
> Von: Robert Scholte <[hidden email]>
> Gesendet: Samstag, August 3, 2019 9:12 PM
> An: Maven Developers List
> Betreff: [SUMMARY] 3 ITs permanently fail on Linux / Maven Core
>
> Here's a very short summary and conclusion about what happened the last
> period.
>
> Half July some integration tests of Maven started to fail, but not always
> (it good happen that a certain JDK would succeed again with the next  
> run).
> At that time we only noticed:
> - it failed only on the Linux nodes of Jenkins.
> - the failing tests were using MAVEN_OPTS
>
> It took quite a while, but in the end Hervé discovered the root cause:
> - On some nodes there was an unexpected ${user.home}/.m2/.mavenrc file
> that set the value of MAVEN_OPTS
>
> INFRA discovered that it was the incubator-shardingsphere that created
> this file when building there project.
>
> We've provided a PR for this project together with an explanation. This  
> PR
> has been accepted and INFRA has removed the .mavenrc files.
>
> Over the past weeks a lot of changes has been done to the code for
> investigation.
> All changes that were done to change the behavior related to environments
> variables and mavenrc file should and will be reverted.
> Mavens master branch was stable, so to ensure the right things are still
> tested the need to be restored (some of them did indeed test the
> MAVEN_OPTS explicitly as issued in its JIRA).
> In general we don't change ITs, we only extend then and provide a range  
> to
> control for which Maven versions the test is valid.
>
> Details and discussions can be found on the slack channel.
>
> thanks,
> Robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]