Re: Jenkins instability

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins instability

Hervé BOUTEMY
when the node involved has memory issues, opening an INFRA issue is the way to
go, like for example https://issues.apache.org/jira/browse/INFRA-20392

when it's an issue about mvn not being recognized on Win nodes, perhaps we
should do the same; even if it's not obvious if the node fails systematically
or not

Regards,

Hervé

Le lundi 8 juin 2020, 20:34:03 CEST Robert Scholte a écrit :

> Hi Elliotte,
>
> great to see there are more looking at the master jobs.
> As you might notice, it is not always test failures.
>
> Here are some other important links:
> - Open Blue Ocean:
> e.g. https://builds.apache.org/blue/organizations/jenkins/maven-box%2Fmaven
> -antrun-plugin/detail/master/69/pipeline
>
> Look for the failing step.
>
> - Pipeline Steps (similar, just a different view):
> e.g builds.apache.org/job/maven-box/job/maven-antrun-plugin/job/master/69/f
> lowGraphTable/
>
> Again, look for the failing step
>
> Based on the cause, you either run the build again or exclude broken nodes
> in https://github.com/apache/maven-jenkins-env/blob/master/vars/jenkinsEnv.
> groovy
>
> thanks,
> Robert
> On 8-6-2020 15:31:13, Elliotte Rusty Harold <[hidden email]> wrote:
> We now have 12 red and 1 yellow job in Jenkins:
>
> https://builds.apache.org/view/M-R/view/Maven/job/dist-tool-plugin/site/dist
> -tool-master-jobs.html
>
> Furthermore the number seems to be increasing over time. This block
> progress in multiple plugins.
>
> I haven't looked into each one, but the ones I've spot checked have
> all shown "no failures" and typically fail in some late message such
> as
>
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] End of Pipeline
> ERROR: script returned exit code 1
> Finished: FAILURE
>
> That is, these appear to be build node failures rather than actual
> test failures.
>
> Is there anything that can be done systematically to prevent this?
> E.g. is there some way we can fix the root cause of the flakiness on
> build nodes? Or can we work around it by configuring Jenkins to accept
> "no failures" as success even if the script returns exit code 1?
>
> --
> Elliotte Rusty Harold
> [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]