Jenkins instability

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Jenkins instability

Elliotte Rusty Harold
We now have 12 red and 1 yellow job in Jenkins:

https://builds.apache.org/view/M-R/view/Maven/job/dist-tool-plugin/site/dist-tool-master-jobs.html

Furthermore the number seems to be increasing over time. This block
progress in multiple plugins.

I haven't looked into each one, but the ones I've spot checked have
all shown "no failures" and typically fail in some late message such
as

[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

That is, these appear to be build node failures rather than actual
test failures.

Is there anything that can be done systematically to prevent this?
E.g. is there some way we can fix the root cause of the flakiness on
build nodes? Or can we work around it by configuring Jenkins to accept
"no failures" as success even if the script returns exit code 1?

--
Elliotte Rusty Harold
[hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Jenkins instability

rfscholte
https://builds.apache.org/job/dist-tool-plugin/site/dist-tool-master-jobs.html


Fixing the Maven mapping restored most broken builds.
Looks like the following projects have had a bad commit and need to be fixed:
maven-gpg-plugin
maven-resources-plugin
maven-scm-publish-plugin


thanks,
Robert
On 8-6-2020 23:26:29, Robert Scholte <[hidden email]> wrote:
I noticed that our 3.5.x versions weren't the latest.
It could be that INFRA started removing the older versions.
I've updated them all to the latest.

Robert
On 8-6-2020 21:25:03, Hervé BOUTEMY <[hidden email]> wrote:
when the node involved has memory issues, opening an INFRA issue is the way to
go, like for example https://issues.apache.org/jira/browse/INFRA-20392

when it's an issue about mvn not being recognized on Win nodes, perhaps we
should do the same; even if it's not obvious if the node fails systematically
or not

Regards,

Hervé

Le lundi 8 juin 2020, 20:34:03 CEST Robert Scholte a écrit :

> Hi Elliotte,
>
> great to see there are more looking at the master jobs.
> As you might notice, it is not always test failures.
>
> Here are some other important links:
> - Open Blue Ocean:
> e.g. https://builds.apache.org/blue/organizations/jenkins/maven-box%2Fmaven
> -antrun-plugin/detail/master/69/pipeline
>
> Look for the failing step.
>
> - Pipeline Steps (similar, just a different view):
> e.g builds.apache.org/job/maven-box/job/maven-antrun-plugin/job/master/69/f
> lowGraphTable/
>
> Again, look for the failing step
>
> Based on the cause, you either run the build again or exclude broken nodes
> in https://github.com/apache/maven-jenkins-env/blob/master/vars/jenkinsEnv.
> groovy
>
> thanks,
> Robert
> On 8-6-2020 15:31:13, Elliotte Rusty Harold wrote:
> We now have 12 red and 1 yellow job in Jenkins:
>
> https://builds.apache.org/view/M-R/view/Maven/job/dist-tool-plugin/site/dist
> -tool-master-jobs.html
>
> Furthermore the number seems to be increasing over time. This block
> progress in multiple plugins.
>
> I haven't looked into each one, but the ones I've spot checked have
> all shown "no failures" and typically fail in some late message such
> as
>
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] End of Pipeline
> ERROR: script returned exit code 1
> Finished: FAILURE
>
> That is, these appear to be build node failures rather than actual
> test failures.
>
> Is there anything that can be done systematically to prevent this?
> E.g. is there some way we can fix the root cause of the flakiness on
> build nodes? Or can we work around it by configuring Jenkins to accept
> "no failures" as success even if the script returns exit code 1?
>
> --
> Elliotte Rusty Harold
> [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]