Re: Jenkins instability

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins instability

Robert Scholte-8
Chris fixed INFRA-20417, and with that all other masters branches are back to blue too.
So all is stable again!

Robert
On 12-6-2020 15:27:06, Robert Scholte <[hidden email]> wrote:
https://issues.apache.org/jira/browse/INFRA-20417 [https://issues.apache.org/jira/browse/INFRA-20417] created for maven-gpg-plugin

On 10-6-2020 22:02:42, Robert Scholte <[hidden email]> wrote:
https://builds.apache.org/job/dist-tool-plugin/site/dist-tool-master-jobs.html


Fixing the Maven mapping restored most broken builds.
Looks like the following projects have had a bad commit and need to be fixed:
maven-gpg-plugin
maven-resources-plugin
maven-scm-publish-plugin


thanks,
Robert
On 8-6-2020 23:26:29, Robert Scholte <[hidden email]> wrote:
I noticed that our 3.5.x versions weren't the latest.
It could be that INFRA started removing the older versions.
I've updated them all to the latest.

Robert
On 8-6-2020 21:25:03, Hervé BOUTEMY <[hidden email]> wrote:
when the node involved has memory issues, opening an INFRA issue is the way to
go, like for example https://issues.apache.org/jira/browse/INFRA-20392

when it's an issue about mvn not being recognized on Win nodes, perhaps we
should do the same; even if it's not obvious if the node fails systematically
or not

Regards,

Hervé

Le lundi 8 juin 2020, 20:34:03 CEST Robert Scholte a écrit :

> Hi Elliotte,
>
> great to see there are more looking at the master jobs.
> As you might notice, it is not always test failures.
>
> Here are some other important links:
> - Open Blue Ocean:
> e.g. https://builds.apache.org/blue/organizations/jenkins/maven-box%2Fmaven
> -antrun-plugin/detail/master/69/pipeline
>
> Look for the failing step.
>
> - Pipeline Steps (similar, just a different view):
> e.g builds.apache.org/job/maven-box/job/maven-antrun-plugin/job/master/69/f
> lowGraphTable/
>
> Again, look for the failing step
>
> Based on the cause, you either run the build again or exclude broken nodes
> in https://github.com/apache/maven-jenkins-env/blob/master/vars/jenkinsEnv.
> groovy
>
> thanks,
> Robert
> On 8-6-2020 15:31:13, Elliotte Rusty Harold wrote:
> We now have 12 red and 1 yellow job in Jenkins:
>
> https://builds.apache.org/view/M-R/view/Maven/job/dist-tool-plugin/site/dist
> -tool-master-jobs.html
>
> Furthermore the number seems to be increasing over time. This block
> progress in multiple plugins.
>
> I haven't looked into each one, but the ones I've spot checked have
> all shown "no failures" and typically fail in some late message such
> as
>
> [Pipeline] }
> [Pipeline] // stage
> [Pipeline] End of Pipeline
> ERROR: script returned exit code 1
> Finished: FAILURE
>
> That is, these appear to be build node failures rather than actual
> test failures.
>
> Is there anything that can be done systematically to prevent this?
> E.g. is there some way we can fix the root cause of the flakiness on
> build nodes? Or can we work around it by configuring Jenkins to accept
> "no failures" as success even if the script returns exit code 1?
>
> --
> Elliotte Rusty Harold
> [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Jenkins instability

Hervé BOUTEMY
great work, thank you Robert

Le vendredi 12 juin 2020, 20:00:10 CEST Robert Scholte a écrit :

> Chris fixed INFRA-20417, and with that all other masters branches are back
> to blue too. So all is stable again!
>
> Robert
> On 12-6-2020 15:27:06, Robert Scholte <[hidden email]> wrote:
> https://issues.apache.org/jira/browse/INFRA-20417
> [https://issues.apache.org/jira/browse/INFRA-20417] created for
> maven-gpg-plugin
>
> On 10-6-2020 22:02:42, Robert Scholte <[hidden email]> wrote:
> https://builds.apache.org/job/dist-tool-plugin/site/dist-tool-master-jobs.ht
> ml
>
>
> Fixing the Maven mapping restored most broken builds.
> Looks like the following projects have had a bad commit and need to be
> fixed: maven-gpg-plugin
> maven-resources-plugin
> maven-scm-publish-plugin
>
>
> thanks,
> Robert
> On 8-6-2020 23:26:29, Robert Scholte <[hidden email]> wrote:
> I noticed that our 3.5.x versions weren't the latest.
> It could be that INFRA started removing the older versions.
> I've updated them all to the latest.
>
> Robert
> On 8-6-2020 21:25:03, Hervé BOUTEMY <[hidden email]> wrote:
> when the node involved has memory issues, opening an INFRA issue is the way
> to go, like for example https://issues.apache.org/jira/browse/INFRA-20392
>
> when it's an issue about mvn not being recognized on Win nodes, perhaps we
> should do the same; even if it's not obvious if the node fails
> systematically or not
>
> Regards,
>
> Hervé
>
> Le lundi 8 juin 2020, 20:34:03 CEST Robert Scholte a écrit :
> > Hi Elliotte,
> >
> > great to see there are more looking at the master jobs.
> > As you might notice, it is not always test failures.
> >
> > Here are some other important links:
> > - Open Blue Ocean:
> > e.g.
> > https://builds.apache.org/blue/organizations/jenkins/maven-box%2Fmaven
> > -antrun-plugin/detail/master/69/pipeline
> >
> > Look for the failing step.
> >
> > - Pipeline Steps (similar, just a different view):
> > e.g
> > builds.apache.org/job/maven-box/job/maven-antrun-plugin/job/master/69/f
> > lowGraphTable/
> >
> > Again, look for the failing step
> >
> > Based on the cause, you either run the build again or exclude broken nodes
> > in
> > https://github.com/apache/maven-jenkins-env/blob/master/vars/jenkinsEnv.
> > groovy
> >
> > thanks,
> > Robert
> > On 8-6-2020 15:31:13, Elliotte Rusty Harold wrote:
> > We now have 12 red and 1 yellow job in Jenkins:
> >
> > https://builds.apache.org/view/M-R/view/Maven/job/dist-tool-plugin/site/di
> > st -tool-master-jobs.html
> >
> > Furthermore the number seems to be increasing over time. This block
> > progress in multiple plugins.
> >
> > I haven't looked into each one, but the ones I've spot checked have
> > all shown "no failures" and typically fail in some late message such
> > as
> >
> > [Pipeline] }
> > [Pipeline] // stage
> > [Pipeline] End of Pipeline
> > ERROR: script returned exit code 1
> > Finished: FAILURE
> >
> > That is, these appear to be build node failures rather than actual
> > test failures.
> >
> > Is there anything that can be done systematically to prevent this?
> > E.g. is there some way we can fix the root cause of the flakiness on
> > build nodes? Or can we work around it by configuring Jenkins to accept
> > "no failures" as success even if the script returns exit code 1?
> >
> > --
> > Elliotte Rusty Harold
> > [hidden email]
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]