Maven builds out of control

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Maven builds out of control

gmcdonald
Hi Maven folks,

Over 800 builds in the Jenkins queue right now, most of them Maven - and
its been above 500 , mostly Maven for over 12 hours now - Please take a
look and see what is going on ASAP, or I might have to cancel all Maven
jobs to allow other projects to proceed

--

*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Reply | Threaded
Open this post in threaded view
|

Re: Maven builds out of control

rfscholte
Looking at it.
It seems like most are waiting for a Windows executor, but these are all taken too.
Need to investigate a bit more, but it looks like all branches are triggered for some reason.
That's weird, because we didn't touch are shared Jenkins scripts

Robert
On 21-7-2020 18:22:12, Gavin McDonald <[hidden email]> wrote:
Hi Maven folks,

Over 800 builds in the Jenkins queue right now, most of them Maven - and
its been above 500 , mostly Maven for over 12 hours now - Please take a
look and see what is going on ASAP, or I might have to cancel all Maven
jobs to allow other projects to proceed

--

*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Reply | Threaded
Open this post in threaded view
|

Re: Maven builds out of control

gmcdonald
Hi!

On Wed, Jul 22, 2020 at 10:29 AM Robert Scholte <[hidden email]>
wrote:

>
>
> *Robert Scholte:*
> That's very unlikely. I expect that Jenkins itself got an update of that
> plugin and that our MultiBranch setup picked up that change.
>

hm, we don't normally do out of band plugin upgrades unless its' necessary
, I'll investigate this end, I have added a plugin or two
recently, and this may have come in as a dependant upgrade, but fairly sure
I did not do any plugins on the 21st.

Gav...


>
>
>
> >
> >
> > This caused new configuration for *all* our branches and triggered new
> > builds.
> > The Ubuntu builds were often ready, but a lot were still waiting for
> > Windows Executors.
> >
>
> Ack
>
>
> >
> > Most of these builds are quite short so under normal circumstances there
> > shouldn't be any issues.
> > However, I already noticed we're having way too much branches, so either
> > we'll clean these up or remove/disable its JenkinsFile, so it won't be
> > triggered anymore.
> >
>
> That would be awesome, things are looking good at the moment, thanks for
> looking into it
>
> Gav...
>
>
> >
> > thanks,
> > Robert
> >
> > On 21-7-2020 19:49:13, Robert Scholte wrote:
> > Looking at it.
> > It seems like most are waiting for a Windows executor, but these are all
> > taken too.
> > Need to investigate a bit more, but it looks like all branches are
> > triggered for some reason.
> > That's weird, because we didn't touch are shared Jenkins scripts
> >
> > Robert
> >
> > On 21-7-2020 18:22:12, Gavin McDonald wrote:
> > Hi Maven folks,
> >
> > Over 800 builds in the Jenkins queue right now, most of them Maven - and
> > its been above 500 , mostly Maven for over 12 hours now - Please take a
> > look and see what is going on ASAP, or I might have to cancel all Maven
> > jobs to allow other projects to proceed
> >
> > --
> >
> > *Gavin McDonald*
> > Systems Administrator
> > ASF Infrastructure Team
> >
> >
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
>
>

--

*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Maven builds out of control

Arnaud Héritier
In reply to this post by gmcdonald
Thanks for the feedback Gavin, it makes a lot of sense to me.
The new infrastructure is a traditional one (ie not hosted on Kubernetes) ?

On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <[hidden email]>
wrote:

> Hi,
>
> On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <[hidden email]> wrote:
>
> > +infra
> >
> > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <[hidden email]>
> > wrote:
> >
> >> If we are consuming too much resources couldn't we have a dedicated
> >> Jenkins
> >>
> >> controller and/or limit our agents capacities to be the one ones to be
> >>
> >> impacted by these problems instead of all the users or this Jenkins
> >>
> >> controller ?
> >>
> >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> >>
> >> https://jenkins-ccos.apache.org/job/masters/
> >>
> >> What are the rules to have a dedicated one ?
> >>
> >
> All of those projects got their own master as they are contributing their
> own nodes and not using the ASF provided ones
>
> HTH
>
>
> >
> >>
> >>
> >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <[hidden email]>
> >> wrote:
> >>
> >>
> >>
> >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> >>
> >> > > Furthermore, we need to clean up unused branches.
> >>
> >> > > What about removing branches older than 1yo?
> >>
> >> > > Then older than 6months?
> >>
> >> >
> >>
> >> > I consider branches which have been merged obsolete, but there is no
> >>
> >> > hard rule that an unmerged branch is obsolete. It may prepresent an
> >>
> >> > unfished state or something which was not agreed on. Dropping work
> does
> >>
> >> > not feel right. It needs to be decided on a case-by-case approach.
> >>
> >> > I have done this recently for Wagon and Resolver.
> >>
> >> >
> >>
> >> > Michael
> >>
> >> >
> >>
> >> > ---------------------------------------------------------------------
> >>
> >> > To unsubscribe, e-mail: [hidden email]
> >>
> >> > For additional commands, e-mail: [hidden email]
> >>
> >> >
> >>
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> Arnaud Héritier
> >>
> >> Twitter/Skype : aheritier
> >>
> >> --
> > Olivier Lamy
> > http://twitter.com/olamy | http://linkedin.com/in/olamy
> >
>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
>


--
Arnaud Héritier
Twitter/Skype : aheritier
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Maven builds out of control

Arnaud Héritier
No I agree, it's a long term solution and orthogonal on cleaning our
branches/builds and limiting our usages

If these are static agents the best we could do at the CI level is to
restrict the usage of our builds to a limited number of agents
https://docs.cloudbees.com/docs/cloudbees-ci/latest/traditional-secure-guide/folders-plus#_controlled_agents


On Mon, Aug 17, 2020 at 10:55 AM Olivier Lamy <[hidden email]> wrote:

> traditional slaves
> I have a goal to help moving to kubernetes but this will not fix this
> problem :)
>
>
> On Mon, 17 Aug 2020 at 16:46, Arnaud Héritier <[hidden email]> wrote:
>
> > Thanks for the feedback Gavin, it makes a lot of sense to me.
> > The new infrastructure is a traditional one (ie not hosted on
> Kubernetes) ?
> >
> > On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <[hidden email]>
> wrote:
> > >
> > > > +infra
> > > >
> > > > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <[hidden email]
> >
> > > > wrote:
> > > >
> > > >> If we are consuming too much resources couldn't we have a dedicated
> > > >> Jenkins
> > > >>
> > > >> controller and/or limit our agents capacities to be the one ones to
> be
> > > >>
> > > >> impacted by these problems instead of all the users or this Jenkins
> > > >>
> > > >> controller ?
> > > >>
> > > >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> > > >>
> > > >> https://jenkins-ccos.apache.org/job/masters/
> > > >>
> > > >> What are the rules to have a dedicated one ?
> > > >>
> > > >
> > > All of those projects got their own master as they are contributing
> their
> > > own nodes and not using the ASF provided ones
> > >
> > > HTH
> > >
> > >
> > > >
> > > >>
> > > >>
> > > >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <
> [hidden email]>
> > > >> wrote:
> > > >>
> > > >>
> > > >>
> > > >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> > > >>
> > > >> > > Furthermore, we need to clean up unused branches.
> > > >>
> > > >> > > What about removing branches older than 1yo?
> > > >>
> > > >> > > Then older than 6months?
> > > >>
> > > >> >
> > > >>
> > > >> > I consider branches which have been merged obsolete, but there is
> no
> > > >>
> > > >> > hard rule that an unmerged branch is obsolete. It may prepresent
> an
> > > >>
> > > >> > unfished state or something which was not agreed on. Dropping work
> > > does
> > > >>
> > > >> > not feel right. It needs to be decided on a case-by-case approach.
> > > >>
> > > >> > I have done this recently for Wagon and Resolver.
> > > >>
> > > >> >
> > > >>
> > > >> > Michael
> > > >>
> > > >> >
> > > >>
> > > >> >
> > ---------------------------------------------------------------------
> > > >>
> > > >> > To unsubscribe, e-mail: [hidden email]
> > > >>
> > > >> > For additional commands, e-mail: [hidden email]
> > > >>
> > > >> >
> > > >>
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Arnaud Héritier
> > > >>
> > > >> Twitter/Skype : aheritier
> > > >>
> > > >> --
> > > > Olivier Lamy
> > > > http://twitter.com/olamy | http://linkedin.com/in/olamy
> > > >
> > >
> > >
> > > --
> > >
> > > *Gavin McDonald*
> > > Systems Administrator
> > > ASF Infrastructure Team
> > >
> >
> >
> > --
> > Arnaud Héritier
> > Twitter/Skype : aheritier
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>


--
Arnaud Héritier
Twitter/Skype : aheritier