Re: Maven memory consumption issue

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Maven memory consumption issue

Hervé BOUTEMY
Hi,

Interesting.
I created MNG-6571 to track this issue: please move the memory dump on this
issue.
Then I created MNG-6571 branch [2] with a cache on VersionRange: this class is
currently immutable, then caching it is trivial. I don't know if it's the best
location, or if there are not other locations where a cache would make sense,
but it was an easy and safe try.

Can you build this branch for yourself and share a new memory dump, please?

Regards,

Hervé

[1] https://issues.apache.org/jira/browse/MNG-6571

[2] https://github.com/apache/maven/tree/MNG-6571

Le mercredi 23 janvier 2019, 10:10:40 CET Dameron, Yann a écrit :

> Hi,
>
>
>
> I'm working on a huge project with plenty modules and we are facing a memory
> consumption issue. We observed it when Maven computes the reactor and then
> during the build itself.
>
>
>
> Quick summary of our situation (as we can't share the POMs)
>
>   *   We have a dedicated POM to manage all versions (3rd parties and our
> own modules). This POM is split in 5 sub poms to organize them a bit
> (apache, netbeans, xxx, ...)
>
> On this POM structure, we declare 3049 <dependency>...</dependency>.
>
>   *   We have a parent pom. This one is used by all our 1628 modules. It
> defines the plugins versions and import the dependency management
> (explained above). *   We have 1628 modules which are using our parent pom
> (and then inherit the dependency management imported by the parent) *   We
> have an aggregator POMs structure which is responsible to include all the
> modules. Root aggregator is the one on which the full rebuild is triggered.
>
>
>
> When Maven computes the reactor, it consumes a huge amount of memory (8 - 12
> Gb)
>
>
>
> I made a memory dump just before the end of reactor computation. I attached
> a screenshot in MNG-5661<https://issues.apache.org/jira/browse/MNG-5661> as
> I feel it's related, please remove it if it's not appropriated. On this
> screenshot we clearly see from where come from our memory issues:
> https://issues.apache.org/jira/secure/attachment/12955925/Maven-Reactor-Dum
> p.png
>
>
>
> The number of Dependency instance seems to be related to our structure. All
> our projects import our big dependency management (inherited from the
> parent) so we have 1628 x 3049 => ~ 5 000 000 instances.
>
> This is also approximately the number of MavenProject, DefaultArtifact,
> VersionRange, etc... so I feel that the memory consumption comes from the
> fact that Maven doesn't detect such cases and duplicate everything.
>
>
>
> Is Maven optimized to detect such cases? It would be great to ensure that
> Dependency (and others) instances are not duplicated.
>
>
>
> Thanks,
> Yann
>
> Information in this e-mail and any attachments is confidential, and may not
> be copied or used by anyone other than the addressee, nor disclosed to any
> third party without our permission. There is no intention to create any
> legally binding contract or other binding commitment through the use of
> this electronic communication unless it is issued in accordance with the
> Experian Limited standard terms and conditions of purchase or other express
> written agreement between Experian Limited and the recipient. Although
> Experian has taken reasonable steps to ensure that this communication and
> any attachments are free from computer viruses, you are advised to take
> your own steps to ensure that they are actually virus free.
>
> Experian Ltd is authorised and regulated by the Financial Conduct Authority.
> Companies Act information: Registered name: Experian Limited. Registered
> office: The Sir John Peace Building, Experian Way, NG2 Business Park,
> Nottingham, NG80 1ZZ, United Kingdom. Place of registration: England and
> Wales. Registered number: 653331.
>
> Vos informations nominatives sont exploitées par la société Scorex S.A.M.
> (Groupe Experian)  dans le cadre du traitement ayant pour finalité "Gestion
> de la messagerie électronique professionnelle". Conformément à la loi n°
> 1.165, modifiée, vous disposez d'un droit d'accès, de rectification et de
> suppression en écrivant au Secrétariat Général de la société Scorex, sise,
> 2, rue de la Lüjerneta à Monaco (98000).
>
> Your personal information is used by Scorex S.A.M. (Experian Group) as part
> of the processing for the purpose of "Professional e-mail management". In
> accordance with the law n ° 1.165, modified, you have a right of access,
> correction and suppression by writing to the General Secretariat of the
> company, located, 2, rue de la Lüjerneta in Monaco (98000).





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Maven memory consumption issue

Hervé BOUTEMY
as discusses in the Jira issue, the cache on Version Range is simple and very
efficient

can someone review and second for merge to master?

Regards,

Hervé

Le mercredi 23 janvier 2019, 11:07:43 CET Hervé BOUTEMY a écrit :

> Hi,
>
> Interesting.
> I created MNG-6571 to track this issue: please move the memory dump on this
> issue.
> Then I created MNG-6571 branch [2] with a cache on VersionRange: this class
> is currently immutable, then caching it is trivial. I don't know if it's
> the best location, or if there are not other locations where a cache would
> make sense, but it was an easy and safe try.
>
> Can you build this branch for yourself and share a new memory dump, please?
>
> Regards,
>
> Hervé
>
> [1] https://issues.apache.org/jira/browse/MNG-6571
>
> [2] https://github.com/apache/maven/tree/MNG-6571
>
> Le mercredi 23 janvier 2019, 10:10:40 CET Dameron, Yann a écrit :
> > Hi,
> >
> >
> >
> > I'm working on a huge project with plenty modules and we are facing a
> > memory consumption issue. We observed it when Maven computes the reactor
> > and then during the build itself.
> >
> >
> >
> > Quick summary of our situation (as we can't share the POMs)
> >
> >   *   We have a dedicated POM to manage all versions (3rd parties and our
> >
> > own modules). This POM is split in 5 sub poms to organize them a bit
> > (apache, netbeans, xxx, ...)
> >
> > On this POM structure, we declare 3049 <dependency>...</dependency>.
> >
> >   *   We have a parent pom. This one is used by all our 1628 modules. It
> >
> > defines the plugins versions and import the dependency management
> > (explained above). *   We have 1628 modules which are using our parent pom
> > (and then inherit the dependency management imported by the parent) *   We
> > have an aggregator POMs structure which is responsible to include all the
> > modules. Root aggregator is the one on which the full rebuild is
> > triggered.
> >
> >
> >
> > When Maven computes the reactor, it consumes a huge amount of memory (8 -
> > 12 Gb)
> >
> >
> >
> > I made a memory dump just before the end of reactor computation. I
> > attached
> > a screenshot in MNG-5661<https://issues.apache.org/jira/browse/MNG-5661>
> > as
> > I feel it's related, please remove it if it's not appropriated. On this
> > screenshot we clearly see from where come from our memory issues:
> > https://issues.apache.org/jira/secure/attachment/12955925/Maven-Reactor-Du
> > m
> > p.png
> >
> >
> >
> > The number of Dependency instance seems to be related to our structure.
> > All
> > our projects import our big dependency management (inherited from the
> > parent) so we have 1628 x 3049 => ~ 5 000 000 instances.
> >
> > This is also approximately the number of MavenProject, DefaultArtifact,
> > VersionRange, etc... so I feel that the memory consumption comes from the
> > fact that Maven doesn't detect such cases and duplicate everything.
> >
> >
> >
> > Is Maven optimized to detect such cases? It would be great to ensure that
> > Dependency (and others) instances are not duplicated.
> >
> >
> >
> > Thanks,
> > Yann
> >
> > Information in this e-mail and any attachments is confidential, and may
> > not
> > be copied or used by anyone other than the addressee, nor disclosed to any
> > third party without our permission. There is no intention to create any
> > legally binding contract or other binding commitment through the use of
> > this electronic communication unless it is issued in accordance with the
> > Experian Limited standard terms and conditions of purchase or other
> > express
> > written agreement between Experian Limited and the recipient. Although
> > Experian has taken reasonable steps to ensure that this communication and
> > any attachments are free from computer viruses, you are advised to take
> > your own steps to ensure that they are actually virus free.
> >
> > Experian Ltd is authorised and regulated by the Financial Conduct
> > Authority. Companies Act information: Registered name: Experian Limited.
> > Registered office: The Sir John Peace Building, Experian Way, NG2
> > Business Park, Nottingham, NG80 1ZZ, United Kingdom. Place of
> > registration: England and Wales. Registered number: 653331.
> >
> > Vos informations nominatives sont exploitées par la société Scorex S.A.M.
> > (Groupe Experian)  dans le cadre du traitement ayant pour finalité
> > "Gestion
> > de la messagerie électronique professionnelle". Conformément à la loi n°
> > 1.165, modifiée, vous disposez d'un droit d'accès, de rectification et de
> > suppression en écrivant au Secrétariat Général de la société Scorex, sise,
> > 2, rue de la Lüjerneta à Monaco (98000).
> >
> > Your personal information is used by Scorex S.A.M. (Experian Group) as
> > part
> > of the processing for the purpose of "Professional e-mail management". In
> > accordance with the law n ° 1.165, modified, you have a right of access,
> > correction and suppression by writing to the General Secretariat of the
> > company, located, 2, rue de la Lüjerneta in Monaco (98000).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]