Maven memory consumption issue

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Maven memory consumption issue

Dameron, Yann
Hi,



I'm working on a huge project with plenty modules and we are facing a memory consumption issue. We observed it when Maven computes the reactor and then during the build itself.



Quick summary of our situation (as we can't share the POMs)

  *   We have a dedicated POM to manage all versions (3rd parties and our own modules). This POM is split in 5 sub poms to organize them a bit (apache, netbeans, xxx, ...)

On this POM structure, we declare 3049 <dependency>...</dependency>.

  *   We have a parent pom. This one is used by all our 1628 modules. It defines the plugins versions and import the dependency management (explained above).
  *   We have 1628 modules which are using our parent pom (and then inherit the dependency management imported by the parent)
  *   We have an aggregator POMs structure which is responsible to include all the modules. Root aggregator is the one on which the full rebuild is triggered.



When Maven computes the reactor, it consumes a huge amount of memory (8 - 12 Gb)



I made a memory dump just before the end of reactor computation. I attached a screenshot in MNG-5661<https://issues.apache.org/jira/browse/MNG-5661> as I feel it's related, please remove it if it's not appropriated.
On this screenshot we clearly see from where come from our memory issues: https://issues.apache.org/jira/secure/attachment/12955925/Maven-Reactor-Dump.png



The number of Dependency instance seems to be related to our structure. All our projects import our big dependency management (inherited from the parent) so we have 1628 x 3049 => ~ 5 000 000 instances.

This is also approximately the number of MavenProject, DefaultArtifact, VersionRange, etc... so I feel that the memory consumption comes from the fact that Maven doesn't detect such cases and duplicate everything.



Is Maven optimized to detect such cases? It would be great to ensure that Dependency (and others) instances are not duplicated.



Thanks,
Yann

Information in this e-mail and any attachments is confidential, and may not be copied or used by anyone other than the addressee, nor disclosed to any third party without our permission. There is no intention to create any legally binding contract or other binding commitment through the use of this electronic communication unless it is issued in accordance with the Experian Limited standard terms and conditions of purchase or other express written agreement between Experian Limited and the recipient. Although Experian has taken reasonable steps to ensure that this communication and any attachments are free from computer viruses, you are advised to take your own steps to ensure that they are actually virus free.

Experian Ltd is authorised and regulated by the Financial Conduct Authority.
Companies Act information: Registered name: Experian Limited. Registered office: The Sir John Peace Building, Experian Way, NG2 Business Park, Nottingham, NG80 1ZZ, United Kingdom. Place of registration: England and Wales. Registered number: 653331.

Vos informations nominatives sont exploitées par la société Scorex S.A.M. (Groupe Experian)  dans le cadre du traitement ayant pour finalité "Gestion de la messagerie électronique professionnelle". Conformément à la loi n° 1.165, modifiée, vous disposez d'un droit d'accès, de rectification et de suppression en écrivant au Secrétariat Général de la société Scorex, sise, 2, rue de la Lüjerneta à Monaco (98000).

Your personal information is used by Scorex S.A.M. (Experian Group) as part of the processing for the purpose of "Professional e-mail management". In accordance with the law n ° 1.165, modified, you have a right of access, correction and suppression by writing to the General Secretariat of the company, located, 2, rue de la Lüjerneta in Monaco (98000).