Plugin dependency resolution

[Tamás Cservenák]

Howdy,

Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
if they are being removed from the classpath afterwards due to classworld
config". Similarly, there is a thread "maven-site-plugin and
sisu-inject-plexus" discussing that plugin pulls down ancient
plexus-container-default (10+ years ago dropped). And finally, we all know
about "maven downloads the whole internet" maxim :)

Hence, I wanted to check this out. My "experiment" was not about maven
"speed" (whatever you mean by it), but more about it's "bandwidth" usage.

My setup:
- am using (primed) MRM, not interested in bashing Central or measuring my
ISP network speed
- am always nuking local repo (hence, starting state is "get everything
needed for build")
- my test bed project is maven itself
(master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
- build + tests ARE executed, and succeed OK, but I was really interested
in local repository post-build state
- command line am executing from maven checkout root is: rm -r /tmp/repo;
~/bin/maven/apache-maven-XXX/bin/mvn clean install
-Dmaven.repo.local=/tmp/repo

Remarks: times and the whole "experiment' is not scientific, they are just
there for "rough comparison" sake. I did not repeat builds multiple times
(to have some "mean time"), as I was not interested in time, but did repeat
enough times to ensure that local repository state (file count, file sizes)
are consistent.

Results:

maven 3.6.3 (released):
total time 1:45 min (as Maven reports)
total files in local repo: 3743
total bytes in local repo: 162600
count of files having "plexus-container-default" in local repo:  37 (18
POM, 10 JAR)

mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
total time 1:08 min (as Maven reports)
total files in local repo: 3741
total bytes in local repo: 162428
count of files having "plexus-container-default" in local repo:  37 (18
POM, 10 JAR)

so, pretty much the same. Then I went to create a (hacked for now) patch,
as can be seen here:
https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack

results with above patched mvn master:
total time 1:04 min (as Maven reports)
total files in local repo: 2992
total bytes in local repo: 149740
count of files having "plexus-container-default" in local repo: 0

Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
downloads in case of Maven being built.

Is there some issue in JIRA covering this behaviour of Maven (I could not
find any)?

Have fun,
Tamas

[Robert Scholte-8]

Hi Tamas,

based on the number of lines you wrote versus the amount of downloads that are prevented (and storage) I think it is worth adding.
My biggest worry about this solution was about the non exported packages of the exported artifacts. Would such changes have a negative and inconsistent effect?
And I think it would make more sense for Maven users
If those people take a closer look at the downloaded file, they should now wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm using Maven 3.6.3?

AFAIK there's no ticket for it yet, these were just the ideas I could think of regarding the WagonExcluder issue.
Looks like it is time to create it.

thanks,
Robert

On 10-2-2021 09:50:24, Tamás Cservenák <[hidden email]> wrote:
Howdy,

Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
if they are being removed from the classpath afterwards due to classworld
config". Similarly, there is a thread "maven-site-plugin and
sisu-inject-plexus" discussing that plugin pulls down ancient
plexus-container-default (10+ years ago dropped). And finally, we all know
about "maven downloads the whole internet" maxim :)

Hence, I wanted to check this out. My "experiment" was not about maven
"speed" (whatever you mean by it), but more about it's "bandwidth" usage.

My setup:
- am using (primed) MRM, not interested in bashing Central or measuring my
ISP network speed
- am always nuking local repo (hence, starting state is "get everything
needed for build")
- my test bed project is maven itself
(master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
- build + tests ARE executed, and succeed OK, but I was really interested
in local repository post-build state
- command line am executing from maven checkout root is: rm -r /tmp/repo;
~/bin/maven/apache-maven-XXX/bin/mvn clean install
-Dmaven.repo.local=/tmp/repo

Remarks: times and the whole "experiment' is not scientific, they are just
there for "rough comparison" sake. I did not repeat builds multiple times
(to have some "mean time"), as I was not interested in time, but did repeat
enough times to ensure that local repository state (file count, file sizes)
are consistent.

Results:

maven 3.6.3 (released):
total time 1:45 min (as Maven reports)
total files in local repo: 3743
total bytes in local repo: 162600
count of files having "plexus-container-default" in local repo: 37 (18
POM, 10 JAR)

mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
total time 1:08 min (as Maven reports)
total files in local repo: 3741
total bytes in local repo: 162428
count of files having "plexus-container-default" in local repo: 37 (18
POM, 10 JAR)

so, pretty much the same. Then I went to create a (hacked for now) patch,
as can be seen here:
https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack

results with above patched mvn master:
total time 1:04 min (as Maven reports)
total files in local repo: 2992
total bytes in local repo: 149740
count of files having "plexus-container-default" in local repo: 0

Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
downloads in case of Maven being built.

Is there some issue in JIRA covering this behaviour of Maven (I could not
find any)?

Have fun,
Tamas

[Romain Manni-Bucau]

Hi everyone,

Does it mean we have two action items:

1. make maven plugin plugin warn or fail (with a toggle) if a maven stack
plugin is in scope compile and not provided
2. filter a bit what we "know" (probably using semver in a whitelist of
dependencies)

2 is not perfect from a design point of view but gain looks huge so
probably worth it

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le jeu. 11 févr. 2021 à 10:20, Tamás Cservenák <[hidden email]> a
écrit :

> Yup, "provided" scope would make it, but, as you say, it would require all
> (ours and non-ours) to be "fixed".
>
> Just for example, in tooling we did for Nexus 2, the NX plugin packaging
> (equivalent of maven plugin packaging)
> was ENFORCING that any NX artifact used as NX plugin dependency be declared
> as "provided", otherwise
> it was failing the build.
>
> HTH
> T
>
> On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY <[hidden email]>
> wrote:
>
> > thinking about plexus-utils case: only XPP3 class is shared by Maven core
> > Then I imagine that it that dependency is filtered at runtime, many calls
> > from
> > plugins to other features of that lib will fail...
> > If you don't beat at me, I'll try to test tonight
> >
> > Another idea: would marking libraries as "provided" in plugins just do
> the
> > job? Of course, each plugin would require to apply this best practice,
> > which
> > will take longer time...
> > need to test also...
> >
> > Regards,
> >
> > Hervé
> >
> > Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > > Hi Robert,
> > >
> > > I agree with you: is a really small change.
> > > Re non-exported packages: will take a look into those a bit more, my
> POC
> > > was just at "artifact level"... (but given Maven toss away downloaded
> > > plugin dependency, is plugin today able to use core artifact's
> > non-exported
> > > package?)
> > > Yes, several maven bits, several plexus bits, and so on, it was just
> > always
> > > annoying me :)
> > > Yup, will create a JIRA.
> > >
> > > Still, I'd really love to have more feedback, have people try the patch
> > on
> > > more projects than I did (maven itself and one another) ;)
> > >
> > > HTH
> > > Tamas
> > >
> > > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <[hidden email]>
> > >
> > > wrote:
> > > > Hi Tamas,
> > > >
> > > > based on the number of lines you wrote versus the amount of downloads
> > that
> > > > are prevented (and storage) I think it is worth adding.
> > > > My biggest worry about this solution was about the non exported
> > packages
> > > > of the exported artifacts. Would such changes have a negative and
> > > > inconsistent effect?
> > > > And I think it would make more sense for Maven users
> > > > If those people take a closer look at the downloaded file, they
> should
> > now
> > > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm
> > using
> > > > Maven 3.6.3?
> > > >
> > > > AFAIK there's no ticket for it yet, these were just the ideas I could
> > > > think of regarding the WagonExcluder issue.
> > > > Looks like it is time to create it.
> > > >
> > > > thanks,
> > > > Robert
> > > >
> > > > On 10-2-2021 09:50:24, Tamás Cservenák <[hidden email]> wrote:
> > > > Howdy,
> > > >
> > > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> > download,
> > > > if they are being removed from the classpath afterwards due to
> > classworld
> > > > config". Similarly, there is a thread "maven-site-plugin and
> > > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > > plexus-container-default (10+ years ago dropped). And finally, we all
> > know
> > > > about "maven downloads the whole internet" maxim :)
> > > >
> > > > Hence, I wanted to check this out. My "experiment" was not about
> maven
> > > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> > usage.
> > > >
> > > > My setup:
> > > > - am using (primed) MRM, not interested in bashing Central or
> > measuring my
> > > > ISP network speed
> > > > - am always nuking local repo (hence, starting state is "get
> everything
> > > > needed for build")
> > > > - my test bed project is maven itself
> > > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > > - build + tests ARE executed, and succeed OK, but I was really
> > interested
> > > > in local repository post-build state
> > > > - command line am executing from maven checkout root is: rm -r
> > /tmp/repo;
> > > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > > -Dmaven.repo.local=/tmp/repo
> > > >
> > > > Remarks: times and the whole "experiment' is not scientific, they are
> > just
> > > > there for "rough comparison" sake. I did not repeat builds multiple
> > times
> > > > (to have some "mean time"), as I was not interested in time, but did
> > > > repeat
> > > > enough times to ensure that local repository state (file count, file
> > > > sizes)
> > > > are consistent.
> > > >
> > > > Results:
> > > >
> > > > maven 3.6.3 (released):
> > > > total time 1:45 min (as Maven reports)
> > > > total files in local repo: 3743
> > > > total bytes in local repo: 162600
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > > total time 1:08 min (as Maven reports)
> > > > total files in local repo: 3741
> > > > total bytes in local repo: 162428
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > so, pretty much the same. Then I went to create a (hacked for now)
> > patch,
> > > > as can be seen here:
> > > >
> > > >
> >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > > -hack
> > > >
> > > > results with above patched mvn master:
> > > > total time 1:04 min (as Maven reports)
> > > > total files in local repo: 2992
> > > > total bytes in local repo: 149740
> > > > count of files having "plexus-container-default" in local repo: 0
> > > >
> > > > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB
> > byte
> > > > downloads in case of Maven being built.
> > > >
> > > > Is there some issue in JIRA covering this behaviour of Maven (I could
> > not
> > > > find any)?
> > > >
> > > > Have fun,
> > > > Tamas
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>