Re: Plugin dependency resolution

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Plugin dependency resolution

Tamás Cservenák
Hi Robert,

I agree with you: is a really small change.
Re non-exported packages: will take a look into those a bit more, my POC
was just at "artifact level"... (but given Maven toss away downloaded
plugin dependency, is plugin today able to use core artifact's non-exported
package?)
Yes, several maven bits, several plexus bits, and so on, it was just always
annoying me :)
Yup, will create a JIRA.

Still, I'd really love to have more feedback, have people try the patch on
more projects than I did (maven itself and one another) ;)

HTH
Tamas

On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <[hidden email]>
wrote:

> Hi Tamas,
>
> based on the number of lines you wrote versus the amount of downloads that
> are prevented (and storage) I think it is worth adding.
> My biggest worry about this solution was about the non exported packages
> of the exported artifacts. Would such changes have a negative and
> inconsistent effect?
> And I think it would make more sense for Maven users
> If those people take a closer look at the downloaded file, they should now
> wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm using
> Maven 3.6.3?
>
> AFAIK there's no ticket for it yet, these were just the ideas I could
> think of regarding the WagonExcluder issue.
> Looks like it is time to create it.
>
> thanks,
> Robert
>
> On 10-2-2021 09:50:24, Tamás Cservenák <[hidden email]> wrote:
> Howdy,
>
> Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
> if they are being removed from the classpath afterwards due to classworld
> config". Similarly, there is a thread "maven-site-plugin and
> sisu-inject-plexus" discussing that plugin pulls down ancient
> plexus-container-default (10+ years ago dropped). And finally, we all know
> about "maven downloads the whole internet" maxim :)
>
> Hence, I wanted to check this out. My "experiment" was not about maven
> "speed" (whatever you mean by it), but more about it's "bandwidth" usage.
>
> My setup:
> - am using (primed) MRM, not interested in bashing Central or measuring my
> ISP network speed
> - am always nuking local repo (hence, starting state is "get everything
> needed for build")
> - my test bed project is maven itself
> (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> - build + tests ARE executed, and succeed OK, but I was really interested
> in local repository post-build state
> - command line am executing from maven checkout root is: rm -r /tmp/repo;
> ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> -Dmaven.repo.local=/tmp/repo
>
> Remarks: times and the whole "experiment' is not scientific, they are just
> there for "rough comparison" sake. I did not repeat builds multiple times
> (to have some "mean time"), as I was not interested in time, but did repeat
> enough times to ensure that local repository state (file count, file sizes)
> are consistent.
>
> Results:
>
> maven 3.6.3 (released):
> total time 1:45 min (as Maven reports)
> total files in local repo: 3743
> total bytes in local repo: 162600
> count of files having "plexus-container-default" in local repo: 37 (18
> POM, 10 JAR)
>
> mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> total time 1:08 min (as Maven reports)
> total files in local repo: 3741
> total bytes in local repo: 162428
> count of files having "plexus-container-default" in local repo: 37 (18
> POM, 10 JAR)
>
> so, pretty much the same. Then I went to create a (hacked for now) patch,
> as can be seen here:
>
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution-hack
>
> results with above patched mvn master:
> total time 1:04 min (as Maven reports)
> total files in local repo: 2992
> total bytes in local repo: 149740
> count of files having "plexus-container-default" in local repo: 0
>
> Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
> downloads in case of Maven being built.
>
> Is there some issue in JIRA covering this behaviour of Maven (I could not
> find any)?
>
> Have fun,
> Tamas
>
Reply | Threaded
Open this post in threaded view
|

Re: Plugin dependency resolution

Hervé BOUTEMY
thinking about plexus-utils case: only XPP3 class is shared by Maven core
Then I imagine that it that dependency is filtered at runtime, many calls from
plugins to other features of that lib will fail...
If you don't beat at me, I'll try to test tonight

Another idea: would marking libraries as "provided" in plugins just do the
job? Of course, each plugin would require to apply this best practice, which
will take longer time...
need to test also...

Regards,

Hervé

Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :

> Hi Robert,
>
> I agree with you: is a really small change.
> Re non-exported packages: will take a look into those a bit more, my POC
> was just at "artifact level"... (but given Maven toss away downloaded
> plugin dependency, is plugin today able to use core artifact's non-exported
> package?)
> Yes, several maven bits, several plexus bits, and so on, it was just always
> annoying me :)
> Yup, will create a JIRA.
>
> Still, I'd really love to have more feedback, have people try the patch on
> more projects than I did (maven itself and one another) ;)
>
> HTH
> Tamas
>
> On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <[hidden email]>
>
> wrote:
> > Hi Tamas,
> >
> > based on the number of lines you wrote versus the amount of downloads that
> > are prevented (and storage) I think it is worth adding.
> > My biggest worry about this solution was about the non exported packages
> > of the exported artifacts. Would such changes have a negative and
> > inconsistent effect?
> > And I think it would make more sense for Maven users
> > If those people take a closer look at the downloaded file, they should now
> > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm using
> > Maven 3.6.3?
> >
> > AFAIK there's no ticket for it yet, these were just the ideas I could
> > think of regarding the WagonExcluder issue.
> > Looks like it is time to create it.
> >
> > thanks,
> > Robert
> >
> > On 10-2-2021 09:50:24, Tamás Cservenák <[hidden email]> wrote:
> > Howdy,
> >
> > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > Remove Maven 2 WagonExcluder backward compat code" thread: "Why download,
> > if they are being removed from the classpath afterwards due to classworld
> > config". Similarly, there is a thread "maven-site-plugin and
> > sisu-inject-plexus" discussing that plugin pulls down ancient
> > plexus-container-default (10+ years ago dropped). And finally, we all know
> > about "maven downloads the whole internet" maxim :)
> >
> > Hence, I wanted to check this out. My "experiment" was not about maven
> > "speed" (whatever you mean by it), but more about it's "bandwidth" usage.
> >
> > My setup:
> > - am using (primed) MRM, not interested in bashing Central or measuring my
> > ISP network speed
> > - am always nuking local repo (hence, starting state is "get everything
> > needed for build")
> > - my test bed project is maven itself
> > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > - build + tests ARE executed, and succeed OK, but I was really interested
> > in local repository post-build state
> > - command line am executing from maven checkout root is: rm -r /tmp/repo;
> > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > -Dmaven.repo.local=/tmp/repo
> >
> > Remarks: times and the whole "experiment' is not scientific, they are just
> > there for "rough comparison" sake. I did not repeat builds multiple times
> > (to have some "mean time"), as I was not interested in time, but did
> > repeat
> > enough times to ensure that local repository state (file count, file
> > sizes)
> > are consistent.
> >
> > Results:
> >
> > maven 3.6.3 (released):
> > total time 1:45 min (as Maven reports)
> > total files in local repo: 3743
> > total bytes in local repo: 162600
> > count of files having "plexus-container-default" in local repo: 37 (18
> > POM, 10 JAR)
> >
> > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > total time 1:08 min (as Maven reports)
> > total files in local repo: 3741
> > total bytes in local repo: 162428
> > count of files having "plexus-container-default" in local repo: 37 (18
> > POM, 10 JAR)
> >
> > so, pretty much the same. Then I went to create a (hacked for now) patch,
> > as can be seen here:
> >
> > https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > -hack
> >
> > results with above patched mvn master:
> > total time 1:04 min (as Maven reports)
> > total files in local repo: 2992
> > total bytes in local repo: 149740
> > count of files having "plexus-container-default" in local repo: 0
> >
> > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB byte
> > downloads in case of Maven being built.
> >
> > Is there some issue in JIRA covering this behaviour of Maven (I could not
> > find any)?
> >
> > Have fun,
> > Tamas





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Plugin dependency resolution

Robert Scholte-8
In the past I've tried to set several maven(core) dependencies of a plugin to provided and it broke!
I didn't understand it and didn't had the time to analyze it.
Just a huge warning that you can expect surprises.

I think we can start thinking of "provided" if we have clear APIs and SPIs defined for Maven, now it is too early.

Robert
On 11-2-2021 11:19:03, Romain Manni-Bucau <[hidden email]> wrote:
Hi everyone,

Does it mean we have two action items:

1. make maven plugin plugin warn or fail (with a toggle) if a maven stack
plugin is in scope compile and not provided
2. filter a bit what we "know" (probably using semver in a whitelist of
dependencies)

2 is not perfect from a design point of view but gain looks huge so
probably worth it

Romain Manni-Bucau
@rmannibucau | Blog
| Old Blog
| Github |
LinkedIn | Book



Le jeu. 11 févr. 2021 à 10:20, Tamás Cservenák a
écrit :

> Yup, "provided" scope would make it, but, as you say, it would require all
> (ours and non-ours) to be "fixed".
>
> Just for example, in tooling we did for Nexus 2, the NX plugin packaging
> (equivalent of maven plugin packaging)
> was ENFORCING that any NX artifact used as NX plugin dependency be declared
> as "provided", otherwise
> it was failing the build.
>
> HTH
> T
>
> On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY
> wrote:
>
> > thinking about plexus-utils case: only XPP3 class is shared by Maven core
> > Then I imagine that it that dependency is filtered at runtime, many calls
> > from
> > plugins to other features of that lib will fail...
> > If you don't beat at me, I'll try to test tonight
> >
> > Another idea: would marking libraries as "provided" in plugins just do
> the
> > job? Of course, each plugin would require to apply this best practice,
> > which
> > will take longer time...
> > need to test also...
> >
> > Regards,
> >
> > Hervé
> >
> > Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > > Hi Robert,
> > >
> > > I agree with you: is a really small change.
> > > Re non-exported packages: will take a look into those a bit more, my
> POC
> > > was just at "artifact level"... (but given Maven toss away downloaded
> > > plugin dependency, is plugin today able to use core artifact's
> > non-exported
> > > package?)
> > > Yes, several maven bits, several plexus bits, and so on, it was just
> > always
> > > annoying me :)
> > > Yup, will create a JIRA.
> > >
> > > Still, I'd really love to have more feedback, have people try the patch
> > on
> > > more projects than I did (maven itself and one another) ;)
> > >
> > > HTH
> > > Tamas
> > >
> > > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte
> > >
> > > wrote:
> > > > Hi Tamas,
> > > >
> > > > based on the number of lines you wrote versus the amount of downloads
> > that
> > > > are prevented (and storage) I think it is worth adding.
> > > > My biggest worry about this solution was about the non exported
> > packages
> > > > of the exported artifacts. Would such changes have a negative and
> > > > inconsistent effect?
> > > > And I think it would make more sense for Maven users
> > > > If those people take a closer look at the downloaded file, they
> should
> > now
> > > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm
> > using
> > > > Maven 3.6.3?
> > > >
> > > > AFAIK there's no ticket for it yet, these were just the ideas I could
> > > > think of regarding the WagonExcluder issue.
> > > > Looks like it is time to create it.
> > > >
> > > > thanks,
> > > > Robert
> > > >
> > > > On 10-2-2021 09:50:24, Tamás Cservenák wrote:
> > > > Howdy,
> > > >
> > > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> > download,
> > > > if they are being removed from the classpath afterwards due to
> > classworld
> > > > config". Similarly, there is a thread "maven-site-plugin and
> > > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > > plexus-container-default (10+ years ago dropped). And finally, we all
> > know
> > > > about "maven downloads the whole internet" maxim :)
> > > >
> > > > Hence, I wanted to check this out. My "experiment" was not about
> maven
> > > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> > usage.
> > > >
> > > > My setup:
> > > > - am using (primed) MRM, not interested in bashing Central or
> > measuring my
> > > > ISP network speed
> > > > - am always nuking local repo (hence, starting state is "get
> everything
> > > > needed for build")
> > > > - my test bed project is maven itself
> > > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > > - build + tests ARE executed, and succeed OK, but I was really
> > interested
> > > > in local repository post-build state
> > > > - command line am executing from maven checkout root is: rm -r
> > /tmp/repo;
> > > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > > -Dmaven.repo.local=/tmp/repo
> > > >
> > > > Remarks: times and the whole "experiment' is not scientific, they are
> > just
> > > > there for "rough comparison" sake. I did not repeat builds multiple
> > times
> > > > (to have some "mean time"), as I was not interested in time, but did
> > > > repeat
> > > > enough times to ensure that local repository state (file count, file
> > > > sizes)
> > > > are consistent.
> > > >
> > > > Results:
> > > >
> > > > maven 3.6.3 (released):
> > > > total time 1:45 min (as Maven reports)
> > > > total files in local repo: 3743
> > > > total bytes in local repo: 162600
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > > total time 1:08 min (as Maven reports)
> > > > total files in local repo: 3741
> > > > total bytes in local repo: 162428
> > > > count of files having "plexus-container-default" in local repo: 37
> (18
> > > > POM, 10 JAR)
> > > >
> > > > so, pretty much the same. Then I went to create a (hacked for now)
> > patch,
> > > > as can be seen here:
> > > >
> > > >
> >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > > -hack
> > > >
> > > > results with above patched mvn master:
> > > > total time 1:04 min (as Maven reports)
> > > > total files in local repo: 2992
> > > > total bytes in local repo: 149740
> > > > count of files having "plexus-container-default" in local repo: 0
> > > >
> > > > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB
> > byte
> > > > downloads in case of Maven being built.
> > > >
> > > > Is there some issue in JIRA covering this behaviour of Maven (I could
> > not
> > > > find any)?
> > > >
> > > > Have fun,
> > > > Tamas
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>