Re: Plugin dependency resolution

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Plugin dependency resolution

Tamás Cservenák
Yup, "provided" scope would make it, but, as you say, it would require all
(ours and non-ours) to be "fixed".

Just for example, in tooling we did for Nexus 2, the NX plugin packaging
(equivalent of maven plugin packaging)
was ENFORCING that any NX artifact used as NX plugin dependency be declared
as "provided", otherwise
it was failing the build.

HTH
T

On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY <[hidden email]> wrote:

> thinking about plexus-utils case: only XPP3 class is shared by Maven core
> Then I imagine that it that dependency is filtered at runtime, many calls
> from
> plugins to other features of that lib will fail...
> If you don't beat at me, I'll try to test tonight
>
> Another idea: would marking libraries as "provided" in plugins just do the
> job? Of course, each plugin would require to apply this best practice,
> which
> will take longer time...
> need to test also...
>
> Regards,
>
> Hervé
>
> Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > Hi Robert,
> >
> > I agree with you: is a really small change.
> > Re non-exported packages: will take a look into those a bit more, my POC
> > was just at "artifact level"... (but given Maven toss away downloaded
> > plugin dependency, is plugin today able to use core artifact's
> non-exported
> > package?)
> > Yes, several maven bits, several plexus bits, and so on, it was just
> always
> > annoying me :)
> > Yup, will create a JIRA.
> >
> > Still, I'd really love to have more feedback, have people try the patch
> on
> > more projects than I did (maven itself and one another) ;)
> >
> > HTH
> > Tamas
> >
> > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <[hidden email]>
> >
> > wrote:
> > > Hi Tamas,
> > >
> > > based on the number of lines you wrote versus the amount of downloads
> that
> > > are prevented (and storage) I think it is worth adding.
> > > My biggest worry about this solution was about the non exported
> packages
> > > of the exported artifacts. Would such changes have a negative and
> > > inconsistent effect?
> > > And I think it would make more sense for Maven users
> > > If those people take a closer look at the downloaded file, they should
> now
> > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while I'm
> using
> > > Maven 3.6.3?
> > >
> > > AFAIK there's no ticket for it yet, these were just the ideas I could
> > > think of regarding the WagonExcluder issue.
> > > Looks like it is time to create it.
> > >
> > > thanks,
> > > Robert
> > >
> > > On 10-2-2021 09:50:24, Tamás Cservenák <[hidden email]> wrote:
> > > Howdy,
> > >
> > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS] MNG-7020
> > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> download,
> > > if they are being removed from the classpath afterwards due to
> classworld
> > > config". Similarly, there is a thread "maven-site-plugin and
> > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > plexus-container-default (10+ years ago dropped). And finally, we all
> know
> > > about "maven downloads the whole internet" maxim :)
> > >
> > > Hence, I wanted to check this out. My "experiment" was not about maven
> > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> usage.
> > >
> > > My setup:
> > > - am using (primed) MRM, not interested in bashing Central or
> measuring my
> > > ISP network speed
> > > - am always nuking local repo (hence, starting state is "get everything
> > > needed for build")
> > > - my test bed project is maven itself
> > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > - build + tests ARE executed, and succeed OK, but I was really
> interested
> > > in local repository post-build state
> > > - command line am executing from maven checkout root is: rm -r
> /tmp/repo;
> > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > -Dmaven.repo.local=/tmp/repo
> > >
> > > Remarks: times and the whole "experiment' is not scientific, they are
> just
> > > there for "rough comparison" sake. I did not repeat builds multiple
> times
> > > (to have some "mean time"), as I was not interested in time, but did
> > > repeat
> > > enough times to ensure that local repository state (file count, file
> > > sizes)
> > > are consistent.
> > >
> > > Results:
> > >
> > > maven 3.6.3 (released):
> > > total time 1:45 min (as Maven reports)
> > > total files in local repo: 3743
> > > total bytes in local repo: 162600
> > > count of files having "plexus-container-default" in local repo: 37 (18
> > > POM, 10 JAR)
> > >
> > > mvn master (4.0.0-alpha-1, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > total time 1:08 min (as Maven reports)
> > > total files in local repo: 3741
> > > total bytes in local repo: 162428
> > > count of files having "plexus-container-default" in local repo: 37 (18
> > > POM, 10 JAR)
> > >
> > > so, pretty much the same. Then I went to create a (hacked for now)
> patch,
> > > as can be seen here:
> > >
> > >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > -hack
> > >
> > > results with above patched mvn master:
> > > total time 1:04 min (as Maven reports)
> > > total files in local repo: 2992
> > > total bytes in local repo: 149740
> > > count of files having "plexus-container-default" in local repo: 0
> > >
> > > Basically patch "cut" almost 1k of unnecessary file downloads, 10+MB
> byte
> > > downloads in case of Maven being built.
> > >
> > > Is there some issue in JIRA covering this behaviour of Maven (I could
> not
> > > find any)?
> > >
> > > Have fun,
> > > Tamas
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Plugin dependency resolution

Tamás Cservenák
Yup,

this can be approached on "two ends": from Maven and from Plugins....

Note: let's name artifacts having GA org.apache.maven:maven-* as "maven
runtime", artifacts being "the maven", in short: MR.

From plugin end, I never understood why MR dependencies are NOT declared as
"provided".
maven-plugin packaged artifacts CANNOT "live" outside of a MR. Is really
the same as "servlet-api" for web apps.

Moreover (am now just opening up a random Apache maven plugin, am not
picking at any of those, just giving an example), look at this:
https://github.com/apache/maven-deploy-plugin/blob/master/pom.xml#L72-L91

The dependencies in this POM are declared using "best practices" (defines
mavenVersion property), BUT let's assume someone does not do it like that.
Hence, it becomes possible that a plugin depends on maven-plugin-api 3.0,
maven-core:3.1 and maven-model:3.2.
Moreover, let's assume that the plugin builds PASSES and is deployed
despite this -- possibly human oversight -- mistake.

My point is: at maven plugin build time
- if MR artifact is NOT provided fail/warn (toggle)
- make sure all MR artifacts are SAME version, otherwise fail/warm (toggle)

And yes, the "filter a bit we know" part is still hazy (at plugin build
time), especially as developers may develop it's maven plugin for Maven
3.0, 3.1, 3.2.... and those all have slightly different MR...

But again, we could draw the line, and make the latest maven-plugin-plugin
raise the bar of "lowest maven version" it can build a plugin for.....
(and for those publish some "BOM" or "inventory", analogy of signature for
animal sniffer?)

And this could be maven-plugin-plugin or maybe even some enforcer rule? (if
there is some like this, am unaware of it, sorry)

HTH
T


On Thu, Feb 11, 2021 at 11:19 AM Romain Manni-Bucau <[hidden email]>
wrote:

> Hi everyone,
>
> Does it mean we have two action items:
>
> 1. make maven plugin plugin warn or fail (with a toggle) if a maven stack
> plugin is in scope compile and not provided
> 2. filter a bit what we "know" (probably using semver in a whitelist of
> dependencies)
>
> 2 is not perfect from a design point of view but gain looks huge so
> probably worth it
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
>
>
> Le jeu. 11 févr. 2021 à 10:20, Tamás Cservenák <[hidden email]> a
> écrit :
>
> > Yup, "provided" scope would make it, but, as you say, it would require
> all
> > (ours and non-ours) to be "fixed".
> >
> > Just for example, in tooling we did for Nexus 2, the NX plugin packaging
> > (equivalent of maven plugin packaging)
> > was ENFORCING that any NX artifact used as NX plugin dependency be
> declared
> > as "provided", otherwise
> > it was failing the build.
> >
> > HTH
> > T
> >
> > On Thu, Feb 11, 2021 at 9:36 AM Hervé BOUTEMY <[hidden email]>
> > wrote:
> >
> > > thinking about plexus-utils case: only XPP3 class is shared by Maven
> core
> > > Then I imagine that it that dependency is filtered at runtime, many
> calls
> > > from
> > > plugins to other features of that lib will fail...
> > > If you don't beat at me, I'll try to test tonight
> > >
> > > Another idea: would marking libraries as "provided" in plugins just do
> > the
> > > job? Of course, each plugin would require to apply this best practice,
> > > which
> > > will take longer time...
> > > need to test also...
> > >
> > > Regards,
> > >
> > > Hervé
> > >
> > > Le jeudi 11 février 2021, 08:41:21 CET Tamás Cservenák a écrit :
> > > > Hi Robert,
> > > >
> > > > I agree with you: is a really small change.
> > > > Re non-exported packages: will take a look into those a bit more, my
> > POC
> > > > was just at "artifact level"... (but given Maven toss away downloaded
> > > > plugin dependency, is plugin today able to use core artifact's
> > > non-exported
> > > > package?)
> > > > Yes, several maven bits, several plexus bits, and so on, it was just
> > > always
> > > > annoying me :)
> > > > Yup, will create a JIRA.
> > > >
> > > > Still, I'd really love to have more feedback, have people try the
> patch
> > > on
> > > > more projects than I did (maven itself and one another) ;)
> > > >
> > > > HTH
> > > > Tamas
> > > >
> > > > On Wed, Feb 10, 2021 at 10:33 PM Robert Scholte <
> [hidden email]>
> > > >
> > > > wrote:
> > > > > Hi Tamas,
> > > > >
> > > > > based on the number of lines you wrote versus the amount of
> downloads
> > > that
> > > > > are prevented (and storage) I think it is worth adding.
> > > > > My biggest worry about this solution was about the non exported
> > > packages
> > > > > of the exported artifacts. Would such changes have a negative and
> > > > > inconsistent effect?
> > > > > And I think it would make more sense for Maven users
> > > > > If those people take a closer look at the downloaded file, they
> > should
> > > now
> > > > > wonder: my is maven-core 2.2.1, 3.0, 3.1.1 etc downloaded, while
> I'm
> > > using
> > > > > Maven 3.6.3?
> > > > >
> > > > > AFAIK there's no ticket for it yet, these were just the ideas I
> could
> > > > > think of regarding the WagonExcluder issue.
> > > > > Looks like it is time to create it.
> > > > >
> > > > > thanks,
> > > > > Robert
> > > > >
> > > > > On 10-2-2021 09:50:24, Tamás Cservenák <[hidden email]>
> wrote:
> > > > > Howdy,
> > > > >
> > > > > Some time ago I stumbled upon Robert S. remark in "[DISCUSS]
> MNG-7020
> > > > > Remove Maven 2 WagonExcluder backward compat code" thread: "Why
> > > download,
> > > > > if they are being removed from the classpath afterwards due to
> > > classworld
> > > > > config". Similarly, there is a thread "maven-site-plugin and
> > > > > sisu-inject-plexus" discussing that plugin pulls down ancient
> > > > > plexus-container-default (10+ years ago dropped). And finally, we
> all
> > > know
> > > > > about "maven downloads the whole internet" maxim :)
> > > > >
> > > > > Hence, I wanted to check this out. My "experiment" was not about
> > maven
> > > > > "speed" (whatever you mean by it), but more about it's "bandwidth"
> > > usage.
> > > > >
> > > > > My setup:
> > > > > - am using (primed) MRM, not interested in bashing Central or
> > > measuring my
> > > > > ISP network speed
> > > > > - am always nuking local repo (hence, starting state is "get
> > everything
> > > > > needed for build")
> > > > > - my test bed project is maven itself
> > > > > (master, ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6)
> > > > > - build + tests ARE executed, and succeed OK, but I was really
> > > interested
> > > > > in local repository post-build state
> > > > > - command line am executing from maven checkout root is: rm -r
> > > /tmp/repo;
> > > > > ~/bin/maven/apache-maven-XXX/bin/mvn clean install
> > > > > -Dmaven.repo.local=/tmp/repo
> > > > >
> > > > > Remarks: times and the whole "experiment' is not scientific, they
> are
> > > just
> > > > > there for "rough comparison" sake. I did not repeat builds multiple
> > > times
> > > > > (to have some "mean time"), as I was not interested in time, but
> did
> > > > > repeat
> > > > > enough times to ensure that local repository state (file count,
> file
> > > > > sizes)
> > > > > are consistent.
> > > > >
> > > > > Results:
> > > > >
> > > > > maven 3.6.3 (released):
> > > > > total time 1:45 min (as Maven reports)
> > > > > total files in local repo: 3743
> > > > > total bytes in local repo: 162600
> > > > > count of files having "plexus-container-default" in local repo: 37
> > (18
> > > > > POM, 10 JAR)
> > > > >
> > > > > mvn master (4.0.0-alpha-1,
> ab20190a1a9fecdc8f85b40e8d03d806f3da4fc6):
> > > > > total time 1:08 min (as Maven reports)
> > > > > total files in local repo: 3741
> > > > > total bytes in local repo: 162428
> > > > > count of files having "plexus-container-default" in local repo: 37
> > (18
> > > > > POM, 10 JAR)
> > > > >
> > > > > so, pretty much the same. Then I went to create a (hacked for now)
> > > patch,
> > > > > as can be seen here:
> > > > >
> > > > >
> > >
> >
> https://github.com/apache/maven/compare/master...cstamas:plugin-resolution
> > > > > -hack
> > > > >
> > > > > results with above patched mvn master:
> > > > > total time 1:04 min (as Maven reports)
> > > > > total files in local repo: 2992
> > > > > total bytes in local repo: 149740
> > > > > count of files having "plexus-container-default" in local repo: 0
> > > > >
> > > > > Basically patch "cut" almost 1k of unnecessary file downloads,
> 10+MB
> > > byte
> > > > > downloads in case of Maven being built.
> > > > >
> > > > > Is there some issue in JIRA covering this behaviour of Maven (I
> could
> > > not
> > > > > find any)?
> > > > >
> > > > > Have fun,
> > > > > Tamas
> > >
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> > >
> >
>