Maven moving to the next level: the build/consumer pom

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Maven moving to the next level: the build/consumer pom

rfscholte
Hi,

One of my long standing wishes has made it to the master branch of Maven: the support for build/consumer pom.
With this we can finally start improving the pom without breaking the Maven eco system.

Up until now the pom.xml has been distributed (installed/deployed) as is to both local and remote repositories.
The good thing is that is it fast and there is no magic.
However, it sometimes implies adding redundant information and it also blocks any chance of improvement for Maven

The challenge is to make it possible to have an locally an improved "build" pom while distributing a model 4.0.0 compatible "consumer" pom.

The whole architecture of Maven was built upon an immutable pom.xml, so it took a while to split this, but I managed to solve this.
And to confirm that it works, some transformations have been added for the next Maven release
The local pom it still model 4.0.0 compatible, but some redundant elements are not required anymore.
- in case the <parent/> is located at its relativePath (default: ../pom.xml), the version can be removed. groupId and artifactId are still required to ensure it is being matched with the right parent.

- dependencies that are part of the reactor don't need a version anymore
These are implemented steps to get from the file model to the raw model, where the required versions are added.

When distributing the pom, the previous transformations are done, but also:
- cifriendly placeholders in versions (${sha1}, ${revision}, ${changelist}) will be resolved.
- <modules> from <project> will be removed
- <relativePath> from <parent> will be removed
These cleanups are context aware, if they appear in some configuration, they won't be touched.
One of our integration-tests[1] demonstrates how new poms in a multimodule project might look like.

Even though the latter steps look quite small (removing elements with relative paths), it should give us enough feedback about the whole process.

The status is that it is ready to be embedded in supporting tools like IDEs.
We should give them time to work on this and share feedback.
It might require some adjustments in Maven to improve user experience.

In the meantime we need to work on plugins that will have impact by these changes.
Most significant is are signing maven plugins such as the maven-gpg-plugin, which needs to work with the distributed pom instead of the local pom.
Also all packaging plugin that can include the pom.xml and pom.properties in their archive should switch to the distributed pom.
The maven-shade-plugin was marked as well, but at first glance this looks fine.
In the end all our plugins must be verified, just to be sure.
So there's enough to work on.

In general I avoid giving timelines about how fast a new release will be available.
Due to the overhead, the small amount of available time of the few volunteers working on Maven, I prefer to have a worth set of changes.
In this case the impact of the changes can be huge, and I want to have enough faith that we won't introduce irreversible mistakes.
Don't expect a new official release in the 3 next months, however we might have alpha or beta releases.

There is a wiki page that explains this topic in more detail[2]
It is still a draft, as there are still parts where we need to reach consensus.
This page is intended as a base for discussions by Maven developers, users and related projects, such as IDEs, Repository Managers, CI Servers, etc.

Looking forward for any feedback,

Robert Scholte
Apache Maven project

[1] https://github.com/apache/maven-integration-testing/tree/master/core-it-suite/src/test/resources/mng-6656-buildconsumer
[2] https://cwiki.apache.org/confluence/display/MAVEN/Build+vs+Consumer+POM
Reply | Threaded
Open this post in threaded view
|

Re: Maven moving to the next level: the build/consumer pom

Matthieu BROUILLARD-3
Hi Robert,

congrats this looks like a great achievement.

I presume that the consumer pom means the end of the flatten plugin or is
there still some benefit in using flatten plugin?

> dependencies that are part of the reactor don't need a version anymore
What is meant here ? Is it that in a multimodule project a dependency from
a module to another of the same multimodule does not need the version ? For
example when having an api and impl in the same multimodule, the impl will
not require the version of the api?

> These are implemented steps to get from the file model to the raw model,
where the required versions are added.
Are those replacements still part of the pom loading by ModelProcessor?
I'll give it a try especially to see the compliance with my existing core
extensions.

Matthieu Brouillard

On Mon, Jun 22, 2020 at 11:18 PM Robert Scholte <[hidden email]>
wrote:

> Hi,
>
> One of my long standing wishes has made it to the master branch of Maven:
> the support for build/consumer pom.
> With this we can finally start improving the pom without breaking the
> Maven eco system.
>
> Up until now the pom.xml has been distributed (installed/deployed) as is
> to both local and remote repositories.
> The good thing is that is it fast and there is no magic.
> However, it sometimes implies adding redundant information and it also
> blocks any chance of improvement for Maven
>
> The challenge is to make it possible to have an locally an improved
> "build" pom while distributing a model 4.0.0 compatible "consumer" pom.
>
> The whole architecture of Maven was built upon an immutable pom.xml, so it
> took a while to split this, but I managed to solve this.
> And to confirm that it works, some transformations have been added for the
> next Maven release
> The local pom it still model 4.0.0 compatible, but some redundant elements
> are not required anymore.
> - in case the <parent/> is located at its relativePath (default:
> ../pom.xml), the version can be removed. groupId and artifactId are still
> required to ensure it is being matched with the right parent.
>
> - dependencies that are part of the reactor don't need a version anymore
> These are implemented steps to get from the file model to the raw model,
> where the required versions are added.
>
> When distributing the pom, the previous transformations are done, but also:
> - cifriendly placeholders in versions (${sha1}, ${revision},
> ${changelist}) will be resolved.
> - <modules> from <project> will be removed
> - <relativePath> from <parent> will be removed
> These cleanups are context aware, if they appear in some configuration,
> they won't be touched.
> One of our integration-tests[1] demonstrates how new poms in a multimodule
> project might look like.
>
> Even though the latter steps look quite small (removing elements with
> relative paths), it should give us enough feedback about the whole process.
>
> The status is that it is ready to be embedded in supporting tools like
> IDEs.
> We should give them time to work on this and share feedback.
> It might require some adjustments in Maven to improve user experience.
>
> In the meantime we need to work on plugins that will have impact by these
> changes.
> Most significant is are signing maven plugins such as the
> maven-gpg-plugin, which needs to work with the distributed pom instead of
> the local pom.
> Also all packaging plugin that can include the pom.xml and pom.properties
> in their archive should switch to the distributed pom.
> The maven-shade-plugin was marked as well, but at first glance this looks
> fine.
> In the end all our plugins must be verified, just to be sure.
> So there's enough to work on.
>
> In general I avoid giving timelines about how fast a new release will be
> available.
> Due to the overhead, the small amount of available time of the few
> volunteers working on Maven, I prefer to have a worth set of changes.
> In this case the impact of the changes can be huge, and I want to have
> enough faith that we won't introduce irreversible mistakes.
> Don't expect a new official release in the 3 next months, however we might
> have alpha or beta releases.
>
> There is a wiki page that explains this topic in more detail[2]
> It is still a draft, as there are still parts where we need to reach
> consensus.
> This page is intended as a base for discussions by Maven developers, users
> and related projects, such as IDEs, Repository Managers, CI Servers, etc.
>
> Looking forward for any feedback,
>
> Robert Scholte
> Apache Maven project
>
> [1]
> https://github.com/apache/maven-integration-testing/tree/master/core-it-suite/src/test/resources/mng-6656-buildconsumer
> [2]
> https://cwiki.apache.org/confluence/display/MAVEN/Build+vs+Consumer+POM
Reply | Threaded
Open this post in threaded view
|

Re: Maven moving to the next level: the build/consumer pom

Mark Derricutt
Love the work here - I'll definitely be keen on trying this out and see how it interacts with our tiles-maven-plugin.

Regarding this "reactor" bit tho - just because a module is in the reactor, doesn't always mean it has the same parent - the version of the dependency could be elided if the parent version matched AND that dependency is part of the reactor.

However, I have seem people use reactors purely as build-coordinators, and not always used. I'd much rather seeing them stay - esp. if using version ranges ( we force [] ranges for instance ).

Mark

On 24/06/20, 9:20 AM, "Matthieu BROUILLARD" <[hidden email]> wrote:

    > dependencies that are part of the reactor don't need a version anymore
    What is meant here ? Is it that in a multimodule project a dependency from
    a module to another of the same multimodule does not need the version ? For
    example when having an api and impl in the same multimodule, the impl will
    not require the version of the api?


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Maven moving to the next level: the build/consumer pom

Elliotte Rusty Harold
In reply to this post by rfscholte
The XML handling code in this work is running into some common design
flaws in the JDK XML classes, in particular in the Factories class.

First, you probably shouldn't use SAXParserFactory or SAXParser at
all. They were added to the JDK to support SAX 1.0, and haven't been
needed for since SAX 2 was released (about 20 years ago). Modern code
should use XMLReader and XMLReaderFactory instead.

If you do stick with SAXParser, it's important to call
setNamespaceAware(true), since it's false by default, a legacy of its
use with the non-namespace supporting SAX 1.

On Mon, Jun 22, 2020 at 5:18 PM Robert Scholte <[hidden email]> wrote:

>
> Hi,
>
> One of my long standing wishes has made it to the master branch of Maven: the support for build/consumer pom.
> With this we can finally start improving the pom without breaking the Maven eco system.
>
> Up until now the pom.xml has been distributed (installed/deployed) as is to both local and remote repositories.
> The good thing is that is it fast and there is no magic.
> However, it sometimes implies adding redundant information and it also blocks any chance of improvement for Maven
>
> The challenge is to make it possible to have an locally an improved "build" pom while distributing a model 4.0.0 compatible "consumer" pom.
>
> The whole architecture of Maven was built upon an immutable pom.xml, so it took a while to split this, but I managed to solve this.
> And to confirm that it works, some transformations have been added for the next Maven release
> The local pom it still model 4.0.0 compatible, but some redundant elements are not required anymore.
> - in case the <parent/> is located at its relativePath (default: ../pom.xml), the version can be removed. groupId and artifactId are still required to ensure it is being matched with the right parent.
>
> - dependencies that are part of the reactor don't need a version anymore
> These are implemented steps to get from the file model to the raw model, where the required versions are added.
>
> When distributing the pom, the previous transformations are done, but also:
> - cifriendly placeholders in versions (${sha1}, ${revision}, ${changelist}) will be resolved.
> - <modules> from <project> will be removed
> - <relativePath> from <parent> will be removed
> These cleanups are context aware, if they appear in some configuration, they won't be touched.
> One of our integration-tests[1] demonstrates how new poms in a multimodule project might look like.
>
> Even though the latter steps look quite small (removing elements with relative paths), it should give us enough feedback about the whole process.
>
> The status is that it is ready to be embedded in supporting tools like IDEs.
> We should give them time to work on this and share feedback.
> It might require some adjustments in Maven to improve user experience.
>
> In the meantime we need to work on plugins that will have impact by these changes.
> Most significant is are signing maven plugins such as the maven-gpg-plugin, which needs to work with the distributed pom instead of the local pom.
> Also all packaging plugin that can include the pom.xml and pom.properties in their archive should switch to the distributed pom.
> The maven-shade-plugin was marked as well, but at first glance this looks fine.
> In the end all our plugins must be verified, just to be sure.
> So there's enough to work on.
>
> In general I avoid giving timelines about how fast a new release will be available.
> Due to the overhead, the small amount of available time of the few volunteers working on Maven, I prefer to have a worth set of changes.
> In this case the impact of the changes can be huge, and I want to have enough faith that we won't introduce irreversible mistakes.
> Don't expect a new official release in the 3 next months, however we might have alpha or beta releases.
>
> There is a wiki page that explains this topic in more detail[2]
> It is still a draft, as there are still parts where we need to reach consensus.
> This page is intended as a base for discussions by Maven developers, users and related projects, such as IDEs, Repository Managers, CI Servers, etc.
>
> Looking forward for any feedback,
>
> Robert Scholte
> Apache Maven project
>
> [1] https://github.com/apache/maven-integration-testing/tree/master/core-it-suite/src/test/resources/mng-6656-buildconsumer
> [2] https://cwiki.apache.org/confluence/display/MAVEN/Build+vs+Consumer+POM



--
Elliotte Rusty Harold
[hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Maven moving to the next level: the build/consumer pom

Anton Vodonosov
In reply to this post by rfscholte
Can this work also allow arbitrary property expression in a module <version>?

Currently, this practice is discouraged because the deployed pom with property
expression is meaningless.
The flatten-maven-plugin can produce correct poms for deployment,
with all properties resolved; despite maven prints the hardcoded warning
 "'version' contains an expression but should be a constant".

Links on the topic:
http://mail-archives.apache.org/mod_mbox/maven-users/201201.mbox/%3CCA+nPnMyb_W4Ldrg5zTn5NX3F2kcrPXDefS4gjzz3OE0cUy1rXw@...%3E
https://issues.apache.org/jira/browse/MNG-4715
https://stackoverflow.com/questions/19123013/maven-version-with-a-property/19123130

Why I'm interested in property expressions for versions is because I want to generate
versions automatically as a hash code of all sources and dependencies.

And also skip building of modules whose has-version has not changed.

That way the large multi-module project can vary significantly improve build speed
 and reduce the artifact storage consumption. Similar to what build caches
give for gradle and bazel, only a bit simplified, less granular - caching done
on the level of final artifacts, but not for intermediate outputs
 (compiled file, generated resources).

If would be very convenient if maven-flatten-plugin become unnecessary
and maven supported property expressions in version spec out of box.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Maven moving to the next level: the build/consumer pom

Hervé BOUTEMY
In reply to this post by rfscholte
I just created a "buildconsumer" branch in Doxia [1] to have a live example of the new simplified build POM that Maven 3.7.0 will allow [2]

As expected, this branch can't build with Maven 3.6.3, but can with Maven 3.7.0-SNAPSHOT.
And in my favorite IDE, as expected, dependency resolution does not work because of missing info in pom.xml (which is now a build pom, with removed tags, then was not a valid POM until now)

What is expected is IDE maintainers to check what they need to do at IDE level to support these new POMs that only build with Maven 3.7.0-SNAPSHOT.

Regards,

Hervé

[1] https://github.com/apache/maven-doxia/tree/buildconsumer

[2] https://github.com/apache/maven-doxia/commit/15e3de1a97bf48a394cb566783cac851c0728d98

Le samedi 4 juillet 2020, 07:35:37 CEST Jaroslav Tulach a écrit :

> Hello Robert,
> I am not sure how to deal with your announcement and given no reaction on
> the dev@netbeans mailing list, I am probably not alone. Can you formulate
> your issue as a bug report? E.g. have you tried to use your new Maven with
> NetBeans and did you face a problem? Having steps to reproduce would make
> it more real for the NetBeans community to take action.
>
> -jt
>
> po 22. 6. 2020 v 23:18 odesílatel Robert Scholte <[hidden email]>
>
> napsal:
> > Hi,
> >
> > One of my long standing wishes has made it to the master branch of Maven:
> > the support for build/consumer pom.
> > With this we can finally start improving the pom without breaking the
> > Maven eco system.
> >
> > Up until now the pom.xml has been distributed (installed/deployed) as is
> > to both local and remote repositories.
> > The good thing is that is it fast and there is no magic.
> > However, it sometimes implies adding redundant information and it also
> > blocks any chance of improvement for Maven
> >
> > The challenge is to make it possible to have an locally an improved
> > "build" pom while distributing a model 4.0.0 compatible "consumer" pom.
> >
> > The whole architecture of Maven was built upon an immutable pom.xml, so it
> > took a while to split this, but I managed to solve this.
> > And to confirm that it works, some transformations have been added for the
> > next Maven release
> > The local pom it still model 4.0.0 compatible, but some redundant elements
> > are not required anymore.
> > - in case the <parent/> is located at its relativePath (default:
> > ../pom.xml), the version can be removed. groupId and artifactId are still
> > required to ensure it is being matched with the right parent.
> >
> > - dependencies that are part of the reactor don't need a version anymore
> > These are implemented steps to get from the file model to the raw model,
> > where the required versions are added.
> >
> > When distributing the pom, the previous transformations are done, but
> > also:
> > - cifriendly placeholders in versions (${sha1}, ${revision},
> > ${changelist}) will be resolved.
> > - <modules> from <project> will be removed
> > - <relativePath> from <parent> will be removed
> > These cleanups are context aware, if they appear in some configuration,
> > they won't be touched.
> > One of our integration-tests[1] demonstrates how new poms in a multimodule
> > project might look like.
> >
> > Even though the latter steps look quite small (removing elements with
> > relative paths), it should give us enough feedback about the whole
> > process.
> >
> > The status is that it is ready to be embedded in supporting tools like
> > IDEs.
> > We should give them time to work on this and share feedback.
> > It might require some adjustments in Maven to improve user experience.
> >
> > In the meantime we need to work on plugins that will have impact by these
> > changes.
> > Most significant is are signing maven plugins such as the
> > maven-gpg-plugin, which needs to work with the distributed pom instead of
> > the local pom.
> > Also all packaging plugin that can include the pom.xml and pom.properties
> > in their archive should switch to the distributed pom.
> > The maven-shade-plugin was marked as well, but at first glance this looks
> > fine.
> > In the end all our plugins must be verified, just to be sure.
> > So there's enough to work on.
> >
> > In general I avoid giving timelines about how fast a new release will be
> > available.
> > Due to the overhead, the small amount of available time of the few
> > volunteers working on Maven, I prefer to have a worth set of changes.
> > In this case the impact of the changes can be huge, and I want to have
> > enough faith that we won't introduce irreversible mistakes.
> > Don't expect a new official release in the 3 next months, however we might
> > have alpha or beta releases.
> >
> > There is a wiki page that explains this topic in more detail[2]
> > It is still a draft, as there are still parts where we need to reach
> > consensus.
> > This page is intended as a base for discussions by Maven developers, users
> > and related projects, such as IDEs, Repository Managers, CI Servers, etc.
> >
> > Looking forward for any feedback,
> >
> > Robert Scholte
> > Apache Maven project
> >
> > [1]
> > https://github.com/apache/maven-integration-testing/tree/master/core-it-su
> > ite/src/test/resources/mng-6656-buildconsumer [2]
> > https://cwiki.apache.org/confluence/display/MAVEN/Build+vs+Consumer+POM





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]