Re: Reactor safe/rally points

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Reactor safe/rally points

Robert Scholte-3
One thing I'd like to change is that if a module is a dependency it must really behave like one.  It should not be the MavenProject, that should only be used for building *that* project. It should act as if it was downloaded via the artifact resolver
Robert

Verzonden vanaf mijn Samsung Galaxy-smartphone.
-------- Oorspronkelijk bericht --------Van: Stephen Connolly <[hidden email]> Datum: 05-11-17  18:57  (GMT+01:00) Aan: Maven Developers List <[hidden email]> Onderwerp: Reactor safe/rally points
There are two sets of problems that, assuming we want to fix, both need
some way to rally a concurrent multimodule build at.

1. There is the shade like class of problems, where a plugin wants to
modify the effective transitive dependencies of a module.

2. There is the extensibility class of problems, where people want to build
a plugin and consume the same plugin in one reactor.

Now in both cases, I think we can solve a subset of these problems by
introducing a concept of a rally / safe point in the build plan.

The mutable model can be solved for reducing scope/ removing
dependencies... could possibly also allow adding dependencies, provided the
dependencies are not part of the reactor. The proviso is that the plugin
goal would need to be marked as a “mutation” goal, and the execution of all
“mutation” goals for a module defines a safe-point. All downstream modules
that declare a dependency must be blocked from execution until the
safe-point is reached.

Discussion points:
* how do we handle a build request that will never reach the safe-point, eg
`mvn test` will not reach package, and hence will not rub shade... should
we say: fine, no safepoint in build plan, so all good... should we say:
oops escalate the module with a safe point to `package`?
* while technically possible to *add* ex-reactor dependencies (if
ex-reactor they cannot affect the build plan) it could be a bad idea for
build requests that wouldn’t reach the safepoint, is that reason enough to
reject addition?

The extensibility issue is actually an important case. Users want features
like scripting because they need something custom. Scripting brings back
imperative builds, which IMHO Maven views as an anti-pattern.

So what is the problem people are trying to solve? They want a plugin for
just one project. Right now you cannot because we cannot instantiate the
full build plan until we can inspect the plugin metadata... but until we
build the plugin we cannot get the metadata.

With a general plugin, I think there is not much we can do. For example a
plugin redefining a lifecycle might actually mean that the build plan will
be different for modules using that plugin as an extension.

But if the plugin is just defining some goals and those goals are just
bound to phases in the pre-existing life cycles... again, it seems we can
allow the build to proceed

Discussion points:
* how to handle `mvn test`. If the plugin is only bound to later phases, we
are good... but what if the plugin is bound to an early phase? Do we just
bomb out and say, sorry you need to invoke at least package? Do we fall
back to local repo? Do we upscale the build plan for the plugin to package
* how to handle `mvn site`... in this case we wouldn’t be producing the
plugin in the reactor anyway... do we bomb out and say “you must do `mvn
package site`“? Do we modify the build plan? Do we use local repo?
* we could be smarter and use the build plan safepoint as a staging point
and allow lifecycles and other things... though obviously more work
* what about invoking a plugin goal from the CLI directly... do we ask for
`mvn package foo:manchu`? For this to work, foo:manchu would have to be an
aggregator goal, otherwise it could be an execution request on earlier
modules by a later module? Could we delay the build plan instantiation
until after the plugin is built? Could we just require the plugin be served
from the local repository and revalidate the build plan after the new
plugin build is available?

I think these are two improvements that could be made *without* changes to
the modelVersion and they would have a real and meaningful improvement for
our users... as such I would like us to consider (as optional goals) for
Maven 4.0.0

Wdyt?
--
Sent from my phone
Reply | Threaded
Open this post in threaded view
|

Re: Reactor safe/rally points

stephenconnolly
Thinking some more, this might be something we could leverage for
incremental builds.

If we save state at the end of each phase (last modified time stamps, hash
of dependencies, etc, also attached artifacts, additional source/resource
roots etc) then, on subsequent builds, if the state file is present and
remains valid then we can skip preceding phases and start that module from
that phase.

In the ideal case that could mean we do nothing with the module as nothing
changed.

On Sun 5 Nov 2017 at 17:57, Stephen Connolly <
[hidden email]> wrote:

> There are two sets of problems that, assuming we want to fix, both need
> some way to rally a concurrent multimodule build at.
>
> 1. There is the shade like class of problems, where a plugin wants to
> modify the effective transitive dependencies of a module.
>
> 2. There is the extensibility class of problems, where people want to
> build a plugin and consume the same plugin in one reactor.
>
> Now in both cases, I think we can solve a subset of these problems by
> introducing a concept of a rally / safe point in the build plan.
>
> The mutable model can be solved for reducing scope/ removing
> dependencies... could possibly also allow adding dependencies, provided the
> dependencies are not part of the reactor. The proviso is that the plugin
> goal would need to be marked as a “mutation” goal, and the execution of all
> “mutation” goals for a module defines a safe-point. All downstream modules
> that declare a dependency must be blocked from execution until the
> safe-point is reached.
>
> Discussion points:
> * how do we handle a build request that will never reach the safe-point,
> eg `mvn test` will not reach package, and hence will not rub shade...
> should we say: fine, no safepoint in build plan, so all good... should we
> say: oops escalate the module with a safe point to `package`?
> * while technically possible to *add* ex-reactor dependencies (if
> ex-reactor they cannot affect the build plan) it could be a bad idea for
> build requests that wouldn’t reach the safepoint, is that reason enough to
> reject addition?
>
> The extensibility issue is actually an important case. Users want features
> like scripting because they need something custom. Scripting brings back
> imperative builds, which IMHO Maven views as an anti-pattern.
>
> So what is the problem people are trying to solve? They want a plugin for
> just one project. Right now you cannot because we cannot instantiate the
> full build plan until we can inspect the plugin metadata... but until we
> build the plugin we cannot get the metadata.
>
> With a general plugin, I think there is not much we can do. For example a
> plugin redefining a lifecycle might actually mean that the build plan will
> be different for modules using that plugin as an extension.
>
> But if the plugin is just defining some goals and those goals are just
> bound to phases in the pre-existing life cycles... again, it seems we can
> allow the build to proceed
>
> Discussion points:
> * how to handle `mvn test`. If the plugin is only bound to later phases,
> we are good... but what if the plugin is bound to an early phase? Do we
> just bomb out and say, sorry you need to invoke at least package? Do we
> fall back to local repo? Do we upscale the build plan for the plugin to
> package
> * how to handle `mvn site`... in this case we wouldn’t be producing the
> plugin in the reactor anyway... do we bomb out and say “you must do `mvn
> package site`“? Do we modify the build plan? Do we use local repo?
> * we could be smarter and use the build plan safepoint as a staging point
> and allow lifecycles and other things... though obviously more work
> * what about invoking a plugin goal from the CLI directly... do we ask for
> `mvn package foo:manchu`? For this to work, foo:manchu would have to be an
> aggregator goal, otherwise it could be an execution request on earlier
> modules by a later module? Could we delay the build plan instantiation
> until after the plugin is built? Could we just require the plugin be served
> from the local repository and revalidate the build plan after the new
> plugin build is available?
>
> I think these are two improvements that could be made *without* changes to
> the modelVersion and they would have a real and meaningful improvement for
> our users... as such I would like us to consider (as optional goals) for
> Maven 4.0.0
>
> Wdyt?
> --
> Sent from my phone
>
--
Sent from my phone
Reply | Threaded
Open this post in threaded view
|

Re: Reactor safe/rally points

Chas Honton
In reply to this post by Robert Scholte-3
So doesn’t this require a new directive such as ‘<compile transitive=false”> scope or similar regardless of safe/rally points?   And to provide backwards compatibility, there must be a split between the legacy consumer view of the pom and the new producer view of the pom?
 

> On Nov 5, 2017, at 11:30 PM, Stephen Connolly <[hidden email]> wrote:
>
>> In the first case, the module creating a shaded jar can mark its
>> constituent jars as optional; preventing the transitive dependencies.
>
>
> Ha! That does not do what you think it does.
>
> Optional dependencies are still there but only as a version constraint that
> gets applied if something else brings the dependency into the tree in as a
> non-optional dependency.
>
> Normally you need a complex deep tree to see the effect (as most people use
> version hints rather than specifications... and with hints, closest wins,
> whereas specifications are globally merged)
>
> Provided scope is more correct, but shade doesn’t operate on provided
> scope... and anyway iirc that also pushes the constraint as transitive.
>
> With shade, the dependency is removed.

Reply | Threaded
Open this post in threaded view
|

Re: Reactor safe/rally points

stephenconnolly
On Tue 7 Nov 2017 at 01:18, Charles Honton <[hidden email]> wrote:

> So doesn’t this require a new directive such as ‘<compile
> transitive=false”> scope or similar regardless of safe/rally points?


No. The plugin would have its descriptor metadata for the goal contain a
flag

Then the plugin is responsible for deciding what dependencies to remove or
lower the scope of

 And to provide backwards compatibility, there must be a split between the
> legacy consumer view of the pom and the new producer view of the pom?


When the dependencies are removed, the pom that gets installed is the
dependency reduced pom.

Shade already does that, but in Maven 3.3.x the model in-memory was made
immutable so now you cannot shade an artifacts and consume it correctly
within the same reactor without having to maintain a set of <exclusions>
tags at every point of use. Outside the reactor use is ok... but then
people who create flyweight local aggregator projects can hit the bug
without knowing it.


>
> > On Nov 5, 2017, at 11:30 PM, Stephen Connolly <
> [hidden email]> wrote:
> >
> >> In the first case, the module creating a shaded jar can mark its
> >> constituent jars as optional; preventing the transitive dependencies.
> >
> >
> > Ha! That does not do what you think it does.
> >
> > Optional dependencies are still there but only as a version constraint
> that
> > gets applied if something else brings the dependency into the tree in as
> a
> > non-optional dependency.
> >
> > Normally you need a complex deep tree to see the effect (as most people
> use
> > version hints rather than specifications... and with hints, closest wins,
> > whereas specifications are globally merged)
> >
> > Provided scope is more correct, but shade doesn’t operate on provided
> > scope... and anyway iirc that also pushes the constraint as transitive.
> >
> > With shade, the dependency is removed.
>
> --
Sent from my phone