Reactor safe/rally points

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Reactor safe/rally points

stephenconnolly
There are two sets of problems that, assuming we want to fix, both need
some way to rally a concurrent multimodule build at.

1. There is the shade like class of problems, where a plugin wants to
modify the effective transitive dependencies of a module.

2. There is the extensibility class of problems, where people want to build
a plugin and consume the same plugin in one reactor.

Now in both cases, I think we can solve a subset of these problems by
introducing a concept of a rally / safe point in the build plan.

The mutable model can be solved for reducing scope/ removing
dependencies... could possibly also allow adding dependencies, provided the
dependencies are not part of the reactor. The proviso is that the plugin
goal would need to be marked as a “mutation” goal, and the execution of all
“mutation” goals for a module defines a safe-point. All downstream modules
that declare a dependency must be blocked from execution until the
safe-point is reached.

Discussion points:
* how do we handle a build request that will never reach the safe-point, eg
`mvn test` will not reach package, and hence will not rub shade... should
we say: fine, no safepoint in build plan, so all good... should we say:
oops escalate the module with a safe point to `package`?
* while technically possible to *add* ex-reactor dependencies (if
ex-reactor they cannot affect the build plan) it could be a bad idea for
build requests that wouldn’t reach the safepoint, is that reason enough to
reject addition?

The extensibility issue is actually an important case. Users want features
like scripting because they need something custom. Scripting brings back
imperative builds, which IMHO Maven views as an anti-pattern.

So what is the problem people are trying to solve? They want a plugin for
just one project. Right now you cannot because we cannot instantiate the
full build plan until we can inspect the plugin metadata... but until we
build the plugin we cannot get the metadata.

With a general plugin, I think there is not much we can do. For example a
plugin redefining a lifecycle might actually mean that the build plan will
be different for modules using that plugin as an extension.

But if the plugin is just defining some goals and those goals are just
bound to phases in the pre-existing life cycles... again, it seems we can
allow the build to proceed

Discussion points:
* how to handle `mvn test`. If the plugin is only bound to later phases, we
are good... but what if the plugin is bound to an early phase? Do we just
bomb out and say, sorry you need to invoke at least package? Do we fall
back to local repo? Do we upscale the build plan for the plugin to package
* how to handle `mvn site`... in this case we wouldn’t be producing the
plugin in the reactor anyway... do we bomb out and say “you must do `mvn
package site`“? Do we modify the build plan? Do we use local repo?
* we could be smarter and use the build plan safepoint as a staging point
and allow lifecycles and other things... though obviously more work
* what about invoking a plugin goal from the CLI directly... do we ask for
`mvn package foo:manchu`? For this to work, foo:manchu would have to be an
aggregator goal, otherwise it could be an execution request on earlier
modules by a later module? Could we delay the build plan instantiation
until after the plugin is built? Could we just require the plugin be served
from the local repository and revalidate the build plan after the new
plugin build is available?

I think these are two improvements that could be made *without* changes to
the modelVersion and they would have a real and meaningful improvement for
our users... as such I would like us to consider (as optional goals) for
Maven 4.0.0

Wdyt?
--
Sent from my phone
Reply | Threaded
Open this post in threaded view
|

Re: Reactor safe/rally points

Robert Scholte-3
One thing I'd like to change is that if a module is a dependency it must really behave like one.  It should not be the MavenProject, that should only be used for building *that* project. It should act as if it was downloaded via the artifact resolver
Robert

Verzonden vanaf mijn Samsung Galaxy-smartphone.
-------- Oorspronkelijk bericht --------Van: Stephen Connolly <[hidden email]> Datum: 05-11-17  18:57  (GMT+01:00) Aan: Maven Developers List <[hidden email]> Onderwerp: Reactor safe/rally points
There are two sets of problems that, assuming we want to fix, both need
some way to rally a concurrent multimodule build at.

1. There is the shade like class of problems, where a plugin wants to
modify the effective transitive dependencies of a module.

2. There is the extensibility class of problems, where people want to build
a plugin and consume the same plugin in one reactor.

Now in both cases, I think we can solve a subset of these problems by
introducing a concept of a rally / safe point in the build plan.

The mutable model can be solved for reducing scope/ removing
dependencies... could possibly also allow adding dependencies, provided the
dependencies are not part of the reactor. The proviso is that the plugin
goal would need to be marked as a “mutation” goal, and the execution of all
“mutation” goals for a module defines a safe-point. All downstream modules
that declare a dependency must be blocked from execution until the
safe-point is reached.

Discussion points:
* how do we handle a build request that will never reach the safe-point, eg
`mvn test` will not reach package, and hence will not rub shade... should
we say: fine, no safepoint in build plan, so all good... should we say:
oops escalate the module with a safe point to `package`?
* while technically possible to *add* ex-reactor dependencies (if
ex-reactor they cannot affect the build plan) it could be a bad idea for
build requests that wouldn’t reach the safepoint, is that reason enough to
reject addition?

The extensibility issue is actually an important case. Users want features
like scripting because they need something custom. Scripting brings back
imperative builds, which IMHO Maven views as an anti-pattern.

So what is the problem people are trying to solve? They want a plugin for
just one project. Right now you cannot because we cannot instantiate the
full build plan until we can inspect the plugin metadata... but until we
build the plugin we cannot get the metadata.

With a general plugin, I think there is not much we can do. For example a
plugin redefining a lifecycle might actually mean that the build plan will
be different for modules using that plugin as an extension.

But if the plugin is just defining some goals and those goals are just
bound to phases in the pre-existing life cycles... again, it seems we can
allow the build to proceed

Discussion points:
* how to handle `mvn test`. If the plugin is only bound to later phases, we
are good... but what if the plugin is bound to an early phase? Do we just
bomb out and say, sorry you need to invoke at least package? Do we fall
back to local repo? Do we upscale the build plan for the plugin to package
* how to handle `mvn site`... in this case we wouldn’t be producing the
plugin in the reactor anyway... do we bomb out and say “you must do `mvn
package site`“? Do we modify the build plan? Do we use local repo?
* we could be smarter and use the build plan safepoint as a staging point
and allow lifecycles and other things... though obviously more work
* what about invoking a plugin goal from the CLI directly... do we ask for
`mvn package foo:manchu`? For this to work, foo:manchu would have to be an
aggregator goal, otherwise it could be an execution request on earlier
modules by a later module? Could we delay the build plan instantiation
until after the plugin is built? Could we just require the plugin be served
from the local repository and revalidate the build plan after the new
plugin build is available?

I think these are two improvements that could be made *without* changes to
the modelVersion and they would have a real and meaningful improvement for
our users... as such I would like us to consider (as optional goals) for
Maven 4.0.0

Wdyt?
--
Sent from my phone
Reply | Threaded
Open this post in threaded view
|

Re: Reactor safe/rally points

stephenconnolly
In reply to this post by stephenconnolly
On Sun 5 Nov 2017 at 21:17, Robert Scholte <[hidden email]> wrote:

> One thing I'd like to change is that if a module is a dependency it must
> really behave like one.  It should not be the MavenProject, that should
> only be used for building *that* project. It should act as if it was
> downloaded via the artifact resolver


Which would need safe points too... eg flatten Maven plugin would need to
flag being a pom modifier goal... otherwise how do you know when the model
has changed ahead of time in creating the build plan


> Robert
>
> Verzonden vanaf mijn Samsung Galaxy-smartphone.
> -------- Oorspronkelijk bericht --------Van: Stephen Connolly <
> [hidden email]> Datum: 05-11-17  18:57  (GMT+01:00) Aan:
> Maven Developers List <[hidden email]> Onderwerp: Reactor
> safe/rally points
> There are two sets of problems that, assuming we want to fix, both need
> some way to rally a concurrent multimodule build at.
>
> 1. There is the shade like class of problems, where a plugin wants to
> modify the effective transitive dependencies of a module.
>
> 2. There is the extensibility class of problems, where people want to build
> a plugin and consume the same plugin in one reactor.
>
> Now in both cases, I think we can solve a subset of these problems by
> introducing a concept of a rally / safe point in the build plan.
>
> The mutable model can be solved for reducing scope/ removing
> dependencies... could possibly also allow adding dependencies, provided the
> dependencies are not part of the reactor. The proviso is that the plugin
> goal would need to be marked as a “mutation” goal, and the execution of all
> “mutation” goals for a module defines a safe-point. All downstream modules
> that declare a dependency must be blocked from execution until the
> safe-point is reached.
>
> Discussion points:
> * how do we handle a build request that will never reach the safe-point, eg
> `mvn test` will not reach package, and hence will not rub shade... should
> we say: fine, no safepoint in build plan, so all good... should we say:
> oops escalate the module with a safe point to `package`?
> * while technically possible to *add* ex-reactor dependencies (if
> ex-reactor they cannot affect the build plan) it could be a bad idea for
> build requests that wouldn’t reach the safepoint, is that reason enough to
> reject addition?
>
> The extensibility issue is actually an important case. Users want features
> like scripting because they need something custom. Scripting brings back
> imperative builds, which IMHO Maven views as an anti-pattern.
>
> So what is the problem people are trying to solve? They want a plugin for
> just one project. Right now you cannot because we cannot instantiate the
> full build plan until we can inspect the plugin metadata... but until we
> build the plugin we cannot get the metadata.
>
> With a general plugin, I think there is not much we can do. For example a
> plugin redefining a lifecycle might actually mean that the build plan will
> be different for modules using that plugin as an extension.
>
> But if the plugin is just defining some goals and those goals are just
> bound to phases in the pre-existing life cycles... again, it seems we can
> allow the build to proceed
>
> Discussion points:
> * how to handle `mvn test`. If the plugin is only bound to later phases, we
> are good... but what if the plugin is bound to an early phase? Do we just
> bomb out and say, sorry you need to invoke at least package? Do we fall
> back to local repo? Do we upscale the build plan for the plugin to package
> * how to handle `mvn site`... in this case we wouldn’t be producing the
> plugin in the reactor anyway... do we bomb out and say “you must do `mvn
> package site`“? Do we modify the build plan? Do we use local repo?
> * we could be smarter and use the build plan safepoint as a staging point
> and allow lifecycles and other things... though obviously more work
> * what about invoking a plugin goal from the CLI directly... do we ask for
> `mvn package foo:manchu`? For this to work, foo:manchu would have to be an
> aggregator goal, otherwise it could be an execution request on earlier
> modules by a later module? Could we delay the build plan instantiation
> until after the plugin is built? Could we just require the plugin be served
> from the local repository and revalidate the build plan after the new
> plugin build is available?
>
> I think these are two improvements that could be made *without* changes to
> the modelVersion and they would have a real and meaningful improvement for
> our users... as such I would like us to consider (as optional goals) for
> Maven 4.0.0
>
> Wdyt?
> --
> Sent from my phone
>
--
Sent from my phone
Reply | Threaded
Open this post in threaded view
|

Re: Reactor safe/rally points

Chas Honton
In reply to this post by stephenconnolly
Are these integration test for the plugin or do the integration tests use the plugin?  If the former, then the invoker plugin is appropriate.  If the latter, then a non-code, non-transitive dependency scope is a possibility.  Again, this new scope has the backwards compatibility problem..

> On Nov 5, 2017, at 11:30 PM, Stephen Connolly <[hidden email]> wrote:
>
> Yes it would, but socially people don’t do that... and anyway, there are
> legitimate use cases, eg tomee has (or had if they gave up fighting and
> moved it out of reactor) a plugin that was tied to the container version.
> To allow integration tests in the same reactor as the main codebase, it is
> reasonable to want the plugin built at the same time.