Re: source directory layout for multirelease jars

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: source directory layout for multirelease jars

rfscholte
I'm afraid this will be long story, but this might be the moment to share the details.

https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html describes already several topics:

- why MultiRelease Jars
- how to build them? There are several patterns, each with its pro's and con's.

But it doesn't explain the technical problems, so I guess we need to explain it a bit better.
I will also take the time to describes things that you might already know.

So you want to build a jar? You take the minimum pom (modelVersion + groupId + artifactId + version)
and you can set the packaging to jar (or omit it, as it is the default value).
You will call "package", which matches a phase of the default lifecycle.
Maven must first figure out the matching lifecycle, by looping over all the available.[1]
It will pick up all mapped Lifecycle components and will search in its phases. The result will be the "default" Lifecycle.
Based on this Maven will look for  "jar" named LifecycleMapper and the configuration for the "default"  lifecycle [2] 

Now it knows which plugins, versions and goals must be called. (it is possible to call multiple calls within one phase)
There is no explicit executionId (I'm actually not sure if you can use it here, I think so), so default-<goal> will be used as executionId.
You can see this in the console output of Maven. You can use this executionId for explicit configuration.
Keep in mind that plugins are very isolated.
They have their own classloader during execution.
They don't share configuration or instances with other plugins(*), you could treat the pom as the shared configuration.

For the creation of a Multirelease jar we have sepatare challenges for the following phases: compile, test and packaging.
The simplest of the three is probably packaging: you need to ensure that the manifest has the attribute Multi-Release: true 
Depending on in which folder/package structure the classes are available you need to build up the right structure, putting classes in their matching META-INF/versions/${release}/ -folder


Now the compiling part: 
My first approach is close to what everybody wants: no additional configuration in the pom (the minimal pom should be enough), use src/main/java<release>/ as predefined folder template.
I did a POC, and it looks promising: the plugin would loop over the folders, extracted the release value and executed javac a couple of times, but the classes in the matching directory.
Well, I was wrong: tests started to fail. I discovered that several parameters should only be used for a subset of a javac calls, or just exactly for one. Especially the includes/excludes caused trouble.
I could try to find a configuration for includes/excludes per release value within the same execution, but I knew there were other properties needing similar features.
And then I came to the conclusion the an "executionblock" should match exactly one javac call, it is already named like that!
It does mean, that you must define multiple execution-blocks. And because execution blocks aren't context aware you need to configure them a little bit. Right now that is:
<release>11</release>

<multiReleaseOutput>true</multiReleaseOutput>
<compileSourceRoots>
<compileSourceRoot>${project.basedir}/src/main/java11</compileSourceRoot>
</compileSourceRoots>


Multirelease is quite a specific feature that is only useful for a specific kind of libraries, so needing extra configuration for a reliable result was acceptable for me.

Testing is by far the hardest part and the most underestimated part: you can only test multirelease jar a a jar (so you cannot test it with surefire, you must use failsafe) and must test it separately with all targeted JDK release versions.
Or you must make multiple test-jars, stripping of every release layer, but that means that you are actually not testing the deliverable.
https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html shows a matrix with all kind of approaches.

My preferred solution is the test it with a CI server with multiple JDKs and use profiles to decide which mulitirelease sources should be compiled, but it all depends on your way of work.
github.com/codehaus-plexus/plexus-languages/blob/master/plexus-java/pom.xml#L105-L151 


The way you want to test has effect on how you can compile and vice versa.
I have my doubt there's a solution that covers it all.

Should Maven be changed for this? Well, maybe. The key is that you need to add executions dynamically, and the first plugin that can inform Maven about that is the maven-compiler-plugin.
I'm saying dynamically, because you don't want to have a static list like
<compile>
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compile,
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compile_java11,
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compile_java14,
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compile_java15
</compile>


It is possible to create MRJARs for several years, but it requires additional configuration.
And I don't mind, because even though the specs are quite simple, developing them is hard, so it should not be too easy to make them.

That said, I'm not sure if this is the topic I want to spend more valuable time and energy on. 
IMO You're only helping a very, very small group of people with something that already works in general
Developers just need to configure the pom to the style they want to build, test and package the MRJAR.

thanks,
Robert


[1] https://github.com/apache/maven/blob/master/maven-core/src/main/resources/META-INF/plexus/components.xml

[2] https://github.com/apache/maven/blob/master/maven-core/src/main/resources/META-INF/plexus/default-bindings.xml#L56-L98
On 24-4-2020 18:06:34, Christian Stein <[hidden email]> wrote:
Robert (?) wrote this page [0] about supporting MR-JAR files.

There're several approaches to the topic. It'd be nice, if the compiler- and
the jar-plugin would pick one by default or based on a directory layout
convention.

[0]:
https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html

On Fri, Apr 24, 2020 at 5:26 PM Elliotte Rusty Harold
wrote:

> I've been thinking about multirelease jars. First step is source layout.
> Here's a strawman modeled after the jar layout.
>
> We add a versions directory to the existing src directory. Inside the
> versions directory one can have directories named 9, 10, 11, etc. for each
> major Java version that has custom code. Each numbered directory is a full
> src folder that contains main, test, etc.
>
>
> Can anyone poke holes in this? Is there a reason why this wouldn't suffice?
>
> --
> Elliotte Rusty Harold
> [hidden email]
>
Reply | Threaded
Open this post in threaded view
|

Re: source directory layout for multirelease jars

Hervé BOUTEMY
when the need is restricted META-INF/versions/9/module-info.class, ie not
really code per Java release, but only module or not module, isn't one
strategy better to use, given there should be less tests to be run?

perhaps the answer is that MR is overkill for that, given JDK 8 will simply
ignore module-info.class?

Regards,

Hervé

Le jeudi 30 avril 2020, 19:40:15 CEST Robert Scholte a écrit :

> Hi Benjamin,
>
> You're confusing me
>
> "Sadly, you cannot just add executions to the maven compiler plugin."
> Why not? The design of the pom contains support of multiple executions
> within the same plugin
>
> "Almost no IDE supports it, ..."
> I think they all do. And even if they don't, they should. I consider an IDE
> as a wrapper around Maven, so it should support this basic feature
>
> "... and when I asked about multiple executions
> in the maven compiler plugin on this list I recevied only one reply
> that this is not an intended way of using this plugin (although I was
> using it for annotation processing with
> useIncrementalCompilation=false)."
>
> I don't recognize this question. And the answer looks incorrect to me.
>
> That said: if you just want a META-INF/versions/9/module-info.class, you can
> already do that (for a long time).
> https://github.com/apache/maven-compiler-plugin/tree/master/src/it/multirel
> ease-patterns However, there is no one ideal pattern, you have to pick one,
>
> Robert
>
>
>
>
> On 30-4-2020 08:49:02, Benjamin Marwell <[hidden email]> wrote:
> Hi,
>
> to make the story a bit shorter - for most libraries it would be
> sufficient to add a way to include a module descriptor at
> "META-INF/versions/9/module-info.class" while still staying at java 8.
> Most libraries will stick to java 8 or even 7 for quite a time.
>
> Sadly, you cannot just add executions to the maven compiler plugin.
> Almost no IDE supports it, and when I asked about multiple executions
> in the maven compiler plugin on this list I recevied only one reply
> that this is not an intended way of using this plugin (although I was
> using it for annotation processing with
> useIncrementalCompilation=false).
>
> I know there is the moditect plugin for such cases, but it doesn't
> play along nicely with the bundle-plugin and hasn't seen much progress
> lately as far as I can tell.
>
> That said, adding such a module descriptor would be a nice first goal
> without using executions.
>
> Of course this will all become obsolete as soon as everyone uses java 9+.
>
> Am Sa., 25. Apr. 2020 um 21:20 Uhr schrieb Robert Scholte
>
> > I'm afraid this will be long story, but this might be the moment to share
> > the details.
> >
> > https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html
> > describes already several topics:
> >
> > - why MultiRelease Jars
> > - how to build them? There are several patterns, each with its pro's and
> > con's.
> >
> > But it doesn't explain the technical problems, so I guess we need to
> > explain it a bit better. I will also take the time to describes things
> > that you might already know.
> >
> > So you want to build a jar? You take the minimum pom (modelVersion +
> > groupId + artifactId + version) and you can set the packaging to jar (or
> > omit it, as it is the default value). You will call "package", which
> > matches a phase of the default lifecycle. Maven must first figure out the
> > matching lifecycle, by looping over all the available.[1] It will pick up
> > all mapped Lifecycle components and will search in its phases. The result
> > will be the "default" Lifecycle. Based on this Maven will look for "jar"
> > named LifecycleMapper and the configuration for the "default" lifecycle
> > [2]
> >
> > Now it knows which plugins, versions and goals must be called. (it is
> > possible to call multiple calls within one phase) There is no explicit
> > executionId (I'm actually not sure if you can use it here, I think so),
> > so default- will be used as executionId. You can see this in the console
> > output of Maven. You can use this executionId for explicit configuration.
> > Keep in mind that plugins are very isolated.
> > They have their own classloader during execution.
> > They don't share configuration or instances with other plugins(*), you
> > could treat the pom as the shared configuration.
> >
> > For the creation of a Multirelease jar we have sepatare challenges for the
> > following phases: compile, test and packaging. The simplest of the three
> > is probably packaging: you need to ensure that the manifest has the
> > attribute Multi-Release: true Depending on in which folder/package
> > structure the classes are available you need to build up the right
> > structure, putting classes in their matching
> > META-INF/versions/${release}/ -folder
> >
> >
> > Now the compiling part:
> > My first approach is close to what everybody wants: no additional
> > configuration in the pom (the minimal pom should be enough), use
> > src/main/java/ as predefined folder template. I did a POC, and it looks
> > promising: the plugin would loop over the folders, extracted the release
> > value and executed javac a couple of times, but the classes in the
> > matching directory. Well, I was wrong: tests started to fail. I
> > discovered that several parameters should only be used for a subset of a
> > javac calls, or just exactly for one. Especially the includes/excludes
> > caused trouble. I could try to find a configuration for includes/excludes
> > per release value within the same execution, but I knew there were other
> > properties needing similar features. And then I came to the conclusion
> > the an "executionblock" should match exactly one javac call, it is
> > already named like that! It does mean, that you must define multiple
> > execution-blocks. And because execution blocks aren't context aware you
> > need to configure them a little bit. Right now that is: 11
> >
> > true
> >
> > ${project.basedir}/src/main/java11
> >
> >
> >
> > Multirelease is quite a specific feature that is only useful for a
> > specific kind of libraries, so needing extra configuration for a reliable
> > result was acceptable for me.
> >
> > Testing is by far the hardest part and the most underestimated part: you
> > can only test multirelease jar a a jar (so you cannot test it with
> > surefire, you must use failsafe) and must test it separately with all
> > targeted JDK release versions. Or you must make multiple test-jars,
> > stripping of every release layer, but that means that you are actually
> > not testing the deliverable.
> > https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html
> > shows a matrix with all kind of approaches.
> >
> > My preferred solution is the test it with a CI server with multiple JDKs
> > and use profiles to decide which mulitirelease sources should be
> > compiled, but it all depends on your way of work.
> > github.com/codehaus-plexus/plexus-languages/blob/master/plexus-java/pom.x
> > ml#L105-L151
> >
> >
> > The way you want to test has effect on how you can compile and vice versa.
> > I have my doubt there's a solution that covers it all.
> >
> > Should Maven be changed for this? Well, maybe. The key is that you need to
> > add executions dynamically, and the first plugin that can inform Maven
> > about that is the maven-compiler-plugin. I'm saying dynamically, because
> > you don't want to have a static list like
> >
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compile
> > ,
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compile
> > _java11,
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compil
> > e_java14,
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile@default-compil
> > e_java15
> >
> >
> >
> > It is possible to create MRJARs for several years, but it requires
> > additional configuration. And I don't mind, because even though the specs
> > are quite simple, developing them is hard, so it should not be too easy
> > to make them.
> >
> > That said, I'm not sure if this is the topic I want to spend more valuable
> > time and energy on. IMO You're only helping a very, very small group of
> > people with something that already works in general Developers just need
> > to configure the pom to the style they want to build, test and package
> > the MRJAR.
> >
> > thanks,
> > Robert
> >
> >
> > [1]
> > https://github.com/apache/maven/blob/master/maven-core/src/main/resources
> > /META-INF/plexus/components.xml
> >
> > [2]
> > https://github.com/apache/maven/blob/master/maven-core/src/main/resources
> > /META-INF/plexus/default-bindings.xml#L56-L98 On 24-4-2020 18:06:34,
> > Christian Stein wrote:
> > Robert (?) wrote this page [0] about supporting MR-JAR files.
> >
> > There're several approaches to the topic. It'd be nice, if the compiler-
> > and the jar-plugin would pick one by default or based on a directory
> > layout convention.
> >
> > [0]:
> > https://maven.apache.org/plugins/maven-compiler-plugin/multirelease.html
> >
> > On Fri, Apr 24, 2020 at 5:26 PM Elliotte Rusty Harold
> >
> > wrote:
> > > I've been thinking about multirelease jars. First step is source layout.
> > > Here's a strawman modeled after the jar layout.
> > >
> > > We add a versions directory to the existing src directory. Inside the
> > > versions directory one can have directories named 9, 10, 11, etc. for
> > > each
> > > major Java version that has custom code. Each numbered directory is a
> > > full
> > > src folder that contains main, test, etc.
> > >
> > >
> > > Can anyone poke holes in this? Is there a reason why this wouldn't
> > > suffice?
> > >
> > > --
> > > Elliotte Rusty Harold
> > > [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]