Asssembly plugin, classifiers - not finding files to assemble (also: slow)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Asssembly plugin, classifiers - not finding files to assemble (also: slow)

Nigel Jones
I'm getting an error when building an assembly. mvn is 3.6.3 and plugins are up to date (we use dependabot to keep them that way, along with pluginManagement)

[WARNING] The following patterns were never triggered in this artifact inclusion filter:
o  'org.odpi.egeria:egeria:pom:root:2.1-SNAPSHOT'

In my assembly descriptor I have (the wildcards were added in desperation!):

       <moduleSet>
            <useAllReactorProjects>true</useAllReactorProjects>
            <includes>
                <!-- Full qualification here is groupId:artifactId:packaging:classifier:version -->
                <!-- This retrieves our extra 'root' pom we use for carrying non-java extras needed -->
                <include>org.odpi.egeria:egeria:*:root</include>
            </includes>
            <sources>
                <includeModuleDirectory>false</includeModuleDirectory>
                <fileSets>
                    <fileSet>
                        <outputDirectory>.</outputDirectory>
                        <includes>
                            <include>*</include>
                            <include>**/*</include>
                            <include>NOTICE</include>
                            <include>LICENSE</include>
                        </includes>
                    </fileSet>
                </fileSets>
            </sources>
        </moduleSet>

The actual artifact is downloadable from my local m2, ie

mvn dependency:get -Dartifact=org.odpi.egeria:egeria:2.1-SNAPSHOT:pom:root -Ddest=/tmp/mvn

succeeds (though the dir ends up empty)

In my local repo I have:

jonesn:egeria/ (issue3231*) $ ls -latr ~/.m2/repository/org/odpi/egeria/egeria/2.1-SNAPSHOT                            
total 464
-rw-r--r--  1 jonesn  staff  137670 Jul 13 17:02 egeria-2.1-SNAPSHOT.pom
drwxr-xr-x  8 jonesn  staff     256 Jul 13 17:04 ..
drwxr-xr-x  6 jonesn  staff     192 Jul 13 17:04 .
-rw-r--r--  1 jonesn  staff   32517 Jul 13 17:23 egeria-2.1-SNAPSHOT-root.jar
-rw-r--r--  1 jonesn  staff     197 Jul 13 17:23 _remote.repositories
-rw-r--r--  1 jonesn  staff     739 Jul 13 17:23 maven-metadata-local.xml


and that particular jar for the root classifier looks like:

jonesn:egeria/ (issue3231*) $ jar -tf ~/.m2/repository/org/odpi/egeria/egeria/2.1-SNAPSHOT/egeria-2.1-SNAPSHOT-root.jar
META-INF/MANIFEST.MF
META-INF/
licenses/
META-INF/maven/
META-INF/maven/org.odpi.egeria/
META-INF/maven/org.odpi.egeria/egeria/
LICENSE
THIRD_PARTY.md
licenses/THIRD_PARTY-full.txt
NOTICE
README.md
META-INF/maven/org.odpi.egeria/egeria/pom.xml
META-INF/maven/org.odpi.egeria/egeria/pom.properties

It was actually built with

           <plugin>
                <inherited>false</inherited>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <executions>
                    <execution>
                        <id>add-resource</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                        <configuration>
                            <!-- Normally jar plugin is used with java source - here we use the top of the module tree as 'source' -->
                            <classesDirectory>${project.basedir}</classesDirectory>
                            <includes>
                                <include>README.md</include>
                                <include>LICENSE</include>
                                <include>licenses/*</include>
                                <include>THIRD_PARTY*</include>
                                <include>NOTICE</include>
                            </includes>
                            <!-- This project doesn't contain source so is type pom, but we can add extra artifacts as needed with a unique classifier -->
                            <classifier>root</classifier>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

Note that this top level project is nominally of packaging=pom. It contains no java source, but we have files needed for the assembly. I tried a few approaches but this seems the most viable.

I am confused why I am not getting a pattern match

Another thing I've noticed is that building the assembly is very slow. I'm on a 2016 MBP 16GB/SSD

The full assembly contains more than the fragment above (the other parts are working). We have 8 modulesets. 1 uses 25 artifacts, one 21 artifacts, 1 uses 8, 1 uses 4, the rest just one each. Mostly binaries are used, but a few definitions access the source (which I see logged below) to pull in readmes - in fact it's interesting that binaries don't result in logging.

Previously the assembly was NOT using artifacts and instead used relative pathnames from all over the >300 module project. This seemed an anti-maven pattern, so I attempted to refactor...

From my build logs:

17:27:42,946 [INFO] --- maven-assembly-plugin:3.3.0:single (default) @ open-metadata-assemblies ---
17:27:42,946 [INFO] Reading assembly descriptor: src/main/assemblies/egeria-omag.xml
17:27:49,613 [INFO] Processing sources for module project: org.odpi.egeria:server-chassis-spring:jar:2.1-SNAPSHOT
17:31:19,643 [INFO] Processing sources for module project: org.odpi.egeria:ui-chassis-spring:jar:2.1-SNAPSHOT
17:31:26,142 [INFO] Processing sources for module project: org.odpi.egeria:open-metadata-conformance-suite-client:jar:2.1-SNAPSHOT
17:31:41,923 [WARNING] The following patterns were never triggered in this artifact inclusion filter:
o  'org.odpi.egeria:egeria:*:root'

17:31:41,947 [INFO] Building tar: /Users/jonesn/IdeaProjects/egeria/open-metadata-distribution/open-metadata-assemblies/target/egeria-2.1-SNAPSHOT-distribution.tar.gz
17:33:09,746 [INFO] Processing sources for module project: org.odpi.egeria:server-chassis-spring:jar:2.1-SNAPSHOT
17:36:42,357 [INFO] Processing sources for module project: org.odpi.egeria:ui-chassis-spring:jar:2.1-SNAPSHOT
17:36:47,952 [INFO] Processing sources for module project: org.odpi.egeria:open-metadata-conformance-suite-client:jar:2.1-SNAPSHOT
17:37:03,728 [WARNING] The following patterns were never triggered in this artifact inclusion filter:
o  'org.odpi.egeria:egeria:*:root'

17:37:03,739 [INFO] Copying files to /Users/jonesn/IdeaProjects/egeria/open-metadata-distribution/open-metadata-assemblies/target/egeria-2.1-SNAPSHOT-distribution
17:37:10,251 [INFO]

The final assembly is 180MB

thanks for any tips/spotting my obvious mistakes, or commenting on the overall approach

nigel.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Asssembly plugin, classifiers - not finding files to assemble (also: slow)

Nigel Jones


On 2020/07/13 17:03:21, Nigel Jones <[hidden email]> wrote:
> I'm getting an error when building an assembly. mvn is 3.6.3 and plugins are up to date (we use dependabot to keep them that way, along with pluginManagement)

In case it helps here are the full versions of some of my files

https://gist.github.com/a1a2e8c24d3b8eb83860969045d77a68 - this is the parent pom (where the README/LICENSE is situated)
https://gist.github.com/12418eec23f0e3f98c9b97ca37af9267 - the pom where the assembly is built (it's 2 levels down)
https://gist.github.com/a51dbb8f80e3bcbc814df83f57a232da - the main assembly descriptor we're concerned with here



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Asssembly plugin, classifiers - not finding files to assemble (also: slow)

Nigel Jones
On 2020/07/13 17:17:14, Nigel Jones <[hidden email]> wrote:

Thinking through this whilst mind idling on other topics, I've realised something (tell me if I'm wrong)

I was thinking 'binaries' in terms of 'stuff to run' and sources in teams of 'stuff that isn't built' but I should be in a maven mindset - ie with the binaries being what goes into the main artifact (or perhaps those with other classifiers), whilst the sources are literally the '-sources' artifact? -- or indeed the full set of files in the module (since they are being built within the reactor)

The documentation on moduleSet isn't very clear IMO in distinguishing between binary and sources

Will adapt to try this approach.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Asssembly plugin, classifiers - not finding files to assemble (also: slow)

Nigel Jones


On 2020/07/14 09:29:22, Nigel Jones <[hidden email]> wrote:
> The documentation on moduleSet isn't very clear IMO in distinguishing between binary and sources

I should have found this before, but a google search hit 'the definitive guide' from 2008. I have a valid o'reilly subscription so had a read through the assemblies chapter. Though much of the content was similar to the docs, it was clearer IMO on the moduleset definitions

I realised all my over-thinking in creating a new classifier was unnecessary -- all I needed was:

      <moduleSet>
            <includeSubModules>false</includeSubModules>
            <useAllReactorProjects>true</useAllReactorProjects>
            <includes>
                <!-- Full qualification here is groupId:artifactId:packaging:classifier:version -->
                <!-- This retrieves our extra 'root' pom we use for carrying non-java extras needed -->
                <include>org.odpi.egeria:egeria:*</include>
            </includes>
            <sources>
                <includeModuleDirectory>false</includeModuleDirectory>
                <fileSets>
                    <fileSet>
                        <outputDirectory>/</outputDirectory>
                        <includes>
                            <include>NOTICE</include>
                            <include>LICENSE</include>
                            <include>truststore.p12</include>
                        </includes>
                    </fileSet>
                </fileSets>
            </sources>
        </moduleSet>

This works. I can now undo that other change, and look again at performance

Hopefully this brain dump will help others!
Nigel.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Asssembly plugin, classifiers - not finding files to assemble (also: slow)

Nigel Jones
A further clarification.

The following moduleset fails to include the license and readme and reports '[WARNING] The following patterns were never triggered in this artifact inclusion filter: o  'org.odpi.egeria:egeria:*'


       <!-- LICENSE & NOTICE and truststore from top level - this is a custom artifact -->
        <moduleSet>
            <useAllReactorProjects>true</useAllReactorProjects>
            <!--includeSubModules>false</includeSubModules-->
            <includes>
                <!-- Full qualification here is groupId:artifactId:packaging:classifier:version -->
                <!-- This retrieves our extra 'root' pom we use for carrying non-java extras needed -->
                <include>org.odpi.egeria:egeria:*</include>
            </includes>
            <sources>
                <includeModuleDirectory>false</includeModuleDirectory>
                <fileSets>
                    <fileSet>
                        <outputDirectory>/</outputDirectory>
                        <includes>
                            <include>NOTICE</include>
                            <include>LICENSE</include>
                            <include>truststore.p12</include>
                        </includes>
                    </fileSet>
                </fileSets>
            </sources>
        </moduleSet>

Uncommenting '<includeSubModules>false</includeSubModules>' results in a warning :

15:48:20,349 [WARNING] includeSubModules == false is incompatible with useAllReactorProjects. Ignoring.

To refactor, remove the <includeSubModules/> flag, and use the <includes/> and <excludes/> sections to fine-tune the modules included.


However in this case there isn't a warning of a missing module AND the license/readme correctly gets included

Is this a bug? It appears to be, since the warning says 'Ignoring' so the result should be the same as if it wasn't specified?

In this particular case the module being referenced in this section is the top level pom (the ultimate parent of the module containing the assembly)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Asssembly plugin, classifiers - not finding files to assemble (also: slow)

Nigel Jones


On 2020/07/14 14:57:05, Nigel Jones <[hidden email]> wrote:

> However in this case there isn't a warning of a missing module AND the license/readme correctly gets included

FYI I got all that I needed included, though I did raise https://issues.apache.org/jira/browse/MASSEMBLY-938?filter=-2 on this warning -- any attempt to resolve the warning leads to incorrect output

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]