Incorrect dependencies being pulled in for module dependencies in assembly

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Incorrect dependencies being pulled in for module dependencies in assembly

Nigel Jones
I have a large multimodule project (https://github.com/odpi/egeria) which has over 300 modules, mostly java (->jar).

One of our submodules is responsible for creating the composite assembly. Previously we were doing it 'the wrong way' (IMO) with lots of relative paths (../../.. etc) and I've recently been trying to switch to using moduleSets, and follow best practices for assembly. We were also previously doing lots of building 'jar with dependencies' but this doesn't scale as every possibly useful for ends up far too big...

I have hit a problem with dependencies and moduleSets

We have a series of 'client' artifacts. These have some dependencies (for example we use restclient from spring, jackson etc). One section of my assembly attempts to package the clients into a folder, and include any dependencies they may need:

       <!-- Clients -->
        <moduleSet>
            <useAllReactorProjects>true</useAllReactorProjects>
            <includes>
                <include>org.odpi.egeria:asset-catalog-client:*</include>
                <include>org.odpi.egeria:asset-consumer-client:*</include>
                <include>org.odpi.egeria:asset-owner-client:*</include>
                <include>org.odpi.egeria:community-profile-client:*</include>
                <include>org.odpi.egeria:data-engine-client:*</include>
                <include>org.odpi.egeria:it-infrastructure-client:*</include>
                <include>org.odpi.egeria:it-infrastructure-client:*</include>
                <include>org.odpi.egeria:data-manager-client:*</include>
                <include>org.odpi.egeria:data-privacy-client:*</include>
                <include>org.odpi.egeria:data-science-client:*</include>
                <include>org.odpi.egeria:dev-ops-client:*</include>
                <include>org.odpi.egeria:digital-architecture-client:*</include>
                <include>org.odpi.egeria:discovery-engine-client:*</include>
                <include>org.odpi.egeria:governance-engine-client:*</include>
                <include>org.odpi.egeria:governance-program-client:*</include>
                <include>org.odpi.egeria:information-view-client:*</include>
                <include>org.odpi.egeria:project-management-client:*</include>
                <include>org.odpi.egeria:security-officer-client:*</include>
                <include>org.odpi.egeria:software-developer-client:*</include>
                <include>org.odpi.egeria:stewardship-action-client:*</include>
                <include>org.odpi.egeria:subject-area-client:*</include>
                <include>org.odpi.egeria:admin-services-client:*</include>
                <include>org.odpi.egeria:project-management-client:*</include>
            </includes>
            <binaries>
                <outputDirectory>./clients</outputDirectory>
                <unpack>false</unpack>
                <includeDependencies>true</includeDependencies>
            </binaries>

We have a similar pattern for utilities, samples, connectors etc.

I had thought 'includeDependencies' would JUST pull in dependencies for the modules listed in the included. However I seem to get every dependency for the entire project (reactor)

I then tried:

             <includeDependencies>false</includeDependencies>
                <dependencySets>
                </dependencySets>

instead, with the same result.

Then switched to this -- since the pom for the module doing the assembly DOES have a lot of dependencies (in fact it needs to be everything the assembly refers to as a module, so that we ensure those modules are built first - and exist) - so thinking (but unsure) if useProjectArtifact was relevant:

             <includeDependencies>false</includeDependencies>
                    <dependencySet>
                        <useTransitiveDependencies>true</useTransitiveDependencies>
                        <useProjectArtifact>false</useProjectArtifact>
                        <useProjectAttachments>false</useProjectAttachments>
                    </dependencySet>

but yet again this still pulls in all dependencies.

The results are inconsistent with 'mvn dependency:tree' - and in fact whatever is built I just always seem to get the same. It's not transitively based off every module in the reactor, but seems to be based of the pom of the project containing the assembly.

Shouldn't I just be pulling in dependencies of the binaries I specify in the moduleSet?

I've read the definitive maven book chapter, and the docs, but I can't quite figure out what I'm missing here.

To clarify as well, here's another example

       <!-- Conformance Suite -->
        <moduleSet>
            <useAllReactorProjects>true</useAllReactorProjects>
            <includes>
                <include>org.odpi.egeria:open-metadata-conformance-suite-client:*</include>
            </includes>
            <binaries>
                <outputDirectory>./conformance-suite</outputDirectory>
                <unpack>false</unpack>
                <includeDependencies>false</includeDependencies>
                <dependencySets>
                    <dependencySet>
                        <useTransitiveDependencies>true</useTransitiveDependencies>
                        <useProjectArtifact>false</useProjectArtifact>
                        <useProjectAttachments>false</useProjectAttachments>
                    </dependencySet>
                </dependencySets>
            </binaries>
        </moduleSet>

Again, I get the same dependencies - this time into ./conformance-suite yet that is just a single module I am putting into that dir - and I can simply show

[INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ open-metadata-conformance-suite-client ---
[INFO] org.odpi.egeria:open-metadata-conformance-suite-client:jar:2.1-SNAPSHOT
[INFO] +- org.odpi.egeria:open-metadata-conformance-suite-api:jar:2.1-SNAPSHOT:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.1:compile
[INFO] +- org.odpi.egeria:rest-client-connectors-api:jar:2.1-SNAPSHOT:compile
[INFO] |  \- org.odpi.egeria:open-connector-framework:jar:2.1-SNAPSHOT:compile
[INFO] |     \- org.odpi.egeria:audit-log-framework:jar:2.1-SNAPSHOT:compile
[INFO] +- org.odpi.egeria:rest-client-factory:jar:2.1-SNAPSHOT:compile
[INFO] |  \- org.odpi.egeria:spring-rest-client-connector:jar:2.1-SNAPSHOT:compile
[INFO] |     +- org.springframework:spring-web:jar:5.2.7.RELEASE:compile
[INFO] |     |  \- org.springframework:spring-beans:jar:5.2.7.RELEASE:compile
[INFO] |     +- org.codehaus.plexus:plexus-utils:jar:3.3.0:compile
[INFO] |     \- org.springframework:spring-core:jar:5.2.7.RELEASE:compile
[INFO] |        \- org.springframework:spring-jcl:jar:5.2.7.RELEASE:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.1:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.1:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] \- org.slf4j:slf4j-simple:jar:1.7.30:test

not so bad, yet deposited into that directory is my usual set of 462 or so jars..

Thanks for any advice!
Nigel.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Incorrect dependencies being pulled in for module dependencies in assembly

Nigel Jones


On 2020/07/17 15:33:26, Nigel Jones <[hidden email]> wrote:
> I have a large multimodule project (https://github.com/odpi/egeria) which has over 300 modules, mostly java (->jar).

Here's the full assembly definition (with the first attempt at including dependencies) -> https://gist.github.com/planetf1/4bef80e8454b3e57a3a8d948775abaf4

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Incorrect dependencies being pulled in for module dependencies in assembly

Nigel Jones

I took a debug log earlier -- as I found my overall assembly was taking 10 minutes to build

For that last example which has a handful of dependencies, what I actually see the assembly plugin reporting is:

17:11:33,629 [DEBUG] Dependencies for project: org.odpi.egeria:open-metadata-conformance-suite-client:jar:2.1-SNAPSHOT are:

17:11:33,629 [DEBUG] First version:2.1-SNAPSHOT
17:11:33,629 [DEBUG]  -> checking org.odpi.egeria:open-metadata-conformance-suite-client:jar:2.1-SNAPSHOT
17:11:33,629 [DEBUG] Processing binary dependencies for module project: org.odpi.egeria:open-metadata-conformance-suite-client:jar:2.1-SNAPSHOT
17:11:33,630 [DEBUG] Processing DependencySet (output=./conformance-suite)
17:11:33,630 [DEBUG] Filtering dependency artifacts WITHOUT transitive dependency path information.
17:11:33,630 [DEBUG] org.slf4j:slf4j-simple:jar:1.7.30 was removed by one or more filters.
17:11:33,630 [DEBUG] Adding 470 dependency artifacts.

There's the 470.... but why. That definately looks more like the overall project -- though if I do a pstree on the dependencies of this module, I actually resolve to 213 dependencies.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Incorrect dependencies being pulled in for module dependencies in assembly

Nigel Jones
Further checks in log file -- I see that for every single 'module' being processed I get

17:02:15,538 [DEBUG] Adding 470 dependency artifacts.

It's clear the current setup is NOT processing dependencies of the artifacts within a module set, but some kind of aggregate list across the entire assembly. This would look like a bug to me?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Incorrect dependencies being pulled in for module dependencies in assembly

Nigel Jones

I've added further notes into issue https://issues.apache.org/jira/projects/MASSEMBLY/issues/MASSEMBLY-940

along with a reference to a sandpit project which shows the issue more clearly (and quickly!)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]