Can a plugin change the reactor 'pom'?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Can a plugin change the reactor 'pom'?

Niels Basjes
Hi,

Last few days I dove into a dependency problem I had (and documented what I
ran into here https://yauaa.basjes.nl/NOTES-shading-dependencies.html ).

Along the way I fixed https://issues.apache.org/jira/browse/MSHADE-36 (use
the dependency-reduced-pom.xml instead of the original pom.xml) and then
ran into the way reactor apparently works.

As far as I can tell the reactor reads the original pom.xml files of all
modules and then builds from there.

So I was wondering: Is it possible for a maven plugin (like shade) to
change the way the reactor sees the module at hand? As in: Can shade make
the relocated dependencies 'vanish' from the view of the reactor?

Is this possible?

--
Best regards / Met vriendelijke groeten,

Niels Basjes
Reply | Threaded
Open this post in threaded view
|

Re: Can a plugin change the reactor 'pom'?

Kyle Marek
Shade used to modify the POM structure during build, but Maven was
changed to disallow this behavior for reasons regarding build consistency.

I ran into this issue somewhat recently. My approach for now, similar to
your "Solution 3", to exclude the shaded dependencies from the module
dependency, anywhere it is used in the project. This is easy with
dependencyManagement in the aggregating POM.

See:
https://mail-archives.apache.org/mod_mbox/maven-users/201903.mbox/%3cc23375da-1c44-10ec-8efd-c6e8e1b2f695@...%3e

Something like:

In Aggregator POM:

  <dependencyManagement>
    <dependencies>

      <!-- intra-module dependencies -->

      <dependency>
        <groupId>${project.groupId}</groupId>
        <artifactId>mss-core-shaded</artifactId>
        <version>${project.version}</version>
        <exclusions>
          <!--
            Workaround MNG-5899 and MSHADE-206

            Maven versions past 3.2.5 disallow modification of POM dependency
            models after they are initially calculated, so the reduced
            dependencies in dependency-reduced-pom.xml are not used.

            By excluding the dependencies that are shaded, we are manually
            reducing the dependency set as used in this multi-module project.

            See: https://issues.apache.org/jira/browse/MSHADE-206
            See: https://issues.apache.org/jira/browse/MNG-5899
          -->
          <exclusion>
            <groupId>org.apache.fluo</groupId>
            <artifactId>fluo-core</artifactId>
          </exclusion>
          <exclusion>
            <groupId>org.apache.fluo</groupId>
            <artifactId>fluo-api</artifactId>
          </exclusion>
          <exclusion>
            <groupId>org.apache.accumulo</groupId>
            <artifactId>accumulo-core</artifactId>
          </exclusion>
        </exclusions>
      </dependency>

    </dependencies>
  </dependencyManagement>


In some other module trying to use your shaded module:

    <dependency>
      <groupId>${project.groupId}</groupId>
      <artifactId>mss-core-shaded</artifactId>
    </dependency>


Furthermore, a side note for your project:

If you are shading+relocating a given dependency due to problems with
*its transitive dependencies*, you might find it necessary to ensure
that you shade the dependency and everything it uses, because shade
can't correct references to refer to the new package if you're not
including the dependency making the reference.

Basically, I copy the relevant portion from `mvn dependency:tree` paste
it into the following command to generate my shade includes:

sed -e 's/..*\s\([^:]\+:[^:]\+\):\([^:]\+:\)\?[^:]\+:[^:]\+:\([^:]\+\)$/\1 \3/;/ test$/d;s/ \S\+$//;s/.\+/                  <include>&<\/include>/' | sort -u


Then put the output in the following plugin configuration to tell shade
exactly what you're shading into your artifact:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml</dependencyReducedPomLocation>
              <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
              </transformers>

              <!--
                include dependencies referencing the problem artifacts
              -->
              <artifactSet>
                <includes>
                  <include>${project.groupId}:mss-core</include>

                  <!--
                    paste generated includes from command here
                  -->

                </includes>
              </artifactSet>

              <!--
                relocate classes of problem artifacts
              -->
              <relocations>
                <relocation>
                  <pattern>javax.servlet.jsp</pattern>
                  <shadedPattern>${project.groupId}.shaded.javax.servlet.jsp</shadedPattern>
                </relocation>
                <relocation>
                  <pattern>javax.ws.rs</pattern>
                  <shadedPattern>${project.groupId}.shaded.javax.ws.rs</shadedPattern>
                </relocation>
              </relocations>

            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

And remember, shading is messy and won't correct issues like classes
being loaded programmatically. You should attempt to segment your code
using the problematic dependency to operate under its own classloader,
with its own classpath, before shading. In my case, I was encountering
the dependency conflict in a webapp, which may be deployed on
application containers that may not allow webapps to read their file
system, which would interfere with my ability to programmatically load
problem classes from a segregated lib directory.

On 8/27/19 10:42 AM, Niels Basjes wrote:

> Hi,
>
> Last few days I dove into a dependency problem I had (and documented what I
> ran into here https://yauaa.basjes.nl/NOTES-shading-dependencies.html ).
>
> Along the way I fixed https://issues.apache.org/jira/browse/MSHADE-36 (use
> the dependency-reduced-pom.xml instead of the original pom.xml) and then
> ran into the way reactor apparently works.
>
> As far as I can tell the reactor reads the original pom.xml files of all
> modules and then builds from there.
>
> So I was wondering: Is it possible for a maven plugin (like shade) to
> change the way the reactor sees the module at hand? As in: Can shade make
> the relocated dependencies 'vanish' from the view of the reactor?
>
> Is this possible?
Reply | Threaded
Open this post in threaded view
|

Re: Can a plugin change the reactor 'pom'?

Niels Basjes
Hi,

Thanks for the feedback.
To me https://issues.apache.org/jira/browse/MSHADE-206 seems very closely
related to https://issues.apache.org/jira/browse/MSHADE-36 for which I put
up a patch yesterday: https://github.com/apache/maven-shade-plugin/pull/25

I did some experimenting last night and found that apparently you can
change the dependencies of a module from code.

So I put together some code and tests and created this:
https://issues.apache.org/jira/browse/MSHADE-326
https://github.com/apache/maven-shade-plugin/pull/26

I would really like to hear what you think about this.

Niels Basjes

On Tue, Aug 27, 2019 at 7:53 PM Kyle Marek <[hidden email]> wrote:

> Shade used to modify the POM structure during build, but Maven was
> changed to disallow this behavior for reasons regarding build consistency.
>
> I ran into this issue somewhat recently. My approach for now, similar to
> your "Solution 3", to exclude the shaded dependencies from the module
> dependency, anywhere it is used in the project. This is easy with
> dependencyManagement in the aggregating POM.
>
> See:
>
> https://mail-archives.apache.org/mod_mbox/maven-users/201903.mbox/%3cc23375da-1c44-10ec-8efd-c6e8e1b2f695@...%3e
>
> Something like:
>
> In Aggregator POM:
>
>   <dependencyManagement>
>     <dependencies>
>
>       <!-- intra-module dependencies -->
>
>       <dependency>
>         <groupId>${project.groupId}</groupId>
>         <artifactId>mss-core-shaded</artifactId>
>         <version>${project.version}</version>
>         <exclusions>
>           <!--
>             Workaround MNG-5899 and MSHADE-206
>
>             Maven versions past 3.2.5 disallow modification of POM
> dependency
>             models after they are initially calculated, so the reduced
>             dependencies in dependency-reduced-pom.xml are not used.
>
>             By excluding the dependencies that are shaded, we are manually
>             reducing the dependency set as used in this multi-module
> project.
>
>             See: https://issues.apache.org/jira/browse/MSHADE-206
>             See: https://issues.apache.org/jira/browse/MNG-5899
>           -->
>           <exclusion>
>             <groupId>org.apache.fluo</groupId>
>             <artifactId>fluo-core</artifactId>
>           </exclusion>
>           <exclusion>
>             <groupId>org.apache.fluo</groupId>
>             <artifactId>fluo-api</artifactId>
>           </exclusion>
>           <exclusion>
>             <groupId>org.apache.accumulo</groupId>
>             <artifactId>accumulo-core</artifactId>
>           </exclusion>
>         </exclusions>
>       </dependency>
>
>     </dependencies>
>   </dependencyManagement>
>
>
> In some other module trying to use your shaded module:
>
>     <dependency>
>       <groupId>${project.groupId}</groupId>
>       <artifactId>mss-core-shaded</artifactId>
>     </dependency>
>
>
> Furthermore, a side note for your project:
>
> If you are shading+relocating a given dependency due to problems with
> *its transitive dependencies*, you might find it necessary to ensure
> that you shade the dependency and everything it uses, because shade
> can't correct references to refer to the new package if you're not
> including the dependency making the reference.
>
> Basically, I copy the relevant portion from `mvn dependency:tree` paste
> it into the following command to generate my shade includes:
>
> sed -e 's/..*\s\([^:]\+:[^:]\+\):\([^:]\+:\)\?[^:]\+:[^:]\+:\([^:]\+\)$/\1
> \3/;/ test$/d;s/ \S\+$//;s/.\+/                  <include>&<\/include>/' |
> sort -u
>
>
> Then put the output in the following plugin configuration to tell shade
> exactly what you're shading into your artifact:
>
>   <build>
>     <plugins>
>       <plugin>
>         <groupId>org.apache.maven.plugins</groupId>
>         <artifactId>maven-shade-plugin</artifactId>
>         <executions>
>           <execution>
>             <phase>package</phase>
>             <goals>
>               <goal>shade</goal>
>             </goals>
>             <configuration>
>
> <dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml</dependencyReducedPomLocation>
>
> <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
>               <transformers>
>                 <transformer
> implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
>               </transformers>
>
>               <!--
>                 include dependencies referencing the problem artifacts
>               -->
>               <artifactSet>
>                 <includes>
>                   <include>${project.groupId}:mss-core</include>
>
>                   <!--
>                     paste generated includes from command here
>                   -->
>
>                 </includes>
>               </artifactSet>
>
>               <!--
>                 relocate classes of problem artifacts
>               -->
>               <relocations>
>                 <relocation>
>                   <pattern>javax.servlet.jsp</pattern>
>
> <shadedPattern>${project.groupId}.shaded.javax.servlet.jsp</shadedPattern>
>                 </relocation>
>                 <relocation>
>                   <pattern>javax.ws.rs</pattern>
>                   <shadedPattern>${project.groupId}.shaded.javax.ws.rs
> </shadedPattern>
>                 </relocation>
>               </relocations>
>
>             </configuration>
>           </execution>
>         </executions>
>       </plugin>
>     </plugins>
>   </build>
>
> And remember, shading is messy and won't correct issues like classes
> being loaded programmatically. You should attempt to segment your code
> using the problematic dependency to operate under its own classloader,
> with its own classpath, before shading. In my case, I was encountering
> the dependency conflict in a webapp, which may be deployed on
> application containers that may not allow webapps to read their file
> system, which would interfere with my ability to programmatically load
> problem classes from a segregated lib directory.
>
> On 8/27/19 10:42 AM, Niels Basjes wrote:
> > Hi,
> >
> > Last few days I dove into a dependency problem I had (and documented
> what I
> > ran into here https://yauaa.basjes.nl/NOTES-shading-dependencies.html ).
> >
> > Along the way I fixed https://issues.apache.org/jira/browse/MSHADE-36
> (use
> > the dependency-reduced-pom.xml instead of the original pom.xml) and then
> > ran into the way reactor apparently works.
> >
> > As far as I can tell the reactor reads the original pom.xml files of all
> > modules and then builds from there.
> >
> > So I was wondering: Is it possible for a maven plugin (like shade) to
> > change the way the reactor sees the module at hand? As in: Can shade make
> > the relocated dependencies 'vanish' from the view of the reactor?
> >
> > Is this possible?
>


--
Best regards / Met vriendelijke groeten,

Niels Basjes