Re: Maven Filtering Plugin should avoid overwriting even when filtering

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Maven Filtering Plugin should avoid overwriting even when filtering

Romain Manni-Bucau
Hi Robert, two thoughts maybe

1. all that work (thinking also of assembly one) should probably end up by
enriching
https://github.com/apache/maven-shared-incremental/blob/master/src/main/java/org/apache/maven/shared/incremental/IncrementalBuildHelper.java
model
and impl. I tend to think it does not need any hash but just a lastUpdated
track (the most recent file invalidating previous cache, it is enough and
faster than any hash computation)
2. by default nothing should be skipped - I guess a -Dxxx.incremental=true
should be added, maybe even to maven core (see next point), all this theory
is based on fully defining inputs/outputs fully. This is what gradle
implemented and I almost never saw a single complex gradle build correctly
configured to not bypass modules it shouldn't bypass (or worse, skip tests
it shouldn't) so I think we should be conservative here + several plugins
refetch data when executed so even if the result of 1 is "no change" you
can need to reexecute the plugin (including the filtering one if the
filtered data depends on the time or so - I don't want to discuss it is a
good or bad practise but it is used a lot in CI/CD pipelines).
3. Not being per plugin but global on maven sounds better than starting to
be specific in all plugins. I really see it as a dev feature you can
activate while working and not enable on the CI for example. Added to
maven-core it can enable to define in mojo the preconditions which would be
checked with 1 and potentially the conditions triggering inconditionally a
rebuild. Maybe a @MojoInput("${project.build.outputDirectory}",
"${project.artifacts}", "method:getState()") or so.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le sam. 25 avr. 2020 à 16:25, Robert Oxspring <[hidden email]> a
écrit :

> Hi
>
> “mvn clean install” is definitely normal for small projects like maven
> plugins because it’s a cheap way of guaranteeing your build is up to date.
>
> For developers working on larger projects “mvn clean install” is typically
> avoided wherever possible as it forces everything to be rebuilt even when
> just a couple of lines of code have been changed. Instead folks rely on
> “mvn install” doing the least work practical to incrementally bring the
> target directory up to date.
>
> In complex multi-module builds its common for one modules filtered
> resources to be treated as inputs to later processes. At the moment, the
> filtering plug-in unconditionally overwrites resources such that the
> filtered output file is newly modified. This means that downstream
> processes see their inputs as newly modified and have to assume their
> outputs are outdated and need to be rebuilt.
>
> (Of course, in CI environments the opposite tradeoff is normal: accept
> slower builds in return for guaranteed clean reliable ones!)
>
> At work, a “mvn install -DskipTests” on our multi-module project takes
> minutes even when nothing has changed, when I think it could be finished in
> seconds. So far I’ve identified the filtering and assembly plugins as
> triggers of unnecessary work and am trying to offer fixes to optimise that
> behaviour!
>
> That context help at all?
>
> Thanks,
>
> Rob
>
> > On 25 Apr 2020, at 14:37, Slawomir Jaranowski <[hidden email]>
> wrote:
> >
> > Hi,
> >
> > Can you describe your case and what you want to achieve.
> >
> > By default all files created during maven running are write to target
> > directory. And in most case target directory is cleaned before new build
> > starting.
> > Usual maven is running  by:
> >  mvn clean install.
> >
> > sob., 25 kwi 2020 o 00:59 Robert Oxspring <[hidden email]>
> > napisał(a):
> >
> >> Hi all,
> >>
> >>
> >> When copying resources with filtering on, files are always overwritten
> >> even when the filters have not changed. I’d like to change this such
> that
> >> repeated filtering copies do not modify the destination file.
> >>
> >> I’ve prepared a change to write the filtered content to a temporary file
> >> and only rename that over the destination if they’re different:
> >>
> >>
> https://github.com/apache/maven-filtering/compare/master...roxspring:feature/avoid-overwrite-on-no-op-filter
> >>
> >> Would something like this be acceptable? I’m guessing the next step is
> to
> >> create an issue in Jira? - agains MNG??
> >>
> >> Thanks,
> >>
> >> Rob
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> > --
> > Sławomir Jaranowski
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Maven Filtering Plugin should avoid overwriting even when filtering

Graham Leggett
On 25 Apr 2020, at 17:12, Robert Oxspring <[hidden email]> wrote:

>> I tend to think it does not need any hash but just a lastUpdated
>> track (the most recent file invalidating previous cache, it is enough and
>> faster than any hash computation)
>
> I’m not totally sure of this. Filtering, for example, is particularly dependant on properties and configuration which aren’t directly linked with a source file change.

Exactly - this is the key limitation.

I think we’re doomed to having to recreate the file each time, because we have no way of knowing whether the file has changed until you generate it.

But - there is nothing stopping us, when we detect the file is the same, from not updating the original file. This will have the effect of not triggering unnecessary downstream changes.

Concrete example: A filter generates a file that is input to a code generator. Because the file changes every time, the code generator runs every time, not good. But if we detected the file as the same, and didn’t write the filtered file, the code generator wouldn’t run, and we would safely avoid an unnecessary rebuild.

Regards,
Graham



smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Maven Filtering Plugin should avoid overwriting even when filtering

Tobias
In reply to this post by Romain Manni-Bucau
Hi,

I just came across this issue the other day, too. A mvn install after a
mvn clean install, i.e. nothing has to be compiled, results in a jar
being created for one module which takes one minute. This is due to
filtered resources being overwriten though this is not necessary. This
happens for multiples modules in our build. I too think that the speed
of the build process could be greatly improved if resources wouldn't be
overwriten if nothing has changed.

> I think we’re doomed to having to recreate the file each time, because we have no way of knowing whether the file has changed until you generate it.

I'm new to Maven and just discovering it, but my sketch of a naive and
simple, not optimized idea was the following:

- Identify all locations where information can be stored which is used
to generate filtered resources (pom.xml, .properties files, ...?)
- For all these locations, get the date it was last written to
- Take the most recent date from all these dates
- For each filtered resource which is already in the target folder from
the last build, take the date of this resource file and check if this
date is newer than the date identified in the previouis step. If it is,
don't copy the filtered resource to the target folder

Is this not possible?

I'd also like to add that I found a similar thread when searching for
this issue:
https://users.maven.apache.narkive.com/FAqDGEfS/avoiding-overwriting-newer-file-when-using-maven-resources-plugin-copy-resources-with-filtering

Best regards
Tobias



On 2020/04/25 13:58:15, Karl Heinz Marbaise <[hidden email]> wrote:
> Hi,>
>
> On 25.04.20 00:58, Robert Oxspring wrote:>
> > Hi all,>
> >>
> >>
> > When copying resources with filtering on, files are always
overwritten even when the filters have not changed. I’d like to change
this such that repeated filtering copies do not modify the destination
file.>
> >>
> > I’ve prepared a change to write the filtered content to a temporary
file and only rename that over the destination if they’re different:>
> >
https://github.com/apache/maven-filtering/compare/master...roxspring:feature/avoid-overwrite-on-no-op-filter>

> >>
> > Would something like this be acceptable? I’m guessing the next step
is to create an issue in Jira? - agains MNG??>

>
> There is a JIRA link on the github repository ...creating a JIRA is a>
> prerequisites..>
>
> Maven Filtering is a component which is used by>
> maven-resources-plugin...If you like to make that effort would be great>
> and see if all tests working fine ...and maybe we will merge that...>
>
> Kind regards>
> Karl Heinz Marbaise>
>
>
> >>
> > Thanks,>
> >>
> > Rob>
> > --------------------------------------------------------------------->
> > To unsubscribe, e-mail: [hidden email]>
> > For additional commands, e-mail: [hidden email]>
> >>
>
> --------------------------------------------------------------------->
> To unsubscribe, e-mail: [hidden email]>
> For additional commands, e-mail: [hidden email]>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]