Maven Filtering Plugin should avoid overwriting even when filtering

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Maven Filtering Plugin should avoid overwriting even when filtering

Robert Oxspring
Hi all,


When copying resources with filtering on, files are always overwritten even when the filters have not changed. I’d like to change this such that repeated filtering copies do not modify the destination file.

I’ve prepared a change to write the filtered content to a temporary file and only rename that over the destination if they’re different:
https://github.com/apache/maven-filtering/compare/master...roxspring:feature/avoid-overwrite-on-no-op-filter

Would something like this be acceptable? I’m guessing the next step is to create an issue in Jira? - agains MNG??

Thanks,

Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Maven Filtering Plugin should avoid overwriting even when filtering

Karl Heinz Marbaise-3
Hi,

On 25.04.20 16:25, Robert Oxspring wrote:
> Hi
>
> “mvn clean install” is definitely normal for small projects like maven plugins because it’s a cheap way of guaranteeing your build is up to date.

What does install guarantee which verify does not?

>
> For developers working on larger projects “mvn clean install” is typically avoided wherever possible as it forces everything to be rebuilt even when just a couple of lines of code have been changed. Instead folks rely on “mvn install” doing the least work practical to incrementally bring the target directory up to date.

The target directory is updated by "verify" as well...no need for install...

Rebuilt is based on the "clean"...nothing else..which is not neccessary...


>
> In complex multi-module builds its common for one modules filtered resources to be treated as inputs to later processes. At the moment, the filtering plug-in unconditionally overwrites resources such that the filtered output file is newly modified. This means that downstream processes see their inputs as newly modified and have to assume their outputs are outdated and need to be rebuilt.

The downstream which are usually jar's which are not related with other
modules which means they don't need to be updated (depending in the
change it might be neccessar to build) but related to resources not..
cause resources are only local the module....

Building only needed things can be achieved by using: mvn -pl xy -amd in
a multi module build...

>
> (Of course, in CI environments the opposite tradeoff is normal: accept slower builds in return for guaranteed clean reliable ones!)

The problem here is usually not the "clean" most of the time it is that
today builds are often running in cloud environements where for each
build a differernt node is taken ...which means nothing can be kept
(some tools trying to handle that like Travis CI, Circle CI etc.; Also
Jenkins such an option)...But that depends on the CI solution you are
using...

>
> At work, a “mvn install -DskipTests” on our multi-module project takes minutes even when nothing has changed,

The question: Using the most recent version of all Maven plugins and how
many modules / classes needs to be compiled..and their might be an issue
with maven-compiler-plugin which is different story...

I have several projects: one with ca. 45.000 Lines of code taking 35
seconds to build with "-DskipTests"...etc.

A Test project which I use for performance testings. Usual build via
"mvn verify" takes 45 seconds (2000 modules;)...(at the moment this
project does not contain resources but that could be good testcase..

Another time building takes: 28 seconds...incremental parts are already
working..but of course can be improved...


> when I think it could be finished in seconds. So far I’ve identified the filtering and assembly plugins as triggers of unnecessary work and am trying to offer fixes to optimise that behaviour!

With filtering you mean maven-resources-plugin which uses
maven-filtering component..

>
> That context help at all?
>
> Thanks,
>
> Rob
>
>> On 25 Apr 2020, at 14:37, Slawomir Jaranowski <[hidden email]> wrote:
>>
>> Hi,
>>
>> Can you describe your case and what you want to achieve.
>>
>> By default all files created during maven running are write to target
>> directory. And in most case target directory is cleaned before new build
>> starting.
>> Usual maven is running  by:
>>   mvn clean install.
>>
>> sob., 25 kwi 2020 o 00:59 Robert Oxspring <[hidden email]>
>> napisał(a):
>>
>>> Hi all,
>>>
>>>
>>> When copying resources with filtering on, files are always overwritten
>>> even when the filters have not changed. I’d like to change this such that
>>> repeated filtering copies do not modify the destination file.
>>>
>>> I’ve prepared a change to write the filtered content to a temporary file
>>> and only rename that over the destination if they’re different:
>>>
>>> https://github.com/apache/maven-filtering/compare/master...roxspring:feature/avoid-overwrite-on-no-op-filter
>>>
>>> Would something like this be acceptable? I’m guessing the next step is to
>>> create an issue in Jira? - agains MNG??
>>>
>>> Thanks,
>>>
>>> Rob
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Maven Filtering Plugin should avoid overwriting even when filtering

Robert Oxspring
In reply to this post by Robert Oxspring

> On 30 Apr 2020, at 07:38, Robert Scholte <[hidden email]> wrote:
>
> I prefer to see an in memory solution.

Well if it’s reasonable to assume that filtered files are always small then we could use replace the temporary file in my solution with an in memory buffer... but I’m not sure that’s what you’re shooting at?

> Key should be to detect if filtering is applied, which is done in the MultiDelimiterInterpolatorFilterReaderLineEnding[1]
> Once a value has been interpolated, you must rewrite the file, otherwise you shouldn't.

Again though, this appears to miss the subtlety: “if filtering is applied” is insufficient, the condition needs to be “if filtering is applied with different results than the previous run”. This requires either attempting to store some state between runs.

We could scan the source file for filtered values and store just that state (or checksum) in a file between runs. The cost would be an extra read of the source file + state comparison + writing out the state of each filtering. Is this what you’re thinking?

The alternative is to just use the target file as the (not minimal) state. We could read and filter the source file once, while reading and comparing with the target file in parallel. As soon as the contents start to differ then truncate the target file and append the rest of the filtered source to it. The cost here would be an extra full read of the target. Is this what you’re after?

Otherwise I’m at a loss to understand what would be acceptable.

Thanks!

Rob

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]