versioning by hashes to speedup multi-module build (a'la nix package manager)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

versioning by hashes to speedup multi-module build (a'la nix package manager)

Anton Vodonosov
Hello.

In order to speed up the build of a big multi-module project,
I'd like to reuse the artifacts of modules that haven't changed.
Manual versioning is tedious and error-prone.

Is it possible to automatically assign versions so that
versions only change if module sources or dependencies change?
In result, only those modules will be recompiled and retested,
and all other modules will be reused from artifact repository.

For example, this could be done if versions are computed as
hash-of( hash-of(module sources) + hashes of all dependencies).

In this approach, every change in code will modify such
hash-based version of all dependent modules automatically.

This would be similar to Nix package manager.

How such things (has based or to do that in maven?

Best regards,
- Anton  


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: versioning by hashes to speedup multi-module build (a'la nix package manager)

Konrad Windszus
Hi,
just look at http://maven.apache.org/maven-ci-friendly.html.
Konrad


> Am 01.02.2020 um 16:08 schrieb Anton Vodonosov <[hidden email]>:
>
> Hello.
>
> In order to speed up the build of a big multi-module project,
> I'd like to reuse the artifacts of modules that haven't changed.
> Manual versioning is tedious and error-prone.
>
> Is it possible to automatically assign versions so that
> versions only change if module sources or dependencies change?
> In result, only those modules will be recompiled and retested,
> and all other modules will be reused from artifact repository.
>
> For example, this could be done if versions are computed as
> hash-of( hash-of(module sources) + hashes of all dependencies).
>
> In this approach, every change in code will modify such
> hash-based version of all dependent modules automatically.
>
> This would be similar to Nix package manager.
>
> How such things (has based or to do that in maven?
>
> Best regards,
> - Anton  
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|

Re: versioning by hashes to speedup multi-module build (a'la nix package manager)

Karl Heinz Marbaise-3
In reply to this post by Anton Vodonosov
Hi,

On 01.02.20 16:08, Anton Vodonosov wrote:
> Hello.
>
> In order to speed up the build of a big multi-module project,
> I'd like to reuse the artifacts of modules that haven't changed.
> Manual versioning is tedious and error-prone.

Can you explain more in detail what you exactly mean and what kind of
problem you have best would be having an example project which shows the
issues...

Kind regards
Karl Heinz Marbaise

>
> Is it possible to automatically assign versions so that
> versions only change if module sources or dependencies change?
> In result, only those modules will be recompiled and retested,
> and all other modules will be reused from artifact repository.
>
> For example, this could be done if versions are computed as
> hash-of( hash-of(module sources) + hashes of all dependencies).
>
> In this approach, every change in code will modify such
> hash-based version of all dependent modules automatically.
>
> This would be similar to Nix package manager.
>
> How such things (has based or to do that in maven?
>
> Best regards,
> - Anton
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: versioning by hashes to speedup multi-module build (a'la nix package manager)

Anton Vodonosov
I want, for unchanged parts of the project, to reuse artifacts
produced by previous builds, and only rebuild the changed parts.

Imagine a project with hundreds of modules stored in a single
git repository, whose full build with tests takes 3 hours.

A developer creates a ticket branch, changes couple lines and
pushes the branch to the repository.

CI build starts.

I wish at this point only parts affected by the change to be rebuild.
And artifacts for unaffected modules simply be fetched from
artifacts repository (because previous builds placed them there).

Speaking of http://maven.apache.org/maven-ci-friendly.html,
if that means incorporating git commit into the version of all modules,
then all modules will be rebuilt in the above scenario,
even unaffected ones, because the new branch has a new git commit.

Is it correct?

Best regards,
- Anton

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: versioning by hashes to speedup multi-module build (a'la nix package manager)

Tommy Svensson
Hello Anton,

If I understand you correctly, you do not want to build all modules of the maven build, only those that has changed. 

As long as you are building in development shell or IDE the result of last build is still there and maven will only recompile sources that are newer than their classes in target. So doing an "mvn install" again will be much faster since it will only compile what has changes since last build. 

When running on a CI however each run has to be treated as a clean build. As the "maven-ci-friendly.html" points out "mvn clean install ..." is always done. This is the only safe thing to do. A CI can have many executor hosts that it delegates jobs to. 2 CI runs of the same thing is not necessarily  run on the same machine! Even if run on same machine different jobs gets different / personal and temporary areas on disk for that job. In other words, you cannot rely on a previous build being available. Every build is from scratch.

But assuming that your build do push built artifacts to a repository somewhere (not ~/.m2/repository, but a Nexus / Artifactory /  Bintray / ...), and that the build do look for artifacts in that repository, then you can make a separate CI configuration that only builds the module or modules you know have been modified, depending on the flexibility of the CI used. That is, instead of building from root module, trigger several builds, one for each submodule you want to rebuild. Depending on CI used it might require one config per sub module.  The configuration of such a build will have to be changed for each time you want to rebuild just some sub modules. It cannot be an automatic job triggered by new commit! You will have to configure it and run manually every time.

Best Regards,
Tommy


Från: Anton Vodonosov <[hidden email]>
Svara: Maven Users List <[hidden email]>
Datum: 2 februari 2020 at 16:10:44
Till: Maven Users List <[hidden email]>, [hidden email] <[hidden email]>
Cc: Konrad Windszus <[hidden email]>
Ämne:  Re: versioning by hashes to speedup multi-module build (a'la nix package manager)  

I want, for unchanged parts of the project, to reuse artifacts  
produced by previous builds, and only rebuild the changed parts.  

Imagine a project with hundreds of modules stored in a single  
git repository, whose full build with tests takes 3 hours.  

A developer creates a ticket branch, changes couple lines and  
pushes the branch to the repository.  

CI build starts.  

I wish at this point only parts affected by the change to be rebuild.  
And artifacts for unaffected modules simply be fetched from  
artifacts repository (because previous builds placed them there).  

Speaking of http://maven.apache.org/maven-ci-friendly.html,  
if that means incorporating git commit into the version of all modules,  
then all modules will be rebuilt in the above scenario,  
even unaffected ones, because the new branch has a new git commit.  

Is it correct?  

Best regards,  
- Anton  

---------------------------------------------------------------------  
To unsubscribe, e-mail: [hidden email]  
For additional commands, e-mail: [hidden email]