Re: Reproducible Builds for Maven

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Reproducible Builds for Maven

Vladimir Sitnikov
Hervé>Maven core itself can be built in a reproducible way!

This is nice, thank you for pushing this.
It would be really great if Maven plugins defaulted to reproducible artifacts as well (e.g. maven-jar produced a jar with sorted entries and fixed timestamps).

Hervé>Interested in joining the effort to bring this feature to releases for end users?

+1.
Does that include reproducible source releases?

It would help to verify them and vote on releases as well, as everybody could build their own copy of Maven and check if the resulting archive is the same as the one that is proposed for the release.

Hervé>- do you build on Windows or any Unix? This impacts newlines...

Can the build enforce certain newlines? For instance, *.sh scripts have to be LF no matter what the build OS is.
The same goes for *.cmd which have to be CRLF and so on.

I've implemented reproducible builds for Apache JMeter, and one more variable is "executable bit".
Git on Windows often produces "executable readme.txt" and things like that, so JMeter's build script enforces executable bits for the resulting archives.

One more variable is "the state of Maven local repository". In other words, if I `mvn install` a library (e.g. while hacking on commons-io), then Maven build would pick that library from my .m2 folder, and it would be included to the Maven build which is bad for reproducibility.
That could have been prevented by pgpverify-maven-plugin which could verify that dependencies have expected PGP signatures.
Apparently my own commons-io build could not have the proper PGP signature, so Maven could warn me like "hey, you're using unexpected commons-io.jar".

I've tried mvn install -DskipTests on my macOS machine, and the result differs

Here's output of mvn clean install -DskipTests -Papache-release:
[INFO] --- checksum-maven-plugin:1.7:files (source-release-checksum) @ apache-maven ---
[INFO] apache-maven-3.6.3-SNAPSHOT-bin.tar.gz - SHA-512 : 4cc59e85d811dc9c900bd4b527db7327f8790b408004d254f6b270bddd4137aa4e9d87733ba2fb14e80fcf31d28ac24874d344ade7746598546bc36d156befa8
[INFO] apache-maven-3.6.3-SNAPSHOT-src.zip - SHA-512 : b7dabdfd8d3b3f4afdea70e46b0be1c33177a487cffeddd325769aa97c121b3983b54f99c35f05ba91d4d6885953b88c681254a833f77c2ab119e71fa56f258e
[INFO] apache-maven-3.6.3-SNAPSHOT-bin.zip - SHA-512 : 5f23cb830991babd610edeeefc3dbb08b01f1d83be19edeb206bb7ea5844bc8d390ecae0c95328cab73c3f957f997ac9cd64cfe35f4ff9097bbe0af17ad567c2
[INFO] apache-maven-3.6.3-SNAPSHOT-src.tar.gz - SHA-512 : a6cea689fefaf07bbeb5dc070b6ebeada7d1590aec7fcebb067ea22dca72cd52580eee99200a87090e8035a9b1b37217e1d4fb556a74312c510a85c3c274e2ee

$ git describe --long --all
heads/reproducible-0-g2d45315a8

$ mvn -X install -DskipTests
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T21:33:14+03:00)
Maven home: /nix/store/wqsccql2a0v048cd5l95f2hfp0l7rz6i-apache-maven-3.5.4/maven
Java version: 1.8.0_121, vendor: Azul Systems, Inc., runtime: /nix/store/89rm62lkhnnd1a0fadfg1z426wnangy2-zulu1.8.0_121-8.20.0.5/jre
Default locale: ru_RU, platform encoding: UTF-8
OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"

Just in case, I've attached (see maven-3.6.3-checksums.zip) the contents and the checksums of the files I get.

Vladimir


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

maven-3.6.3-checksums.zip (142K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Reproducible Builds for Maven

Emmanuel Bourg
Le 23/09/2019 à 01:52, Hervé BOUTEMY a écrit :
> after a few years of testing, thinking, procrastination and hard work (thank you Thomas for your talk at Devoxx France 2016 [1]), I think I achieved a key step this week-end toward native Reproducible Builds with Maven [2]: Maven core itself can be built in a reproducible way!

An important milestone on a very long journey. Well done!

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Reproducible Builds for Maven

Mark Derricutt
In reply to this post by Vladimir Sitnikov
Oh right! I missed the second half of our pom.deps file:

    blacklisted org.hibernate:hibernate-ehcache:4.2.19.Final from
smx3:smx3.bill-of-materials:2.1.1;
    deprecated org.hibernate:hibernate-search-engine:4.4.0.Final from
smx3:smx3.bill-of-materials:2.1.1;
    resolved org.hibernate:hibernate-search-orm:4.4.0.Final from
smx3:smx3.bill-of-materials:2.1.1;
    locked org.hibernate:hibernate-validator:4.3.1.Final from
smx3:smx3.upstream.bill-of-materials:1.0.16;

I provide 4 types of resolution, deprecated/blacklisted means the build
will fail, unless you specifically opt-in to deprecated/blacklisted
dependencies.

On resolve, anything that's changed moves from `locked` to either
resolved/deprecated/blacklisted, and on release anything resolved goes
to locked.

In the end user projects, we "import" the bill of materials .deps
artifact which pulls in any locked version, but I also have support for
adding:

    allow unlocked /^.*someregex.*$/;

which will trigger a re-resolve, which we use for our own internal
artifacts, so we pick up the latest internal releases IF we re-resolve.

The whole process seems to work well, it does cause some annoying round
trip releases or quirks now and then, but for the most part - it's
really solved a heap of issues where we've needed to back a quick
backport fix in a production environment.

Since we publish the pom.deps file for each artifact, when doing the
backports, we simply import the deps used in the distribution version
we're patching, and manually allow any updated deps that we need to fix.

Tomo Suzuki wrote:
>
> For reproducible builds, I expected the lock file contains specific
> versions, rather than ranges. Would you share the background why your tool
> records the ranges?
Reply | Threaded
Open this post in threaded view
|

Re: Reproducible Builds for Maven

Tomo Suzuki
In reply to this post by Vladimir Sitnikov
Cool. Thanks.