Computing and publishing checksums

[Andres Almiray]

Hello there,

What's the recommended way to compute and publish checksums (several
algorithms such as MD5, SHA256, SHA512) for all artifacts (POMs y binary
JARs at the very least, sources & javadocs may be excluded).

I'm aware that reproducible builds[1] requires publishing checksums in a
.buildinfo file and that the Maven artifacts plugin 3.0.0 will be released
soon (-ish).

Cheers,
Andres

[1] https://reproducible-builds.org/docs/jvm/

[Hervé BOUTEMY]

Hi Andres,

Le lundi 15 février 2021, 21:52:03 CET Andres Almiray a écrit :
> Hello there,
>
> What's the recommended way to compute and publish checksums (several
> algorithms such as MD5, SHA256, SHA512) for all artifacts (POMs y binary
> JARs at the very least, sources & javadocs may be excluded).
the question is: where do you want to publish, with what process?

If it's to your remote repository, maven-deploy-plugin does the job

If it's local to your build (then you do whatever you want with the files in
your own publication process), we use net.nicoulaj.maven.plugins:checksum-
maven-plugin at Apache Software Foundation parent build level to compute
sha512 checksum of source release archives, that are later published to Apache
Software Distribution area.
see https://maven.apache.org/pom/asf/#The_apache-release_Profile and
corresponding pom.xml

>
> I'm aware that reproducible builds[1] requires publishing checksums in a
> .buildinfo file and that the Maven artifacts plugin 3.0.0 will be released
> soon (-ish).
I would not say that " reproducible builds[1] requires publishing checksums in
a .buildinfo file": Reproducible Builds require that if you build twice and
compute checksums, you'll get the same value.

You don't need to publish your checksums: anybody can compute checksums at any
time, that's what Maven Artifact Plugin does on the fly both for current build
result and from remote reference build binaries.
That's what is done in Reproducible Central, for example:
https://github.com/jvm-repo-rebuild/reproducible-central

And to be clear, I'm not sure that publishing .buildinfo to Central repository
is a good idea, particularly when the spec I wrote [1] is SNAPSHOT,
implemented only by few tools, never had any real feedback, and that I
implemented in Maven Artifact Plugin something that is finally different from
the theory I wrote 1 year before in the spec.

First part of REX: only SBT implemented .buildinfo, and published the output
to Central for com.scalapenos:stamina releases (visible in Reproducible
Central) and a few other. And it showed one of the first issues: on a build
that produces many gavs (multi-modules in Maven, multi-scala versions in SBT),
is it better to produce 1 unique .buildinfo for the whole multi-module build
(like I did in Maven Artifact Plugin) or to produce 1 .buildinfo file per gav
(like done in SBT case)?

I know that I need to take time to write some REX on the issues I encountered
when implementing the spec.
If you're interested, I will probably take time to write this REX: I'm tired
on writing many documentations that nobody gives feedback on, after having
really used.

Regards,

Hervé

>
> Cheers,
> Andres
>
> [1] https://reproducible-builds.org/docs/jvm/





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Andres Almiray]

Hi Hervé,

That's right, I can see MD5 and SHA1 checksums for the
sdkman-maven-plugin[1] which uses the deploy, release, and nexus-staging
plugins for publication.
However I didn't see an option for SHA256 or SHA512 which is why I asked. I
also found the net.nicoulaj.maven.plugins:checksum-maven-plugin plugin and
wondered if that would be the one to use going forward, thanks for
confirming.

Given that MD5 and SHA1 are already published during deployment I'd prefer
if SHA256 and SHA512 were also published at the same location, I suppose I
can configure checksum-maven-plugin for that to happen.
Regarding reproducible builds, while it may not be required to publish
checksums you do need to check them against a previous value. Of course you
can check the value locally if you build twice, but you'd also want to
check the value against an "official" valkue, or at least a previously
published value. I'm keen in discussing reproducible builds as I
implemented a similar plugin for Gradle (which provides similar behavior,
yet no aggregates so far).

Cheers,
Andres

[1] https://repo1.maven.org/maven2/io/sdkman/sdkman-maven-plugin/2.0.0/



On Mon, Feb 15, 2021 at 11:51 PM Hervé BOUTEMY <[hidden email]>
wrote:

> Hi Andres,
>
> Le lundi 15 février 2021, 21:52:03 CET Andres Almiray a écrit :
> > Hello there,
> >
> > What's the recommended way to compute and publish checksums (several
> > algorithms such as MD5, SHA256, SHA512) for all artifacts (POMs y binary
> > JARs at the very least, sources & javadocs may be excluded).
> the question is: where do you want to publish, with what process?
>
> If it's to your remote repository, maven-deploy-plugin does the job
>
> If it's local to your build (then you do whatever you want with the files
> in
> your own publication process), we use net.nicoulaj.maven.plugins:checksum-
> maven-plugin at Apache Software Foundation parent build level to compute
> sha512 checksum of source release archives, that are later published to
> Apache
> Software Distribution area.
> see https://maven.apache.org/pom/asf/#The_apache-release_Profile and
> corresponding pom.xml
>
> >
> > I'm aware that reproducible builds[1] requires publishing checksums in a
> > .buildinfo file and that the Maven artifacts plugin 3.0.0 will be
> released
> > soon (-ish).
> I would not say that " reproducible builds[1] requires publishing
> checksums in
> a .buildinfo file": Reproducible Builds require that if you build twice
> and
> compute checksums, you'll get the same value.
>
> You don't need to publish your checksums: anybody can compute checksums at
> any
> time, that's what Maven Artifact Plugin does on the fly both for current
> build
> result and from remote reference build binaries.
> That's what is done in Reproducible Central, for example:
> https://github.com/jvm-repo-rebuild/reproducible-central
>
> And to be clear, I'm not sure that publishing .buildinfo to Central
> repository
> is a good idea, particularly when the spec I wrote [1] is SNAPSHOT,
> implemented only by few tools, never had any real feedback, and that I
> implemented in Maven Artifact Plugin something that is finally different
> from
> the theory I wrote 1 year before in the spec.
>
> First part of REX: only SBT implemented .buildinfo, and published the
> output
> to Central for com.scalapenos:stamina releases (visible in Reproducible
> Central) and a few other. And it showed one of the first issues: on a
> build
> that produces many gavs (multi-modules in Maven, multi-scala versions in
> SBT),
> is it better to produce 1 unique .buildinfo for the whole multi-module
> build
> (like I did in Maven Artifact Plugin) or to produce 1 .buildinfo file per
> gav
> (like done in SBT case)?
>
> I know that I need to take time to write some REX on the issues I
> encountered
> when implementing the spec.
> If you're interested, I will probably take time to write this REX: I'm
> tired
> on writing many documentations that nobody gives feedback on, after having
> really used.
>
> Regards,
>
> Hervé
>
> >
> > Cheers,
> > Andres
> >
> > [1] https://reproducible-builds.org/docs/jvm/
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

[Hervé BOUTEMY]

Le mardi 16 février 2021, 00:34:45 CET Andres Almiray a écrit :
> Hi Hervé,
>
> That's right, I can see MD5 and SHA1 checksums for the
> sdkman-maven-plugin[1] which uses the deploy, release, and nexus-staging
> plugins for publication.
> However I didn't see an option for SHA256 or SHA512 which is why I asked. I
> also found the net.nicoulaj.maven.plugins:checksum-maven-plugin plugin and
> wondered if that would be the one to use going forward, thanks for
> confirming.
I did not confirm *if it's to deploy to your repository*, but only if it's for
other purposes.

>
> Given that MD5 and SHA1 are already published during deployment I'd prefer
> if SHA256 and SHA512 were also published at the same location, I suppose I
> can configure checksum-maven-plugin for that to happen.
I suppose checksum-maven-plugin won't properly work, because when it will
attach .sha512, deploy will calculate .sha512.sha1 and .sha512.md5: you should
check for such noise

I don't know if someone has a plan to add sha256/sha512 to deploy (Robert
Scholte?).
Personally, I think this feature just gives wrong sense of safety:
1. if checksum is for network control, sha1 and even md5 are sufficient,
2. if it's against malicious actors, malicious actor will tamper the checksum
when tampering the binary.
If you want protection against malicious actors, pgp signatures verification is
the way to go.

> Regarding reproducible builds, while it may not be required to publish
> checksums you do need to check them against a previous value. Of course you
> can check the value locally if you build twice, but you'd also want to
> check the value against an "official" valkue, or at least a previously
> published value.
yes: maven-artifact-plugin downloads reference "official" binaries and
calculates checksums, no need to download pre-calculated checksums (that you
eventually should not trust).
It's what Reproducible Central does successfully at large scale, without any
.buildinfo being available on Central repository.

REX part 2: the only really useful info in .buildinfo to publish to Central
repository may be the effective environment: which OS, which JVM, which build
tool version? = something currently Maven Artifact Plugins guesses from jar
content: that's why I wrote "eventually", because these guesses are currently
enough in real world...
Everything else can be computed later, when trying to compare local build
output against reference binaries.

> I'm keen in discussing reproducible builds as I
> implemented a similar plugin for Gradle (which provides similar behavior,
> yet no aggregates so far).
I'm very interested to add Gradle support to Reproducible Central: do you know
any artifact in Central that is expected to be reproducible?

We can first check by hand (basically rebuilding by hand and checking by hand
output binaries), then automating the reproducible build check = what
Reproducible Central is, with currently Maven and SBT support, but I would
love to add other build tools

Regards,

Hervé

>
> Cheers,
> Andres
>
> [1] https://repo1.maven.org/maven2/io/sdkman/sdkman-maven-plugin/2.0.0/
>
>
>
> On Mon, Feb 15, 2021 at 11:51 PM Hervé BOUTEMY <[hidden email]>
>
> wrote:
> > Hi Andres,
> >
> > Le lundi 15 février 2021, 21:52:03 CET Andres Almiray a écrit :
> > > Hello there,
> > >
> > > What's the recommended way to compute and publish checksums (several
> > > algorithms such as MD5, SHA256, SHA512) for all artifacts (POMs y binary
> > > JARs at the very least, sources & javadocs may be excluded).
> >
> > the question is: where do you want to publish, with what process?
> >
> > If it's to your remote repository, maven-deploy-plugin does the job
> >
> > If it's local to your build (then you do whatever you want with the files
> > in
> > your own publication process), we use net.nicoulaj.maven.plugins:checksum-
> > maven-plugin at Apache Software Foundation parent build level to compute
> > sha512 checksum of source release archives, that are later published to
> > Apache
> > Software Distribution area.
> > see https://maven.apache.org/pom/asf/#The_apache-release_Profile and
> > corresponding pom.xml
> >
> > > I'm aware that reproducible builds[1] requires publishing checksums in a
> > > .buildinfo file and that the Maven artifacts plugin 3.0.0 will be
> >
> > released
> >
> > > soon (-ish).
> >
> > I would not say that " reproducible builds[1] requires publishing
> > checksums in
> > a .buildinfo file": Reproducible Builds require that if you build twice
> > and
> > compute checksums, you'll get the same value.
> >
> > You don't need to publish your checksums: anybody can compute checksums at
> > any
> > time, that's what Maven Artifact Plugin does on the fly both for current
> > build
> > result and from remote reference build binaries.
> > That's what is done in Reproducible Central, for example:
> > https://github.com/jvm-repo-rebuild/reproducible-central
> >
> > And to be clear, I'm not sure that publishing .buildinfo to Central
> > repository
> > is a good idea, particularly when the spec I wrote [1] is SNAPSHOT,
> > implemented only by few tools, never had any real feedback, and that I
> > implemented in Maven Artifact Plugin something that is finally different
> > from
> > the theory I wrote 1 year before in the spec.
> >
> > First part of REX: only SBT implemented .buildinfo, and published the
> > output
> > to Central for com.scalapenos:stamina releases (visible in Reproducible
> > Central) and a few other. And it showed one of the first issues: on a
> > build
> > that produces many gavs (multi-modules in Maven, multi-scala versions in
> > SBT),
> > is it better to produce 1 unique .buildinfo for the whole multi-module
> > build
> > (like I did in Maven Artifact Plugin) or to produce 1 .buildinfo file per
> > gav
> > (like done in SBT case)?
> >
> > I know that I need to take time to write some REX on the issues I
> > encountered
> > when implementing the spec.
> > If you're interested, I will probably take time to write this REX: I'm
> > tired
> > on writing many documentations that nobody gives feedback on, after having
> > really used.
> >
> > Regards,
> >
> > Hervé
> >
> > > Cheers,
> > > Andres
> > >
> > > [1] https://reproducible-builds.org/docs/jvm/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]