Re: [DISCUSS] checking reproducible builds

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
Am 2020-03-07 um 13:12 schrieb Karl Heinz Marbaise:

> Hi Hervé,
>
> I've tried to check my release via the suggested recipe...
>
>
> Downloaded the maven-studies repo and build the following commit:
> 90b426758363123af6fcc9aa7190b837c0551359 (mvn clean install)
>
> Downloaded the source package
>
> curl -O
> https://repository.apache.org/content/repositories/maven-1555/org/apache/maven/plugins/maven-dependency-plugin/3.1.2/maven-dependency-plugin-3.1.2-source-release.zip 
>
>
> unzip maven-dependency-plugin-3.1.2-source-release.zip
>
> cd maven-dependency-plugin-3.1.2 and tried to run the following:
>
> mvn -Papache-release verify buildinfo:save -Dgpg.skip
> -Dreference.repo=https://repository.apache.org/content/repositories/maven-1555/ 
>
>
> and got the following:
>
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save
> (default-cli) on project maven-dependency-plugin: Error resolving
> reference artifact
> org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2: Could
> not transfer artifact
> org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2 from/to
> reference
> (https://repository.apache.org/content/repositories/maven-1555/): Cannot
> access https://repository.apache.org/content/repositories/maven-1555/
> with type  using the available connector factories:
> BasicRepositoryConnectorFactory: Cannot access
> https://repository.apache.org/content/repositories/maven-1555/ with type
>   using the available layout factories: Maven2RepositoryLayoutFactory:
> Unsupported repository layout -> [Help 1]
> [ERROR]

Same here with Maven 3.5.4.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Romain Manni-Bucau
Hmm, thinking out loud but cant a reproducible build check just build the
project twice staging locally first artifacts and comparing second pass
outputs to the staged ones?

Le dim. 8 mars 2020 à 23:25, Hervé BOUTEMY <[hidden email]> a écrit :

> clearly, save goal is not a good choice: buildinfo would be better
>
> I know buildinfo is not a usual term, but it's widely used in Reproducible
> Builds [1] & [2], then it would be nice us Maven not to reinvent a wheel
> that
> has already been invented
>
> on separating checking, I really don't see how this improves experience
>
> I love this idea of maven-artifact-plugin, but I don't see which goals od
> maven-dependency-plugin could go in:
> https://maven.apache.org/plugins/maven-dependency-plugin/
>
> Regards,
>
> Hervé
>
> [1] https://reproducible-builds.org/docs/jvm/
>
> [2] https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles
>
> Le dimanche 8 mars 2020, 21:04:56 CET Robert Scholte a écrit :
> > I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> > That implies that the save goal should be renamed.
> > A couple of goals of the maven-dependency-plugin are actually more
> > artifact-related are might be worth moving.
> >
> > Robert
> >
> > On 8-3-2020 13:44:07, Michael Osipov <[hidden email]> wrote:
> >
> > Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> > >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> > >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> > >>>> This is expected because I am on 1.8.0_242. I don't have Java 7
> > >>>> installed anymore on the server.
> > >>>
> > >>> for the discussion I wanted us to have, just being able to test and
> see
> > >>> how we detect issues, this is perfect, isn't it?
> > >>
> > >> This is really nice. Here is the diffoscope output:
> > > you're discovering the wonders of diffoscope :)
> > >
> > >>> --- maven-site-plugin-3.9.0.jar
> > >>> +++ reference/maven-site-plugin-3.9.0.jar
> > >>> ├── zipinfo {}
> > >>> │ @@ -1,8 +1,8 @@
> > >
> > > [...]
> > >
> > >>> META-INF/MANIFEST.MF
> > >>> │ @@ -1,10 +1,10 @@
> > >>> │ Manifest-Version: 1.0
> > >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> > >>> │ Implementation-Title: Apache Maven Site Plugin
> > >>> │ Implementation-Version: 3.9.0
> > >>> │ +Build-Jdk-Spec: 1.7^M
> > >>> │ Specification-Vendor: The Apache Software Foundation
> > >>> │ -Specification-Title: Apache Maven Site Plugin^M
> > >>> │ -Build-Jdk-Spec: 1.8^M
> > >>> │ Created-By: Maven Jar Plugin 3.2.0
> > >>> │ +Specification-Title: Apache Maven Site Plugin^M
> > >>> │ Specification-Version: 3.9
> > >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> > >>
> > >> I wonder where the CRs code from...this could be the default
> > >> serialization format on every platform.
> > >
> > > FYI I don't have such CRs in output on my Linux box
> >
> > This cannot be. See
> >
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/sha
> > re/classes/java/util/jar/Manifest.java and search for \r\n. Old Sun code
> > uses *always* CRLF. Plase recheck on your side and run a hexdump on the
> > Manifest file.
> >
> > >>> how did you find the experience? any improvement proposal?
> > >>> and any idea on where to put this goal in the future?
> > >>
> > >> There is room for improvement when I quickly read the code. I will
> write
> > >> separately on this.
> > >
> > > sure, code can be improved: don't hesitate
> > > but I was not asking yet for code improvement (I'm confident, it will
> > > happen) but *experience* improvement
> > >
> > >> I'd leave as a plugin for now.
> > >
> > > you mean a separate plugin? same "buildinfo" name as current? "save"
> goal
> > > name?
> >
> > OK, let's talk about experience:
> >
> > * buildinfo may be changed to broader name, e.g.,
> > maven-reproducibility-plugin. Explanain follows
> > * 'save' does too much. It should save only and not compare. Save should
> > either run at initialize or at build-resources phase, imho
> > * Add a 'compare' goal, not phase bound. It performs the actual
> comparsion.
> >
> > Strictly speaking if the plugin is called buildinfo it should handle the
> > buildinfo files only.
> >
> > >> At least in 3.7.x.
> > >
> > > 3.7.x as Maven 3.7.x?
> > > does that mean that you think it should be one day integrated into
> Maven
> > > core? what's the rationale?
> >
> > Not really, but if this happens, not before 4.x. I don't have any
> > rationale or entry point for this yet.
> >
> > Michael
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Hervé BOUTEMY
In reply to this post by michaelo
please "git pull": you're one commit behind HEAD
https://github.com/apache/maven-studies/commits/maven-buildinfo-plugin

----- Mail original -----
De: "Karl Heinz Marbaise" <[hidden email]>
À: "Maven Developers List" <[hidden email]>, "Hervé BOUTEMY" <[hidden email]>
Envoyé: Samedi 7 Mars 2020 12:12:08
Objet: Re: [DISCUSS] checking reproducible builds

Hi Hervé,

I've tried to check my release via the suggested recipe...


Downloaded the maven-studies repo and build the following commit:
90b426758363123af6fcc9aa7190b837c0551359 (mvn clean install)

Downloaded the source package

curl -O
https://repository.apache.org/content/repositories/maven-1555/org/apache/maven/plugins/maven-dependency-plugin/3.1.2/maven-dependency-plugin-3.1.2-source-release.zip

unzip maven-dependency-plugin-3.1.2-source-release.zip

cd maven-dependency-plugin-3.1.2 and tried to run the following:

mvn -Papache-release verify buildinfo:save -Dgpg.skip
-Dreference.repo=https://repository.apache.org/content/repositories/maven-1555/

and got the following:


[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save
(default-cli) on project maven-dependency-plugin: Error resolving
reference artifact
org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2: Could
not transfer artifact
org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2 from/to
reference
(https://repository.apache.org/content/repositories/maven-1555/): Cannot
access https://repository.apache.org/content/repositories/maven-1555/
with type  using the available connector factories:
BasicRepositoryConnectorFactory: Cannot access
https://repository.apache.org/content/repositories/maven-1555/ with type
  using the available layout factories: Maven2RepositoryLayoutFactory:
Unsupported repository layout -> [Help 1]
[ERROR]



Kind regards
Karl Heinz Marbaise

On 07.03.20 11:36, Hervé BOUTEMY wrote:

> Hi,
>
> Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository.
>
> For a live example, see the last paragraph of Maven Site Plugin vote that just started [1].
>
> Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]:
> 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository
> 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference.
>
> Now I want to discuss: is it clear? can you test and report, please?
>
> If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository.
>
> Thanks for your feedback
>
> Regards,
>
> Hervé
>
> [1] https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E
>
> [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]