Re: last review of Reproducible Builds proposal

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Vladimir Sitnikov
OpenJDK8 uses hashmap for manifest entries, so jar files are not really
reproducible there.

Note: there's run-to-run randomization in j.u.HashMap, so the manifest
contents is not predictable.

Vladimir
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :

> Hi Herve,
>
> thanks for that will try it asap and report any findings back.
>
> But good to know that there is a difference between JDK major versions and
> OSes ... so it would probably be best to stage releases on Linux with an
> OpenJDK of the minimum supported version?
> Just thinking how to make it
> possible to verify without having to buy Mac or Windows licenses ... guess
> on every machine you could whip up a Ubuntu VM for verification. Just
> thinking about it ... perhaps it would be best to create a Docker image for
> doing the reproducible stuff ...
just to be clear: the difference is on newline only, then Windows vs anything
else. You get the same result on Linux, MacOS, FreeBSD, or any other Unix.

And if I want to be complete, if you get a source tarball from Windows,
extract it to a Linux/MacOS/... box and build with "mvn -
Dline.separator='\r\n'", in general, you get the same result as a build under
Windows: I tested multiple times, it worked, but we'll see if it works always
or just "in general"

> Are there any plans on creating a plugin to allow verification?
>
> Sort of something like this:
> "mvn package release:verify-reproduicble
> -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
> (Which doesn't deploy the
> artifacts, but instead download them and do a binary comparison)
yes, but for now it was completely another form: this is the "Buildinfo file"
proposal https://reproducible-builds.org/docs/jvm/
that I stopped to work on 1 year ago given I had no chance to get the same
output: it's now a good time to restart working on it as next steps


> Also it could be great if the release-plugin could automatically set the
> property:
> a) if it finds the "project.build.outputTimestamp" set to some
> placeholder value b) if some switch tells it to prepare a reproducible
> build by using some sort of "switch" parameter
> Guess that would sort of close the loop to get the biggest benefit out of
> the reproducible builds.
+1
issue has been created https://issues.apache.org/jira/browse/MRELEASE-1029
But I didn't work on it yet, help welcome

> I would be happy to help as I think this is one
> of the greatest new features. (Ok ... perhaps besides the
> sound-output-extension I learned about yesterday ;-) )
+1
the current step is important, but it's only the beginning of the story from
an effective usage perspective

Regards,

Hervé

>
> Chris
>
>
> Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <[hidden email]>:
>
>     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
>
>     > Hi all,
>     >
>     > as I can see you're voting on releasing the reproducible build
>     > extended
>     > plugin versions.
>
>      Is there any documentation on how to use this new
>
>     > feature?
>     >
>     > I had a look at the confluence page, but that seemed like a
>     > brainstorming
>     > session.
>
>     ok, the Wiki page [1] started as a brainstorming session, was updated to
> a proposal (the "Output Archive Entries Timestamp" parapgraph).
 And now I
> probably should order paragraph a little bit, and add a "Making your build
> reproducible" section for end uses to have a quick explanation.
>     I'll write the explanation here as a first try before working on the
> Wiki:
 
>     1. upgrade your plugins to reproducible version, particularly
> mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion
> 3.2.0
 2. add project.build.outputTimestamp property with the timestamp
> value that will be used in zip/jar/tar archives: <properties>
>        
> <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTi
> mestamp>
 </properties>
>    
>     Notice:
>     - there is no Maven version prerequisite, everything happens at plugins
> level
 - Reproducible Builds require to have no version ranges in
> dependencies, generally gives different result on Unixes vs Windows, and
> generally depend on the major version of JDK used to compile (see "Out of
> Scope" paragraph)
>     You have the basis configured, the output should be reproducible now.
>     If something is still not reproducible, use diffoscope to find the
> unstable output, find the plugin that generated this output and check if
> there is a reproducible version available: if not, please open an issue to
> help plugin maintainers improving Reproducible Builds support at every
> plugin level.
 
>     [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
 

>    
>      
>
>     > I would love to add this to the PLC4X build asap.
>
>     I'd love to have feedback
>     Don't hesitate to ping me.
>
>     >
>     > So I would like to test the release-candidates and vote too.
>
>     I would love to have many tester and votes :)
>    
>
>     >
>     > Chris
>     >
>     >
>     >
>     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <[hidden email]>:
>     >
>     >
>     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
>     >
>     >
>     >
>     >     > Emmanuel Bourg wrote:
>     >     >
>     >     >
>     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
>     >     > >
>     >     > >
>     >     > >> last question: now that we seem to fully understand each
>     >     > >> other,
>     >     > >> does it
>     >     > >> mean that you don't need any more "seconds since the epoch"
>     >     > >> format
>     >     > >> support for the property?
>     >     > >
>     >     > >
>     >     > >
>     >     > > If Maven supports the SOURCE_DATE_EPOCH environment variable
>     >     > > I
>     >     > > don't
>     >     > > think that's necessary, otherwise it would be nice to be able
>     >     > > to
>     >     > > invoke
>     >     > >
>     >     > > Maven with:
>     >     > >
>     >     > >
>     >     > >    mvn package
>     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
>     >     > >
>     >     > >
>     >     > >
>     >     > > and this means supporting a timestamp formatted as seconds
>     >     > > since
>     >     > > the
>     >     > > epoch.
>     >     >
>     >     >
>     >     >
>     >     > +1
>     >     >
>     >     > The above would be a nice, simple way of bridging the gap
>     >     > between
>     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
>     >
>     >
>     >
>     >     told like that, ok, why not
>     >    
>     >
>     >
>     >
>     >     >
>     >     > If it is not too much trouble to implement the "\d+ -> seconds
>     >     > since
>     >     > epoch" heuristic, them I would love to see it included.
>     >
>     >
>     >
>     >     ok, I'll do and prepare the release
>     >    
>     >     Regards,
>     >    
>     >     Hervé
>     >    
>     >
>     >
>     >
>     >     >
>     >     > Best wishes,
>     >     >
>     >     > Andreas
>     >
>     >
>     >
>     >    
>     >    
>     >    
>     >    
>     >    
>     >     ------------------------------------------------------------------
>     >     ---
>     >     To unsubscribe, e-mail: [hidden email]
>     >     For additional commands, e-mail: [hidden email]
>     >    
>     >    
>     >
>     >
>     >
>     > ---------------------------------------------------------------------
>     > To unsubscribe, e-mail: [hidden email]
>     > For additional commands, e-mail: [hidden email]
>     >
>
>    
>    
>    
>    
>    
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: [hidden email]
>     For additional commands, e-mail: [hidden email]
>    
>    
>





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
In reply to this post by Vladimir Sitnikov
Well I have a new candidate:

<artifactId>maven-bundle-plugin</artifactId>
 
Creates: Bnd-LastModified in the MANIFEST.MF

Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)

Chris



Am 03.11.19, 20:25 schrieb "[hidden email]" <[hidden email]>:

    ok, Reproducible Builds are not so easy to get: each plugin that you use can cause an issue
   
    I really recommend you diffoscope to investigate differences, it really helps a lot by easily giving you precise differences
   
    Good luck for the investigations :)
   
    Regards,
   
    Hervé
   
    ----- Mail original -----
    De: "Christofer Dutz" <[hidden email]>
    À: "Maven Developers List" <[hidden email]>
    Envoyé: Dimanche 3 Novembre 2019 20:16:42
    Objet: Re: last review of Reproducible Builds proposal
   
    Hi Herve,
   
    unfortunately I now have implemented some tooling to compare the stuff produced by the updated reproducible builds.
    And it does show quite a number of non binary equal files.
   
    Will investigate what's the difference.
   
    Chris
   
   
   
    Am 03.11.19, 11:08 schrieb "Hervé BOUTEMY" <[hidden email]>:
   
        great feedback, thank you: this proves the initial set of only 3 plugins is a
        good basis.
        For checking the many output artifacts from a build, future buildinfo file [1]
        should help a lot. I still have many usability concerns for Maven multi-module
        builds (starting with: should we generate only the root buildinfo or one
        buildinfo per Maven module?)
        Let's keep the ball rolling: today, it's plugins release day!!!
       
        Regards,
       
        Hervé
       
        [1] https://reproducible-builds.org/docs/jvm/
       
        Le vendredi 1 novembre 2019, 13:34:32 CET Christofer Dutz a écrit :
        > Hi,
        >
        > so I just updated the versions of the 3 plugins and gave the Apache PLC4X
        > build a little spin ...
        > https://github.com/apache/plc4x/tree/feature/reproducible-builds
        >
        > Did two executions of this:
        > mvn -U -P
        > with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-san
        > dbox,with
        > -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-di
        > r clean deploy
         (but with a different second local repo location)
        >
        > Then I used "diff" and "cmp" to compare individual files and it didn't show
        > differences with the artifacts I chose ...
         now I guess I'll have to whip
        > up some little bash script to sort of compare the artifacts from one
        > directory with the other (Unfortunately the SNAPSHOT timestamps are a
        > little annoying :-/
        > We do have some C++, C# and Python modules ... but I wouldn't expect the C++
        > and C# to be reproducible.
         
        > Chris
        >
        >
        > Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <[hidden email]>:
        >
        >     Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
        >
        >     > Hi Herve,
        >     >
        >     > thanks for that will try it asap and report any findings back.
        >     >
        >     > But good to know that there is a difference between JDK major versions
        >     > and
        >     > OSes ... so it would probably be best to stage releases on Linux with
        >     > an
        >     > OpenJDK of the minimum supported version?
        >     > Just thinking how to make it
        >     > possible to verify without having to buy Mac or Windows licenses ...
        >     > guess
        >     > on every machine you could whip up a Ubuntu VM for verification. Just
        >     > thinking about it ... perhaps it would be best to create a Docker
        >     > image for
         doing the reproducible stuff ...
        >
        >     just to be clear: the difference is on newline only, then Windows vs
        > anything
         else. You get the same result on Linux, MacOS, FreeBSD, or any
        > other Unix.
        >     And if I want to be complete, if you get a source tarball from Windows,
        >
         extract it to a Linux/MacOS/... box and build with "mvn -
        >     Dline.separator='\r\n'", in general, you get the same result as a build
        > under
         Windows: I tested multiple times, it worked, but we'll see if it
        > works always or just "in general"
        >    
        >
        >     > Are there any plans on creating a plugin to allow verification?
        >     >
        >     > Sort of something like this:
        >     > "mvn package release:verify-reproduicble
        >     > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
        >     > (Which doesn't deploy the
        >     > artifacts, but instead download them and do a binary comparison)
        >
        >     yes, but for now it was completely another form: this is the "Buildinfo
        > file"
         proposal https://reproducible-builds.org/docs/jvm/
        >     that I stopped to work on 1 year ago given I had no chance to get the
        > same
         output: it's now a good time to restart working on it as next steps
        >    
        >
        >     > Also it could be great if the release-plugin could automatically set
        >     > the
        >     > property:
        >     > a) if it finds the "project.build.outputTimestamp" set to some
        >     > placeholder value b) if some switch tells it to prepare a
        >     > reproducible
        >     > build by using some sort of "switch" parameter
        >     > Guess that would sort of close the loop to get the biggest benefit out
        >     > of
        >     > the reproducible builds.
        >
        >     +1
        >     issue has been created
        > https://issues.apache.org/jira/browse/MRELEASE-1029
         But I didn't work on
        > it yet, help welcome
        >    
        >
        >     > I would be happy to help as I think this is one
        >     > of the greatest new features. (Ok ... perhaps besides the
        >     > sound-output-extension I learned about yesterday ;-) )
        >
        >     +1
        >     the current step is important, but it's only the beginning of the story
        > from
         an effective usage perspective
        >    
        >     Regards,
        >    
        >     Hervé
        >    
        >
        >     >
        >     > Chris
        >     >
        >     >
        >     > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <[hidden email]>:
        >     >
        >     >
        >     >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
        >     >
        >     >
        >     >
        >     >     > Hi all,
        >     >     >
        >     >     > as I can see you're voting on releasing the reproducible build
        >     >     > extended
        >     >     > plugin versions.
        >     >
        >     >
        >     >
        >     >      Is there any documentation on how to use this new
        >     >
        >     >
        >     >
        >     >     > feature?
        >     >     >
        >     >     > I had a look at the confluence page, but that seemed like a
        >     >     > brainstorming
        >     >     > session.
        >     >
        >     >
        >     >
        >     >     ok, the Wiki page [1] started as a brainstorming session, was
        >     >     updated to
        >     >
        >     > a proposal (the "Output Archive Entries Timestamp" parapgraph).
        >
        >      And now I
        >
        >     > probably should order paragraph a little bit, and add a "Making your
        >     > build
        >     > reproducible" section for end uses to have a quick explanation.
        >     >
        >     >     I'll write the explanation here as a first try before working on
        >     >     the
        >     >
        >     > Wiki:
        >
        >      
        >
        >     >     1. upgrade your plugins to reproducible version, particularly
        >     >
        >     > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to
        >     > vesion
        >     > 3.2.0
        >
        >      2. add project.build.outputTimestamp property with the timestamp
        >
        >     > value that will be used in zip/jar/tar archives: <properties>
        >     >
        >     >        
        >     >
        >     > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.out
        >     > putTi
         mestamp>
        >
        >      </properties>
        >
        >     >    
        >     >     Notice:
        >     >     - there is no Maven version prerequisite, everything happens at
        >     >     plugins
        >     >
        >     > level
        >
        >      - Reproducible Builds require to have no version ranges in
        >
        >     > dependencies, generally gives different result on Unixes vs Windows,
        >     > and
        >     > generally depend on the major version of JDK used to compile (see "Out
        >     > of
        >     > Scope" paragraph)
        >     >
        >     >     You have the basis configured, the output should be reproducible
        >     >     now.
        >     >     If something is still not reproducible, use diffoscope to find
        >     >     the
        >     >
        >     > unstable output, find the plugin that generated this output and check
        >     > if
        >     > there is a reproducible version available: if not, please open an
        >     > issue to
        >     > help plugin maintainers improving Reproducible Builds support at
        >     > every
        >     > plugin level.
        >
        >      
        >
        >     >     [1]
        >     >
        >     > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682
        >     > 318
        >
        >      
        >
        >     >    
        >     >    
        >     >      
        >     >
        >     >
        >     >
        >     >     > I would love to add this to the PLC4X build asap.
        >     >
        >     >
        >     >
        >     >     I'd love to have feedback
        >     >     Don't hesitate to ping me.
        >     >
        >     >
        >     >
        >     >     >
        >     >     > So I would like to test the release-candidates and vote too.
        >     >
        >     >
        >     >
        >     >     I would love to have many tester and votes :)
        >     >    
        >     >
        >     >
        >     >
        >     >     >
        >     >     > Chris
        >     >     >
        >     >     >
        >     >     >
        >     >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY"
        >     >     > <[hidden email]>:
        >     >     >
        >     >     >
        >     >     >
        >     >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a
        >     >     >     écrit :
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >     > Emmanuel Bourg wrote:
        >     >     >     >
        >     >     >     >
        >     >     >     >
        >     >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >> last question: now that we seem to fully understand
        >     >     >     > >> each
        >     >     >     > >> other,
        >     >     >     > >> does it
        >     >     >     > >> mean that you don't need any more "seconds since the
        >     >     >     > >> epoch"
        >     >     >     > >> format
        >     >     >     > >> support for the property?
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >
        >     >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment
        >     >     >     > > variable
        >     >     >     > > I
        >     >     >     > > don't
        >     >     >     > > think that's necessary, otherwise it would be nice to be
        >     >     >     > > able
        >     >     >     > > to
        >     >     >     > > invoke
        >     >     >     > >
        >     >     >     > > Maven with:
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >    mvn package
        >     >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >
        >     >     >     > >
        >     >     >     > > and this means supporting a timestamp formatted as
        >     >     >     > > seconds
        >     >     >     > > since
        >     >     >     > > the
        >     >     >     > > epoch.
        >     >     >     >
        >     >     >     >
        >     >     >     >
        >     >     >     >
        >     >     >     > +1
        >     >     >     >
        >     >     >     > The above would be a nice, simple way of bridging the gap
        >     >     >     > between
        >     >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >     told like that, ok, why not
        >     >     >    
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >     >
        >     >     >     > If it is not too much trouble to implement the "\d+ ->
        >     >     >     > seconds
        >     >     >     > since
        >     >     >     > epoch" heuristic, them I would love to see it included.
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >     ok, I'll do and prepare the release
        >     >     >    
        >     >     >     Regards,
        >     >     >    
        >     >     >     Hervé
        >     >     >    
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >     >
        >     >     >     > Best wishes,
        >     >     >     >
        >     >     >     > Andreas
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     >    
        >     >     >    
        >     >     >    
        >     >     >    
        >     >     >    
        >     >     >     ------------------------------------------------------------
        >     >     >     ------
        >     >     >     ---
        >     >     >     To unsubscribe, e-mail: [hidden email]
        >     >     >     For additional commands, e-mail: [hidden email]
        >     >     >    
        >     >     >    
        >     >     >
        >     >     >
        >     >     >
        >     >     >
        >     >     > ----------------------------------------------------------------
        >     >     > -----
        >     >     > To unsubscribe, e-mail: [hidden email]
        >     >     > For additional commands, e-mail: [hidden email]
        >     >     >
        >     >
        >     >
        >     >
        >     >    
        >     >    
        >     >    
        >     >    
        >     >    
        >     >     ------------------------------------------------------------------
        >     >     ---
        >     >     To unsubscribe, e-mail: [hidden email]
        >     >     For additional commands, e-mail: [hidden email]
        >     >    
        >     >    
        >     >
        >     >
        >
        >    
        >    
        >    
        >    
        >    
        >     ---------------------------------------------------------------------
        >     To unsubscribe, e-mail: [hidden email]
        >     For additional commands, e-mail: [hidden email]
        >    
        >    
        >
       
       
       
       
       
        ---------------------------------------------------------------------
        To unsubscribe, e-mail: [hidden email]
        For additional commands, e-mail: [hidden email]
       
       
   
   
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]
   
   
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]
   
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]