Re: last review of Reproducible Builds proposal

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
Hi,

so I just updated the versions of the 3 plugins and gave the Apache PLC4X build a little spin ...
https://github.com/apache/plc4x/tree/feature/reproducible-builds

Did two executions of this:
mvn -U -P with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-sandbox,with -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-dir clean deploy
 (but with a different second local repo location)

Then I used "diff" and "cmp" to compare individual files and it didn't show differences with the artifacts I chose ...
now I guess I'll have to whip up some little bash script to sort of compare the artifacts from one directory with the other (Unfortunately the SNAPSHOT timestamps are a little annoying :-/

We do have some C++, C# and Python modules ... but I wouldn't expect the C++ and C# to be reproducible.

Chris


Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <[hidden email]>:

    Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
    > Hi Herve,
    >
    > thanks for that will try it asap and report any findings back.
    >
    > But good to know that there is a difference between JDK major versions and
    > OSes ... so it would probably be best to stage releases on Linux with an
    > OpenJDK of the minimum supported version?
    > Just thinking how to make it
    > possible to verify without having to buy Mac or Windows licenses ... guess
    > on every machine you could whip up a Ubuntu VM for verification. Just
    > thinking about it ... perhaps it would be best to create a Docker image for
    > doing the reproducible stuff ...
    just to be clear: the difference is on newline only, then Windows vs anything
    else. You get the same result on Linux, MacOS, FreeBSD, or any other Unix.
   
    And if I want to be complete, if you get a source tarball from Windows,
    extract it to a Linux/MacOS/... box and build with "mvn -
    Dline.separator='\r\n'", in general, you get the same result as a build under
    Windows: I tested multiple times, it worked, but we'll see if it works always
    or just "in general"
   
    > Are there any plans on creating a plugin to allow verification?
    >
    > Sort of something like this:
    > "mvn package release:verify-reproduicble
    > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
    > (Which doesn't deploy the
    > artifacts, but instead download them and do a binary comparison)
    yes, but for now it was completely another form: this is the "Buildinfo file"
    proposal https://reproducible-builds.org/docs/jvm/
    that I stopped to work on 1 year ago given I had no chance to get the same
    output: it's now a good time to restart working on it as next steps
   
   
    > Also it could be great if the release-plugin could automatically set the
    > property:
    > a) if it finds the "project.build.outputTimestamp" set to some
    > placeholder value b) if some switch tells it to prepare a reproducible
    > build by using some sort of "switch" parameter
    > Guess that would sort of close the loop to get the biggest benefit out of
    > the reproducible builds.
    +1
    issue has been created https://issues.apache.org/jira/browse/MRELEASE-1029
    But I didn't work on it yet, help welcome
   
    > I would be happy to help as I think this is one
    > of the greatest new features. (Ok ... perhaps besides the
    > sound-output-extension I learned about yesterday ;-) )
    +1
    the current step is important, but it's only the beginning of the story from
    an effective usage perspective
   
    Regards,
   
    Hervé
   
    >
    > Chris
    >
    >
    > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <[hidden email]>:
    >
    >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
    >
    >     > Hi all,
    >     >
    >     > as I can see you're voting on releasing the reproducible build
    >     > extended
    >     > plugin versions.
    >
    >      Is there any documentation on how to use this new
    >
    >     > feature?
    >     >
    >     > I had a look at the confluence page, but that seemed like a
    >     > brainstorming
    >     > session.
    >
    >     ok, the Wiki page [1] started as a brainstorming session, was updated to
    > a proposal (the "Output Archive Entries Timestamp" parapgraph).
     And now I
    > probably should order paragraph a little bit, and add a "Making your build
    > reproducible" section for end uses to have a quick explanation.
    >     I'll write the explanation here as a first try before working on the
    > Wiki:
     
    >     1. upgrade your plugins to reproducible version, particularly
    > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion
    > 3.2.0
     2. add project.build.outputTimestamp property with the timestamp
    > value that will be used in zip/jar/tar archives: <properties>
    >        
    > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTi
    > mestamp>
     </properties>
    >    
    >     Notice:
    >     - there is no Maven version prerequisite, everything happens at plugins
    > level
     - Reproducible Builds require to have no version ranges in
    > dependencies, generally gives different result on Unixes vs Windows, and
    > generally depend on the major version of JDK used to compile (see "Out of
    > Scope" paragraph)
    >     You have the basis configured, the output should be reproducible now.
    >     If something is still not reproducible, use diffoscope to find the
    > unstable output, find the plugin that generated this output and check if
    > there is a reproducible version available: if not, please open an issue to
    > help plugin maintainers improving Reproducible Builds support at every
    > plugin level.
     
    >     [1]
    > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
     
    >    
    >      
    >
    >     > I would love to add this to the PLC4X build asap.
    >
    >     I'd love to have feedback
    >     Don't hesitate to ping me.
    >
    >     >
    >     > So I would like to test the release-candidates and vote too.
    >
    >     I would love to have many tester and votes :)
    >    
    >
    >     >
    >     > Chris
    >     >
    >     >
    >     >
    >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <[hidden email]>:
    >     >
    >     >
    >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
    >     >
    >     >
    >     >
    >     >     > Emmanuel Bourg wrote:
    >     >     >
    >     >     >
    >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
    >     >     > >
    >     >     > >
    >     >     > >> last question: now that we seem to fully understand each
    >     >     > >> other,
    >     >     > >> does it
    >     >     > >> mean that you don't need any more "seconds since the epoch"
    >     >     > >> format
    >     >     > >> support for the property?
    >     >     > >
    >     >     > >
    >     >     > >
    >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment variable
    >     >     > > I
    >     >     > > don't
    >     >     > > think that's necessary, otherwise it would be nice to be able
    >     >     > > to
    >     >     > > invoke
    >     >     > >
    >     >     > > Maven with:
    >     >     > >
    >     >     > >
    >     >     > >    mvn package
    >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
    >     >     > >
    >     >     > >
    >     >     > >
    >     >     > > and this means supporting a timestamp formatted as seconds
    >     >     > > since
    >     >     > > the
    >     >     > > epoch.
    >     >     >
    >     >     >
    >     >     >
    >     >     > +1
    >     >     >
    >     >     > The above would be a nice, simple way of bridging the gap
    >     >     > between
    >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
    >     >
    >     >
    >     >
    >     >     told like that, ok, why not
    >     >    
    >     >
    >     >
    >     >
    >     >     >
    >     >     > If it is not too much trouble to implement the "\d+ -> seconds
    >     >     > since
    >     >     > epoch" heuristic, them I would love to see it included.
    >     >
    >     >
    >     >
    >     >     ok, I'll do and prepare the release
    >     >    
    >     >     Regards,
    >     >    
    >     >     Hervé
    >     >    
    >     >
    >     >
    >     >
    >     >     >
    >     >     > Best wishes,
    >     >     >
    >     >     > Andreas
    >     >
    >     >
    >     >
    >     >    
    >     >    
    >     >    
    >     >    
    >     >    
    >     >     ------------------------------------------------------------------
    >     >     ---
    >     >     To unsubscribe, e-mail: [hidden email]
    >     >     For additional commands, e-mail: [hidden email]
    >     >    
    >     >    
    >     >
    >     >
    >     >
    >     > ---------------------------------------------------------------------
    >     > To unsubscribe, e-mail: [hidden email]
    >     > For additional commands, e-mail: [hidden email]
    >     >
    >
    >    
    >    
    >    
    >    
    >    
    >     ---------------------------------------------------------------------
    >     To unsubscribe, e-mail: [hidden email]
    >     For additional commands, e-mail: [hidden email]
    >    
    >    
    >
   
   
   
   
   
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]
   
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

AW: last review of Reproducible Builds proposal

Christofer Dutz
Hi all,

And I might want to add, that includes pom, jar, sources jar, javadoc, release-asembly and fat-jars ...

Currently applying these changes to other apache projects as an experiment.

So is anyone already working on the release plugin?

Chris
________________________________
Von: Christofer Dutz <[hidden email]>
Gesendet: Montag, 4. November 2019 18:58:10
An: Maven Developers List <[hidden email]>
Betreff: Re: last review of Reproducible Builds proposal

Hi all,

so today I made sure the LastModified and the Creator username are omitted and now the Apache PLC4X build had a 100% reproducible build (at least on my one machine) ... will be checking this out on other machines.

Chris

Am 04.11.19, 10:12 schrieb "Christofer Dutz" <[hidden email]>:

    Hi Andreas,

    but the format of the timestamp is completely different ... doesn't that matter?
    I was currently going for the option number 1 with removing the header.

    In order to be 100% happy with this, I would prefer a setup where the normal mechanisms are used if no maven.build.outputTimestap is defined, but if it is (because a future version of the maven release plugin set it there) it uses that.

    Will try out your suggestions as soon as I'm able to build again (I unfortunately installed that Mac OS update ... now things aren't working as they should be)

    Chris



    Am 04.11.19, 09:42 schrieb "Andreas Sewe" <[hidden email]>:

        Christofer Dutz wrote:
        > Well I have a new candidate:
        >
        > <artifactId>maven-bundle-plugin</artifactId>
        >
        > Creates: Bnd-LastModified in the MANIFEST.MF
        >
        > Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)

        <configuration>
          <instructions>
            <_removeheaders>Bnd-LastModified</_removeheaders>
          </instructions>
        <configuration>

        Alternatively

        <configuration>
          <instructions>
            <Bnd-LastModified>${maven.build.outputTimestap}</Bnd-LastModified>
          </instructions>
        <configuration

        Hope that helps,

        Andreas

        ---------------------------------------------------------------------
        To unsubscribe, e-mail: [hidden email]
        For additional commands, e-mail: [hidden email]




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]