last review of Reproducible Builds proposal

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

last review of Reproducible Builds proposal

Hervé BOUTEMY
based on the feedback I got, I updated the proposal:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318

The archives entries timestamp is now configured with project.build.outputTimestamp property, in ISO-8601 format
<project>
  <properties>
    <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
  </properties>
</project>

The shared components, plugins, parent poms and Maven core branches have been updated to match this new proposal


If no one objects, next week-end, I'll start the (heavy) release train to bring (binary) Reproducible Builds plugins to general availability

Regards,

Hervé



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Michael Osipov-2
Am 2019-10-05 um 19:52 schrieb Hervé BOUTEMY:

> based on the feedback I got, I updated the proposal:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
>
> The archives entries timestamp is now configured with project.build.outputTimestamp property, in ISO-8601 format
> <project>
>    <properties>
>      <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
>    </properties>
> </project>
>
> The shared components, plugins, parent poms and Maven core branches have been updated to match this new proposal
>
>
> If no one objects, next week-end, I'll start the (heavy) release train to bring (binary) Reproducible Builds plugins to general availability

Really, really nice work..

I do like <project.build.outputTimestamp>. Why did you make it a String?
Why not go directly with Instant? It gives your ISO 8601 for free

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Vladimir Sitnikov
In reply to this post by Hervé BOUTEMY
>but
>who really looks at the timestamp of entries in release zips/jars/tar.gz
>honestly?

Tomcat when it decides on what to send in the "Last-Modified" header.
For instance, current Gradle does not allow to configure the timestamp, and
for reproducible builds it always sets the timestamp to 0 or so.
It breaks Tomcat's assumptions:
https://github.com/gradle/gradle/issues/10917

Vladimir
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
Le dimanche 6 octobre 2019, 22:18:59 CEST Emmanuel Bourg a écrit :

> Le 06/10/2019 à 20:13, Hervé BOUTEMY a écrit :
> > no, it does not add any dependency on developer configuration:
> > 2019-10-05T18:37:42Z == 2019-10-05T20:37:42+02:00 ==
> > 2019-10-05T16:37:42-02:00
> yes but:
>
>  "2019-10-05T18:37:42Z" != "2019-10-05T20:37:42+02:00" !=
> "2019-10-05T16:37:42-02:00"
>
> The point is, two developers may generate a different pom if the local
> timezone is used. A fixed timezone is necessary to ensure the pom itself
> is reproducible.
There is a misunderstanding here: the pom.xml is saved in the source control,
with the timestamp in it.
There is no question of "reproducible pom.xml"

>
> > when will this value be written in the pom.xml is independant: currently,
> > in my PoC, I wrote the values by hand. In the future, it will probably be
> > updated by maven-release-plugin, and we'll have to choose if the
> > timestamp is written in Z or if it is written in local timezone with its
> > offset: both ways of expressing the timestamp are valid and will give
> > reproducible result
> The jar generated is reproducible, but not the pom. I suspect that if
> the jar includes the pom it'll break its reproducibility too (this is
> the default for maven-jar-plugin, but I don't know if it embeds the
> original pom without the timestamp, or the generated pom with the
> timestamp).
>
> > once again, war files taken apart for web servers, who looks at timestamp
> > in zip files?
>
> archive timestamps are just the tip of the iceberg. There are more
> visible timestamps elsewhere, for example in the javadoc headers, in
> .properties files, in OSGi attributes, sometimes in the source files...
Sure, many plugins have already been modified to remove such noise in output,
and probably others will require to be updated. Because in Reproducible
Builds, timestamps to know when something has been generated is less useful:
Reproduciblity is the ultimate proof of knowledge of what has been built,
since the result will be the same now, tomorrow, in one year...

Regards,

Hervé


>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Emmanuel Bourg
In reply to this post by Hervé BOUTEMY
Le 07/10/2019 à 14:40, Andreas Sewe a écrit :

> - Can we get something analogous to maven.build.timestamp.format? One
> could then write the following to be compatible with env.SOURCE_DATE_EPOCH:
>
> <project>
>   <properties>
>
> <project.build.outputTimestamp>${env.SOURCE_DATE_EPOCH}<project.build.outputTimestamp>
>
> <project.build.outputTimestamp.format>...</project.build.outputTimestamp.format>
>   </properties>
> </project>

+1, some kind of support for SOURCE_DATE_EPOCH would be nice. Either
this (but maybe with only one property supporting both formats) or by
overriding automatically the property when SOURCE_DATE_EPOCH is set.

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
Le mardi 8 octobre 2019, 23:42:55 CEST Mark Derricutt a écrit :

> On 6 Oct 2019, at 9:14, Hervé BOUTEMY wrote:
> > if anybody cares about the exact value: but
> > who really looks at the timestamp of entries in release zips/jars/tar.gz
> > honestly?
>
> I've actually done so in the past trying to find differences between two
> versions of a jar for repair reasons.
>
> VERY infrequent tho - if you're wanting to generate a delta-patch between
> two jars/zips then I guess it might also be handy (altho you might be
> better off going direct for checksumdifferences  there ).
FYI, when you try to know the difference between 2 archives, the ideal tool is
diffoscope [1]: it has been done exactly for that

[1] https://diffoscope.org/

>
>
> ---
> "The ease with which a change can be implemented has no relevance at all to
> whether it is the right change for the (Java) Platform for all time."
> &mdash; Mark Reinhold.
>
> Mark Derricutt
> http://www.theoryinpractice.net
> http://www.chaliceofblood.net
> http://plus.google.com/+MarkDerricutt
> http://twitter.com/talios
> http://facebook.com/mderricutt





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
In reply to this post by Hervé BOUTEMY
Just a small question. I have been following this thread with great interest.

I think this is going to be a big thing when it makes the changes available to the main maven system.

So as far as I understand the core part will be a fixed timestamp which will then be used throughout the build by multiple pluggins.

So if I provide the same timestamp the result should be binary identical.

Would it be possible to have this timestamp written/updated in the pom as part of the release:prepare step?

Sort of setting it to some constant (ie REPODUCEABLE_BUILD_TIMESTAMP) simply takes the current time but if there is a concrete value, it uses that instead?

Hope im not asking anything obvious. To me it looked as if the timestamp has to be manipulated manually.

Chris

Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
________________________________
From: Emmanuel Bourg <[hidden email]>
Sent: Thursday, October 10, 2019 11:50:34 PM
To: [hidden email] <[hidden email]>
Subject: Re: last review of Reproducible Builds proposal

Le 10/10/2019 à 19:28, Hervé BOUTEMY a écrit :

> the only little mis-interpretation is that it is a pure build information,
> then I don't see why this property would appear in a consumer POM

Thank you for the clarification, that makes perfectly sense. And I now
see the benefit of using a property that can be inherited. In a multi
modules project it's only necessary to define the timestamp once in the
root pom. Parent poms deployed to Maven Central will never include a
timestamp and there is no risk of affecting other projects deriving from
the pom.

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
Hi Enrico,

so I would definitely +1 to be able to do it this way ...
I think reproducible builds are important for releases, but I am not that sure the same applies for the daily business in a project.
For being able to do releases it would be a huge improvement, if this is automatically handled the same way the versions are updated.

For the plc4x project I was even thinking of building some maven tooling to automatically verify the built archives with the ones staged in nexus for binary equality.

Chris

Am 11.10.19, 09:05 schrieb "Enrico Olivelli" <[hidden email]>:

    Il ven 11 ott 2019, 08:31 Christofer Dutz <[hidden email]> ha
    scritto:
   
    > Just a small question. I have been following this thread with great
    > interest.
    >
    > I think this is going to be a big thing when it makes the changes
    > available to the main maven system.
    >
    > So as far as I understand the core part will be a fixed timestamp which
    > will then be used throughout the build by multiple pluggins.
    >
    > So if I provide the same timestamp the result should be binary identical.
    >
    > Would it be possible to have this timestamp written/updated in the pom as
    > part of the release:prepare step?
    >
   
    Yep, this was one of my questions at the beginning of this thread.
    I see value in this proposal
   
    Enrico
   
   
   
   
    > Sort of setting it to some constant (ie REPODUCEABLE_BUILD_TIMESTAMP)
    > simply takes the current time but if there is a concrete value, it uses
    > that instead?
    >
    > Hope im not asking anything obvious. To me it looked as if the timestamp
    > has to be manipulated manually.
    >
    > Chris
    >
    > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
    > ________________________________
    > From: Emmanuel Bourg <[hidden email]>
    > Sent: Thursday, October 10, 2019 11:50:34 PM
    > To: [hidden email] <[hidden email]>
    > Subject: Re: last review of Reproducible Builds proposal
    >
    > Le 10/10/2019 à 19:28, Hervé BOUTEMY a écrit :
    >
    > > the only little mis-interpretation is that it is a pure build
    > information,
    > > then I don't see why this property would appear in a consumer POM
    >
    > Thank you for the clarification, that makes perfectly sense. And I now
    > see the benefit of using a property that can be inherited. In a multi
    > modules project it's only necessary to define the timestamp once in the
    > root pom. Parent poms deployed to Maven Central will never include a
    > timestamp and there is no risk of affecting other projects deriving from
    > the pom.
    >
    > Emmanuel Bourg
    >
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: [hidden email]
    > For additional commands, e-mail: [hidden email]
    >
    >
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
Le vendredi 11 octobre 2019, 10:03:10 CEST Christofer Dutz a écrit :
> Hi Enrico,
>
> so I would definitely +1 to be able to do it this way ...
> I think reproducible builds are important for releases, but I am not that
> sure the same applies for the daily business in a project. For being able
> to do releases it would be a huge improvement, if this is automatically
> handled the same way the versions are updated.
see https://issues.apache.org/jira/browse/MRELEASE-1029

> For the plc4x project I was even thinking of building some maven tooling to
> automatically verify the built archives with the ones staged in nexus for
> binary equality.
yes, that's a next step: I see we have the same idea :)

At the moment, this is what I expect from Buildinfo file proposal we wrote
(with a sbt developer, since that proposal is globally JVM-oriented, whatever
the build tool is):
https://reproducible-builds.org/docs/jvm/

If the release manager publishes the buildinfo of his "reference" release
build (= 1 buildinfo, even if there are thousands of artifacts), it could be
easily compared to a rebuilder's buildinfo done during his local rebuild.

One year ago, I was ready to write a Maven plugin for generating such
buildinfo, when I thought that starting from this plugin would not bring real
value until we had a chance to get reproducible results. Now that native
Reproducible Builds for Maven is near, writing the plugin that will generate a
buildinfo file makes sense and should not be really complex (less complex than
MRELEASE-1029 , for example)

For those interested in Reproducible Builds, next yearly meeting is in
december https://reproducible-builds.org/events/
Last year event in Paris was key to my understanding of so many aspects we'll
need to manage in future years to get the full value of Reproducible Builds in
Java

Of course, we can also work on this during Apache Conference Europe 2019
https://aceu19.apachecon.com/
I hope to meet a lot of people and discuss a lot of good steps forward

Regards,

Hervé
 
> Chris
>
> Am 11.10.19, 09:05 schrieb "Enrico Olivelli" <[hidden email]>:
>
>     Il ven 11 ott 2019, 08:31 Christofer Dutz <[hidden email]>
> ha
 scritto:

>    
>
>     > Just a small question. I have been following this thread with great
>     > interest.
>     >
>     >
>     >
>     > I think this is going to be a big thing when it makes the changes
>     > available to the main maven system.
>     >
>     >
>     >
>     > So as far as I understand the core part will be a fixed timestamp
>     > which
>     > will then be used throughout the build by multiple pluggins.
>     >
>     >
>     >
>     > So if I provide the same timestamp the result should be binary
>     > identical.
>     >
>     >
>     >
>     > Would it be possible to have this timestamp written/updated in the pom
>     > as
>     > part of the release:prepare step?
>     >
>     >
>
>    
>     Yep, this was one of my questions at the beginning of this thread.
>     I see value in this proposal
>    
>     Enrico
>    
>    
>    
>    
>
>     > Sort of setting it to some constant (ie REPODUCEABLE_BUILD_TIMESTAMP)
>     > simply takes the current time but if there is a concrete value, it
>     > uses
>     > that instead?
>     >
>     >
>     >
>     > Hope im not asking anything obvious. To me it looked as if the
>     > timestamp
>     > has to be manipulated manually.
>     >
>     >
>     >
>     > Chris
>     >
>     >
>     >
>     > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
>     > ________________________________
>     > From: Emmanuel Bourg <[hidden email]>
>     > Sent: Thursday, October 10, 2019 11:50:34 PM
>     > To: [hidden email] <[hidden email]>
>     > Subject: Re: last review of Reproducible Builds proposal
>     >
>     >
>     >
>     > Le 10/10/2019 à 19:28, Hervé BOUTEMY a écrit :
>     >
>     >
>     >
>     > > the only little mis-interpretation is that it is a pure build
>     >
>     > information,
>     >
>     > > then I don't see why this property would appear in a consumer POM
>     >
>     >
>     >
>     > Thank you for the clarification, that makes perfectly sense. And I
>     > now
>     > see the benefit of using a property that can be inherited. In a multi
>     > modules project it's only necessary to define the timestamp once in
>     > the
>     > root pom. Parent poms deployed to Maven Central will never include a
>     > timestamp and there is no risk of affecting other projects deriving
>     > from
>     > the pom.
>     >
>     >
>     >
>     > Emmanuel Bourg
>     >
>     >
>     >
>     > ---------------------------------------------------------------------
>     > To unsubscribe, e-mail: [hidden email]
>     > For additional commands, e-mail: [hidden email]
>     >
>     >
>     >
>
>    
>





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
last question: now that we seem to fully understand each other, does it mean
that you don't need any more "seconds since the epoch" format support for the
property?

I can add it if you feel it may be useful, since it won't break any logic,
even if I don't really see the use case given the approach of "native Maven
binary Reproducible Builds".

It's the last little decision to do before I do the maven-archiver release
that will lead to packaging plugins releases

Regards,

Hervé

Le jeudi 10 octobre 2019, 23:50:34 CEST Emmanuel Bourg a écrit :

> Le 10/10/2019 à 19:28, Hervé BOUTEMY a écrit :
> > the only little mis-interpretation is that it is a pure build information,
> > then I don't see why this property would appear in a consumer POM
>
> Thank you for the clarification, that makes perfectly sense. And I now
> see the benefit of using a property that can be inherited. In a multi
> modules project it's only necessary to define the timestamp once in the
> root pom. Parent poms deployed to Maven Central will never include a
> timestamp and there is no risk of affecting other projects deriving from
> the pom.
>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Michael Osipov-2
In reply to this post by Hervé BOUTEMY
Am 2019-10-05 um 19:52 schrieb Hervé BOUTEMY:

> based on the feedback I got, I updated the proposal:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
>
> The archives entries timestamp is now configured with project.build.outputTimestamp property, in ISO-8601 format
> <project>
>    <properties>
>      <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
>    </properties>
> </project>
>
> The shared components, plugins, parent poms and Maven core branches have been updated to match this new proposal
>
>
> If no one objects, next week-end, I'll start the (heavy) release train to bring (binary) Reproducible Builds plugins to general availability

One more thing: if there is a slight chance that more properties will be
added in the future, it'd be worth to group them into
<project.build.reproducible. ...>

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Andreas Sewe-2
In reply to this post by Hervé BOUTEMY
Emmanuel Bourg wrote:

> Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
>> last question: now that we seem to fully understand each other, does it mean
>> that you don't need any more "seconds since the epoch" format support for the
>> property?
>
> If Maven supports the SOURCE_DATE_EPOCH environment variable I don't
> think that's necessary, otherwise it would be nice to be able to invoke
> Maven with:
>
>    mvn package -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
>
> and this means supporting a timestamp formatted as seconds since the epoch.
+1

The above would be a nice, simple way of bridging the gap between
SOURCE_DATE_EPOCH and project.build.outputTimestamp.

If it is not too much trouble to implement the "\d+ -> seconds since
epoch" heuristic, them I would love to see it included.

Best wishes,

Andreas


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
> Hi all,
>
> as I can see you're voting on releasing the reproducible build extended
> plugin versions.
 Is there any documentation on how to use this new
> feature?
>
> I had a look at the confluence page, but that seemed like a brainstorming
> session.
ok, the Wiki page [1] started as a brainstorming session, was updated to a proposal (the "Output Archive Entries Timestamp" parapgraph).
And now I probably should order paragraph a little bit, and add a "Making your build reproducible" section for end uses to have a quick explanation.

I'll write the explanation here as a first try before working on the Wiki:

1. upgrade your plugins to reproducible version, particularly mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion 3.2.0
2. add project.build.outputTimestamp property with the timestamp value that will be used in zip/jar/tar archives:
  <properties>
    <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
  </properties>

Notice:
- there is no Maven version prerequisite, everything happens at plugins level
- Reproducible Builds require to have no version ranges in dependencies, generally gives different result on Unixes vs Windows, and generally depend on the major version of JDK used to compile (see "Out of Scope" paragraph)

You have the basis configured, the output should be reproducible now.
If something is still not reproducible, use diffoscope to find the unstable output, find the plugin that generated this output and check if there is a reproducible version available: if not, please open an issue to help plugin maintainers improving Reproducible Builds support at every plugin level.

[1]  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318


 
> I would love to add this to the PLC4X build asap.
I'd love to have feedback
Don't hesitate to ping me.
>
> So I would like to test the release-candidates and vote too.
I would love to have many tester and votes :)

>
> Chris
>
>
>
> Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <[hidden email]>:
>
>     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
>
>     > Emmanuel Bourg wrote:
>     >
>     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
>     > >
>     > >> last question: now that we seem to fully understand each other,
>     > >> does it
>     > >> mean that you don't need any more "seconds since the epoch" format
>     > >> support for the property?
>     > >
>     > >
>     > > If Maven supports the SOURCE_DATE_EPOCH environment variable I
>     > > don't
>     > > think that's necessary, otherwise it would be nice to be able to
>     > > invoke
>     > >
>     > > Maven with:
>     > >
>     > >    mvn package -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
>     > >
>     > >
>     > > and this means supporting a timestamp formatted as seconds since
>     > > the
>     > > epoch.
>     >
>     >
>     > +1
>     >
>     > The above would be a nice, simple way of bridging the gap between
>     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
>
>     told like that, ok, why not
>    
>
>     >
>     > If it is not too much trouble to implement the "\d+ -> seconds since
>     > epoch" heuristic, them I would love to see it included.
>
>     ok, I'll do and prepare the release
>    
>     Regards,
>    
>     Hervé
>    
>
>     >
>     > Best wishes,
>     >
>     > Andreas
>
>    
>    
>    
>    
>    
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: [hidden email]
>     For additional commands, e-mail: [hidden email]
>    
>    
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
Hi Herve,

thanks for that will try it asap and report any findings back.

But good to know that there is a difference between JDK major versions and OSes ... so it would probably be best to stage releases on Linux with an OpenJDK of the minimum supported version?
Just thinking how to make it possible to verify without having to buy Mac or Windows licenses ... guess on every machine you could whip up a Ubuntu VM for verification.
Just thinking about it ... perhaps it would be best to create a Docker image for doing the reproducible stuff ...

Are there any plans on creating a plugin to allow verification?

Sort of something like this:
"mvn package release:verify-reproduicble -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
(Which doesn't deploy the artifacts, but instead download them and do a binary comparison)

Also it could be great if the release-plugin could automatically set the property:
a) if it finds the "project.build.outputTimestamp" set to some placeholder value
b) if some switch tells it to prepare a reproducible build by using some sort of "switch" parameter

Guess that would sort of close the loop to get the biggest benefit out of the reproducible builds.
I would be happy to help as I think this is one of the greatest new features.
(Ok ... perhaps besides the sound-output-extension I learned about yesterday ;-) )


Chris


Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <[hidden email]>:

    Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
    > Hi all,
    >
    > as I can see you're voting on releasing the reproducible build extended
    > plugin versions.
     Is there any documentation on how to use this new
    > feature?
    >
    > I had a look at the confluence page, but that seemed like a brainstorming
    > session.
    ok, the Wiki page [1] started as a brainstorming session, was updated to a proposal (the "Output Archive Entries Timestamp" parapgraph).
    And now I probably should order paragraph a little bit, and add a "Making your build reproducible" section for end uses to have a quick explanation.
   
    I'll write the explanation here as a first try before working on the Wiki:
   
    1. upgrade your plugins to reproducible version, particularly mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to vesion 3.2.0
    2. add project.build.outputTimestamp property with the timestamp value that will be used in zip/jar/tar archives:
      <properties>
        <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
      </properties>
   
    Notice:
    - there is no Maven version prerequisite, everything happens at plugins level
    - Reproducible Builds require to have no version ranges in dependencies, generally gives different result on Unixes vs Windows, and generally depend on the major version of JDK used to compile (see "Out of Scope" paragraph)
   
    You have the basis configured, the output should be reproducible now.
    If something is still not reproducible, use diffoscope to find the unstable output, find the plugin that generated this output and check if there is a reproducible version available: if not, please open an issue to help plugin maintainers improving Reproducible Builds support at every plugin level.
   
    [1]  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
   
   
     
    > I would love to add this to the PLC4X build asap.
    I'd love to have feedback
    Don't hesitate to ping me.
    >
    > So I would like to test the release-candidates and vote too.
    I would love to have many tester and votes :)
   
    >
    > Chris
    >
    >
    >
    > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY" <[hidden email]>:
    >
    >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a écrit :
    >
    >     > Emmanuel Bourg wrote:
    >     >
    >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
    >     > >
    >     > >> last question: now that we seem to fully understand each other,
    >     > >> does it
    >     > >> mean that you don't need any more "seconds since the epoch" format
    >     > >> support for the property?
    >     > >
    >     > >
    >     > > If Maven supports the SOURCE_DATE_EPOCH environment variable I
    >     > > don't
    >     > > think that's necessary, otherwise it would be nice to be able to
    >     > > invoke
    >     > >
    >     > > Maven with:
    >     > >
    >     > >    mvn package -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
    >     > >
    >     > >
    >     > > and this means supporting a timestamp formatted as seconds since
    >     > > the
    >     > > epoch.
    >     >
    >     >
    >     > +1
    >     >
    >     > The above would be a nice, simple way of bridging the gap between
    >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
    >
    >     told like that, ok, why not
    >    
    >
    >     >
    >     > If it is not too much trouble to implement the "\d+ -> seconds since
    >     > epoch" heuristic, them I would love to see it included.
    >
    >     ok, I'll do and prepare the release
    >    
    >     Regards,
    >    
    >     Hervé
    >    
    >
    >     >
    >     > Best wishes,
    >     >
    >     > Andreas
    >
    >    
    >    
    >    
    >    
    >    
    >     ---------------------------------------------------------------------
    >     To unsubscribe, e-mail: [hidden email]
    >     For additional commands, e-mail: [hidden email]
    >    
    >    
    >
    >
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: [hidden email]
    > For additional commands, e-mail: [hidden email]
    >
   
   
   
   
   
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]
   
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
please provide an example of a visible issue so we can work on it

Regards,

Hervé

Le vendredi 1 novembre 2019, 10:27:36 CET Vladimir Sitnikov a écrit :
> OpenJDK8 uses hashmap for manifest entries, so jar files are not really
> reproducible there.
>
> Note: there's run-to-run randomization in j.u.HashMap, so the manifest
> contents is not predictable.
>
> Vladimir





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
In reply to this post by Hervé BOUTEMY
Hi Herve,

unfortunately I now have implemented some tooling to compare the stuff produced by the updated reproducible builds.
And it does show quite a number of non binary equal files.

Will investigate what's the difference.

Chris



Am 03.11.19, 11:08 schrieb "Hervé BOUTEMY" <[hidden email]>:

    great feedback, thank you: this proves the initial set of only 3 plugins is a
    good basis.
    For checking the many output artifacts from a build, future buildinfo file [1]
    should help a lot. I still have many usability concerns for Maven multi-module
    builds (starting with: should we generate only the root buildinfo or one
    buildinfo per Maven module?)
    Let's keep the ball rolling: today, it's plugins release day!!!
   
    Regards,
   
    Hervé
   
    [1] https://reproducible-builds.org/docs/jvm/
   
    Le vendredi 1 novembre 2019, 13:34:32 CET Christofer Dutz a écrit :
    > Hi,
    >
    > so I just updated the versions of the 3 plugins and gave the Apache PLC4X
    > build a little spin ...
    > https://github.com/apache/plc4x/tree/feature/reproducible-builds
    >
    > Did two executions of this:
    > mvn -U -P
    > with-boost,with-java,with-dotnet,with-cpp,with-python,with-proxies,with-san
    > dbox,with
    > -DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-di
    > r clean deploy
     (but with a different second local repo location)
    >
    > Then I used "diff" and "cmp" to compare individual files and it didn't show
    > differences with the artifacts I chose ...
     now I guess I'll have to whip
    > up some little bash script to sort of compare the artifacts from one
    > directory with the other (Unfortunately the SNAPSHOT timestamps are a
    > little annoying :-/
    > We do have some C++, C# and Python modules ... but I wouldn't expect the C++
    > and C# to be reproducible.
     
    > Chris
    >
    >
    > Am 01.11.19, 12:04 schrieb "Hervé BOUTEMY" <[hidden email]>:
    >
    >     Le vendredi 1 novembre 2019, 09:40:40 CET Christofer Dutz a écrit :
    >
    >     > Hi Herve,
    >     >
    >     > thanks for that will try it asap and report any findings back.
    >     >
    >     > But good to know that there is a difference between JDK major versions
    >     > and
    >     > OSes ... so it would probably be best to stage releases on Linux with
    >     > an
    >     > OpenJDK of the minimum supported version?
    >     > Just thinking how to make it
    >     > possible to verify without having to buy Mac or Windows licenses ...
    >     > guess
    >     > on every machine you could whip up a Ubuntu VM for verification. Just
    >     > thinking about it ... perhaps it would be best to create a Docker
    >     > image for
     doing the reproducible stuff ...
    >
    >     just to be clear: the difference is on newline only, then Windows vs
    > anything
     else. You get the same result on Linux, MacOS, FreeBSD, or any
    > other Unix.
    >     And if I want to be complete, if you get a source tarball from Windows,
    >
     extract it to a Linux/MacOS/... box and build with "mvn -
    >     Dline.separator='\r\n'", in general, you get the same result as a build
    > under
     Windows: I tested multiple times, it worked, but we'll see if it
    > works always or just "in general"
    >    
    >
    >     > Are there any plans on creating a plugin to allow verification?
    >     >
    >     > Sort of something like this:
    >     > "mvn package release:verify-reproduicble
    >     > -DstagingRepoUrl=a.b.c.de/repo/blahblahblah"
    >     > (Which doesn't deploy the
    >     > artifacts, but instead download them and do a binary comparison)
    >
    >     yes, but for now it was completely another form: this is the "Buildinfo
    > file"
     proposal https://reproducible-builds.org/docs/jvm/
    >     that I stopped to work on 1 year ago given I had no chance to get the
    > same
     output: it's now a good time to restart working on it as next steps
    >    
    >
    >     > Also it could be great if the release-plugin could automatically set
    >     > the
    >     > property:
    >     > a) if it finds the "project.build.outputTimestamp" set to some
    >     > placeholder value b) if some switch tells it to prepare a
    >     > reproducible
    >     > build by using some sort of "switch" parameter
    >     > Guess that would sort of close the loop to get the biggest benefit out
    >     > of
    >     > the reproducible builds.
    >
    >     +1
    >     issue has been created
    > https://issues.apache.org/jira/browse/MRELEASE-1029
     But I didn't work on
    > it yet, help welcome
    >    
    >
    >     > I would be happy to help as I think this is one
    >     > of the greatest new features. (Ok ... perhaps besides the
    >     > sound-output-extension I learned about yesterday ;-) )
    >
    >     +1
    >     the current step is important, but it's only the beginning of the story
    > from
     an effective usage perspective
    >    
    >     Regards,
    >    
    >     Hervé
    >    
    >
    >     >
    >     > Chris
    >     >
    >     >
    >     > Am 01.11.19, 09:24 schrieb "Hervé BOUTEMY" <[hidden email]>:
    >     >
    >     >
    >     >     Le jeudi 31 octobre 2019, 17:26:52 CET Christofer Dutz a écrit :
    >     >
    >     >
    >     >
    >     >     > Hi all,
    >     >     >
    >     >     > as I can see you're voting on releasing the reproducible build
    >     >     > extended
    >     >     > plugin versions.
    >     >
    >     >
    >     >
    >     >      Is there any documentation on how to use this new
    >     >
    >     >
    >     >
    >     >     > feature?
    >     >     >
    >     >     > I had a look at the confluence page, but that seemed like a
    >     >     > brainstorming
    >     >     > session.
    >     >
    >     >
    >     >
    >     >     ok, the Wiki page [1] started as a brainstorming session, was
    >     >     updated to
    >     >
    >     > a proposal (the "Output Archive Entries Timestamp" parapgraph).
    >
    >      And now I
    >
    >     > probably should order paragraph a little bit, and add a "Making your
    >     > build
    >     > reproducible" section for end uses to have a quick explanation.
    >     >
    >     >     I'll write the explanation here as a first try before working on
    >     >     the
    >     >
    >     > Wiki:
    >
    >      
    >
    >     >     1. upgrade your plugins to reproducible version, particularly
    >     >
    >     > mpaven-source-plugin, maven-jar-plugin and maven-assembly-plugin to
    >     > vesion
    >     > 3.2.0
    >
    >      2. add project.build.outputTimestamp property with the timestamp
    >
    >     > value that will be used in zip/jar/tar archives: <properties>
    >     >
    >     >        
    >     >
    >     > <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.out
    >     > putTi
     mestamp>
    >
    >      </properties>
    >
    >     >    
    >     >     Notice:
    >     >     - there is no Maven version prerequisite, everything happens at
    >     >     plugins
    >     >
    >     > level
    >
    >      - Reproducible Builds require to have no version ranges in
    >
    >     > dependencies, generally gives different result on Unixes vs Windows,
    >     > and
    >     > generally depend on the major version of JDK used to compile (see "Out
    >     > of
    >     > Scope" paragraph)
    >     >
    >     >     You have the basis configured, the output should be reproducible
    >     >     now.
    >     >     If something is still not reproducible, use diffoscope to find
    >     >     the
    >     >
    >     > unstable output, find the plugin that generated this output and check
    >     > if
    >     > there is a reproducible version available: if not, please open an
    >     > issue to
    >     > help plugin maintainers improving Reproducible Builds support at
    >     > every
    >     > plugin level.
    >
    >      
    >
    >     >     [1]
    >     >
    >     > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682
    >     > 318
    >
    >      
    >
    >     >    
    >     >    
    >     >      
    >     >
    >     >
    >     >
    >     >     > I would love to add this to the PLC4X build asap.
    >     >
    >     >
    >     >
    >     >     I'd love to have feedback
    >     >     Don't hesitate to ping me.
    >     >
    >     >
    >     >
    >     >     >
    >     >     > So I would like to test the release-candidates and vote too.
    >     >
    >     >
    >     >
    >     >     I would love to have many tester and votes :)
    >     >    
    >     >
    >     >
    >     >
    >     >     >
    >     >     > Chris
    >     >     >
    >     >     >
    >     >     >
    >     >     > Am 16.10.19, 14:42 schrieb "Hervé BOUTEMY"
    >     >     > <[hidden email]>:
    >     >     >
    >     >     >
    >     >     >
    >     >     >     Le mercredi 16 octobre 2019, 13:40:48 CEST Andreas Sewe a
    >     >     >     écrit :
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >     > Emmanuel Bourg wrote:
    >     >     >     >
    >     >     >     >
    >     >     >     >
    >     >     >     > > Le 16/10/2019 à 08:35, Hervé BOUTEMY a écrit :
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >> last question: now that we seem to fully understand
    >     >     >     > >> each
    >     >     >     > >> other,
    >     >     >     > >> does it
    >     >     >     > >> mean that you don't need any more "seconds since the
    >     >     >     > >> epoch"
    >     >     >     > >> format
    >     >     >     > >> support for the property?
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >
    >     >     >     > > If Maven supports the SOURCE_DATE_EPOCH environment
    >     >     >     > > variable
    >     >     >     > > I
    >     >     >     > > don't
    >     >     >     > > think that's necessary, otherwise it would be nice to be
    >     >     >     > > able
    >     >     >     > > to
    >     >     >     > > invoke
    >     >     >     > >
    >     >     >     > > Maven with:
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >    mvn package
    >     >     >     > >    -Dproject.build.outputTimestamp=$SOURCE_DATE_EPOCH
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >
    >     >     >     > >
    >     >     >     > > and this means supporting a timestamp formatted as
    >     >     >     > > seconds
    >     >     >     > > since
    >     >     >     > > the
    >     >     >     > > epoch.
    >     >     >     >
    >     >     >     >
    >     >     >     >
    >     >     >     >
    >     >     >     > +1
    >     >     >     >
    >     >     >     > The above would be a nice, simple way of bridging the gap
    >     >     >     > between
    >     >     >     > SOURCE_DATE_EPOCH and project.build.outputTimestamp.
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >     told like that, ok, why not
    >     >     >    
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >     >
    >     >     >     > If it is not too much trouble to implement the "\d+ ->
    >     >     >     > seconds
    >     >     >     > since
    >     >     >     > epoch" heuristic, them I would love to see it included.
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >     ok, I'll do and prepare the release
    >     >     >    
    >     >     >     Regards,
    >     >     >    
    >     >     >     Hervé
    >     >     >    
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >     >
    >     >     >     > Best wishes,
    >     >     >     >
    >     >     >     > Andreas
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >    
    >     >     >    
    >     >     >    
    >     >     >    
    >     >     >    
    >     >     >     ------------------------------------------------------------
    >     >     >     ------
    >     >     >     ---
    >     >     >     To unsubscribe, e-mail: [hidden email]
    >     >     >     For additional commands, e-mail: [hidden email]
    >     >     >    
    >     >     >    
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     > ----------------------------------------------------------------
    >     >     > -----
    >     >     > To unsubscribe, e-mail: [hidden email]
    >     >     > For additional commands, e-mail: [hidden email]
    >     >     >
    >     >
    >     >
    >     >
    >     >    
    >     >    
    >     >    
    >     >    
    >     >    
    >     >     ------------------------------------------------------------------
    >     >     ---
    >     >     To unsubscribe, e-mail: [hidden email]
    >     >     For additional commands, e-mail: [hidden email]
    >     >    
    >     >    
    >     >
    >     >
    >
    >    
    >    
    >    
    >    
    >    
    >     ---------------------------------------------------------------------
    >     To unsubscribe, e-mail: [hidden email]
    >     For additional commands, e-mail: [hidden email]
    >    
    >    
    >
   
   
   
   
   
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]
   
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Andreas Sewe-2
Christofer Dutz wrote:
> Well I have a new candidate:
>
> <artifactId>maven-bundle-plugin</artifactId>
>  
> Creates: Bnd-LastModified in the MANIFEST.MF
>
> Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)

<configuration>
  <instructions>
    <_removeheaders>Bnd-LastModified</_removeheaders>
  </instructions>
<configuration>

Alternatively

<configuration>
  <instructions>
    <Bnd-LastModified>${maven.build.outputTimestap}</Bnd-LastModified>
  </instructions>
<configuration

Hope that helps,

Andreas

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
Hi Andreas,

but the format of the timestamp is completely different ... doesn't that matter?
I was currently going for the option number 1 with removing the header.

In order to be 100% happy with this, I would prefer a setup where the normal mechanisms are used if no maven.build.outputTimestap is defined, but if it is (because a future version of the maven release plugin set it there) it uses that.

Will try out your suggestions as soon as I'm able to build again (I unfortunately installed that Mac OS update ... now things aren't working as they should be)

Chris



Am 04.11.19, 09:42 schrieb "Andreas Sewe" <[hidden email]>:

    Christofer Dutz wrote:
    > Well I have a new candidate:
    >
    > <artifactId>maven-bundle-plugin</artifactId>
    >  
    > Creates: Bnd-LastModified in the MANIFEST.MF
    >
    > Gotta find out a way to either suppress that entry (Would be great if it could consume the same property)
   
    <configuration>
      <instructions>
        <_removeheaders>Bnd-LastModified</_removeheaders>
      </instructions>
    <configuration>
   
    Alternatively
   
    <configuration>
      <instructions>
        <Bnd-LastModified>${maven.build.outputTimestap}</Bnd-LastModified>
      </instructions>
    <configuration
   
    Hope that helps,
   
    Andreas
   
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]
   
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]