last review of Reproducible Builds proposal

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

last review of Reproducible Builds proposal

Hervé BOUTEMY
based on the feedback I got, I updated the proposal:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318

The archives entries timestamp is now configured with project.build.outputTimestamp property, in ISO-8601 format
<project>
  <properties>
    <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
  </properties>
</project>

The shared components, plugins, parent poms and Maven core branches have been updated to match this new proposal


If no one objects, next week-end, I'll start the (heavy) release train to bring (binary) Reproducible Builds plugins to general availability

Regards,

Hervé



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Michael Osipov-2
Am 2019-10-05 um 19:52 schrieb Hervé BOUTEMY:

> based on the feedback I got, I updated the proposal:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
>
> The archives entries timestamp is now configured with project.build.outputTimestamp property, in ISO-8601 format
> <project>
>    <properties>
>      <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
>    </properties>
> </project>
>
> The shared components, plugins, parent poms and Maven core branches have been updated to match this new proposal
>
>
> If no one objects, next week-end, I'll start the (heavy) release train to bring (binary) Reproducible Builds plugins to general availability

Really, really nice work..

I do like <project.build.outputTimestamp>. Why did you make it a String?
Why not go directly with Instant? It gives your ISO 8601 for free

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Vladimir Sitnikov
In reply to this post by Hervé BOUTEMY
>but
>who really looks at the timestamp of entries in release zips/jars/tar.gz
>honestly?

Tomcat when it decides on what to send in the "Last-Modified" header.
For instance, current Gradle does not allow to configure the timestamp, and
for reproducible builds it always sets the timestamp to 0 or so.
It breaks Tomcat's assumptions:
https://github.com/gradle/gradle/issues/10917

Vladimir
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
Le dimanche 6 octobre 2019, 22:18:59 CEST Emmanuel Bourg a écrit :

> Le 06/10/2019 à 20:13, Hervé BOUTEMY a écrit :
> > no, it does not add any dependency on developer configuration:
> > 2019-10-05T18:37:42Z == 2019-10-05T20:37:42+02:00 ==
> > 2019-10-05T16:37:42-02:00
> yes but:
>
>  "2019-10-05T18:37:42Z" != "2019-10-05T20:37:42+02:00" !=
> "2019-10-05T16:37:42-02:00"
>
> The point is, two developers may generate a different pom if the local
> timezone is used. A fixed timezone is necessary to ensure the pom itself
> is reproducible.
There is a misunderstanding here: the pom.xml is saved in the source control,
with the timestamp in it.
There is no question of "reproducible pom.xml"

>
> > when will this value be written in the pom.xml is independant: currently,
> > in my PoC, I wrote the values by hand. In the future, it will probably be
> > updated by maven-release-plugin, and we'll have to choose if the
> > timestamp is written in Z or if it is written in local timezone with its
> > offset: both ways of expressing the timestamp are valid and will give
> > reproducible result
> The jar generated is reproducible, but not the pom. I suspect that if
> the jar includes the pom it'll break its reproducibility too (this is
> the default for maven-jar-plugin, but I don't know if it embeds the
> original pom without the timestamp, or the generated pom with the
> timestamp).
>
> > once again, war files taken apart for web servers, who looks at timestamp
> > in zip files?
>
> archive timestamps are just the tip of the iceberg. There are more
> visible timestamps elsewhere, for example in the javadoc headers, in
> .properties files, in OSGi attributes, sometimes in the source files...
Sure, many plugins have already been modified to remove such noise in output,
and probably others will require to be updated. Because in Reproducible
Builds, timestamps to know when something has been generated is less useful:
Reproduciblity is the ultimate proof of knowledge of what has been built,
since the result will be the same now, tomorrow, in one year...

Regards,

Hervé


>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Emmanuel Bourg
In reply to this post by Hervé BOUTEMY
Le 07/10/2019 à 14:40, Andreas Sewe a écrit :

> - Can we get something analogous to maven.build.timestamp.format? One
> could then write the following to be compatible with env.SOURCE_DATE_EPOCH:
>
> <project>
>   <properties>
>
> <project.build.outputTimestamp>${env.SOURCE_DATE_EPOCH}<project.build.outputTimestamp>
>
> <project.build.outputTimestamp.format>...</project.build.outputTimestamp.format>
>   </properties>
> </project>

+1, some kind of support for SOURCE_DATE_EPOCH would be nice. Either
this (but maybe with only one property supporting both formats) or by
overriding automatically the property when SOURCE_DATE_EPOCH is set.

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
Le mardi 8 octobre 2019, 23:42:55 CEST Mark Derricutt a écrit :

> On 6 Oct 2019, at 9:14, Hervé BOUTEMY wrote:
> > if anybody cares about the exact value: but
> > who really looks at the timestamp of entries in release zips/jars/tar.gz
> > honestly?
>
> I've actually done so in the past trying to find differences between two
> versions of a jar for repair reasons.
>
> VERY infrequent tho - if you're wanting to generate a delta-patch between
> two jars/zips then I guess it might also be handy (altho you might be
> better off going direct for checksumdifferences  there ).
FYI, when you try to know the difference between 2 archives, the ideal tool is
diffoscope [1]: it has been done exactly for that

[1] https://diffoscope.org/

>
>
> ---
> "The ease with which a change can be implemented has no relevance at all to
> whether it is the right change for the (Java) Platform for all time."
> &mdash; Mark Reinhold.
>
> Mark Derricutt
> http://www.theoryinpractice.net
> http://www.chaliceofblood.net
> http://plus.google.com/+MarkDerricutt
> http://twitter.com/talios
> http://facebook.com/mderricutt





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
In reply to this post by Hervé BOUTEMY
Just a small question. I have been following this thread with great interest.

I think this is going to be a big thing when it makes the changes available to the main maven system.

So as far as I understand the core part will be a fixed timestamp which will then be used throughout the build by multiple pluggins.

So if I provide the same timestamp the result should be binary identical.

Would it be possible to have this timestamp written/updated in the pom as part of the release:prepare step?

Sort of setting it to some constant (ie REPODUCEABLE_BUILD_TIMESTAMP) simply takes the current time but if there is a concrete value, it uses that instead?

Hope im not asking anything obvious. To me it looked as if the timestamp has to be manipulated manually.

Chris

Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
________________________________
From: Emmanuel Bourg <[hidden email]>
Sent: Thursday, October 10, 2019 11:50:34 PM
To: [hidden email] <[hidden email]>
Subject: Re: last review of Reproducible Builds proposal

Le 10/10/2019 à 19:28, Hervé BOUTEMY a écrit :

> the only little mis-interpretation is that it is a pure build information,
> then I don't see why this property would appear in a consumer POM

Thank you for the clarification, that makes perfectly sense. And I now
see the benefit of using a property that can be inherited. In a multi
modules project it's only necessary to define the timestamp once in the
root pom. Parent poms deployed to Maven Central will never include a
timestamp and there is no risk of affecting other projects deriving from
the pom.

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Christofer Dutz
Hi Enrico,

so I would definitely +1 to be able to do it this way ...
I think reproducible builds are important for releases, but I am not that sure the same applies for the daily business in a project.
For being able to do releases it would be a huge improvement, if this is automatically handled the same way the versions are updated.

For the plc4x project I was even thinking of building some maven tooling to automatically verify the built archives with the ones staged in nexus for binary equality.

Chris

Am 11.10.19, 09:05 schrieb "Enrico Olivelli" <[hidden email]>:

    Il ven 11 ott 2019, 08:31 Christofer Dutz <[hidden email]> ha
    scritto:
   
    > Just a small question. I have been following this thread with great
    > interest.
    >
    > I think this is going to be a big thing when it makes the changes
    > available to the main maven system.
    >
    > So as far as I understand the core part will be a fixed timestamp which
    > will then be used throughout the build by multiple pluggins.
    >
    > So if I provide the same timestamp the result should be binary identical.
    >
    > Would it be possible to have this timestamp written/updated in the pom as
    > part of the release:prepare step?
    >
   
    Yep, this was one of my questions at the beginning of this thread.
    I see value in this proposal
   
    Enrico
   
   
   
   
    > Sort of setting it to some constant (ie REPODUCEABLE_BUILD_TIMESTAMP)
    > simply takes the current time but if there is a concrete value, it uses
    > that instead?
    >
    > Hope im not asking anything obvious. To me it looked as if the timestamp
    > has to be manipulated manually.
    >
    > Chris
    >
    > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
    > ________________________________
    > From: Emmanuel Bourg <[hidden email]>
    > Sent: Thursday, October 10, 2019 11:50:34 PM
    > To: [hidden email] <[hidden email]>
    > Subject: Re: last review of Reproducible Builds proposal
    >
    > Le 10/10/2019 à 19:28, Hervé BOUTEMY a écrit :
    >
    > > the only little mis-interpretation is that it is a pure build
    > information,
    > > then I don't see why this property would appear in a consumer POM
    >
    > Thank you for the clarification, that makes perfectly sense. And I now
    > see the benefit of using a property that can be inherited. In a multi
    > modules project it's only necessary to define the timestamp once in the
    > root pom. Parent poms deployed to Maven Central will never include a
    > timestamp and there is no risk of affecting other projects deriving from
    > the pom.
    >
    > Emmanuel Bourg
    >
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: [hidden email]
    > For additional commands, e-mail: [hidden email]
    >
    >
   


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
Le vendredi 11 octobre 2019, 10:03:10 CEST Christofer Dutz a écrit :
> Hi Enrico,
>
> so I would definitely +1 to be able to do it this way ...
> I think reproducible builds are important for releases, but I am not that
> sure the same applies for the daily business in a project. For being able
> to do releases it would be a huge improvement, if this is automatically
> handled the same way the versions are updated.
see https://issues.apache.org/jira/browse/MRELEASE-1029

> For the plc4x project I was even thinking of building some maven tooling to
> automatically verify the built archives with the ones staged in nexus for
> binary equality.
yes, that's a next step: I see we have the same idea :)

At the moment, this is what I expect from Buildinfo file proposal we wrote
(with a sbt developer, since that proposal is globally JVM-oriented, whatever
the build tool is):
https://reproducible-builds.org/docs/jvm/

If the release manager publishes the buildinfo of his "reference" release
build (= 1 buildinfo, even if there are thousands of artifacts), it could be
easily compared to a rebuilder's buildinfo done during his local rebuild.

One year ago, I was ready to write a Maven plugin for generating such
buildinfo, when I thought that starting from this plugin would not bring real
value until we had a chance to get reproducible results. Now that native
Reproducible Builds for Maven is near, writing the plugin that will generate a
buildinfo file makes sense and should not be really complex (less complex than
MRELEASE-1029 , for example)

For those interested in Reproducible Builds, next yearly meeting is in
december https://reproducible-builds.org/events/
Last year event in Paris was key to my understanding of so many aspects we'll
need to manage in future years to get the full value of Reproducible Builds in
Java

Of course, we can also work on this during Apache Conference Europe 2019
https://aceu19.apachecon.com/
I hope to meet a lot of people and discuss a lot of good steps forward

Regards,

Hervé
 
> Chris
>
> Am 11.10.19, 09:05 schrieb "Enrico Olivelli" <[hidden email]>:
>
>     Il ven 11 ott 2019, 08:31 Christofer Dutz <[hidden email]>
> ha
 scritto:

>    
>
>     > Just a small question. I have been following this thread with great
>     > interest.
>     >
>     >
>     >
>     > I think this is going to be a big thing when it makes the changes
>     > available to the main maven system.
>     >
>     >
>     >
>     > So as far as I understand the core part will be a fixed timestamp
>     > which
>     > will then be used throughout the build by multiple pluggins.
>     >
>     >
>     >
>     > So if I provide the same timestamp the result should be binary
>     > identical.
>     >
>     >
>     >
>     > Would it be possible to have this timestamp written/updated in the pom
>     > as
>     > part of the release:prepare step?
>     >
>     >
>
>    
>     Yep, this was one of my questions at the beginning of this thread.
>     I see value in this proposal
>    
>     Enrico
>    
>    
>    
>    
>
>     > Sort of setting it to some constant (ie REPODUCEABLE_BUILD_TIMESTAMP)
>     > simply takes the current time but if there is a concrete value, it
>     > uses
>     > that instead?
>     >
>     >
>     >
>     > Hope im not asking anything obvious. To me it looked as if the
>     > timestamp
>     > has to be manipulated manually.
>     >
>     >
>     >
>     > Chris
>     >
>     >
>     >
>     > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
>     > ________________________________
>     > From: Emmanuel Bourg <[hidden email]>
>     > Sent: Thursday, October 10, 2019 11:50:34 PM
>     > To: [hidden email] <[hidden email]>
>     > Subject: Re: last review of Reproducible Builds proposal
>     >
>     >
>     >
>     > Le 10/10/2019 à 19:28, Hervé BOUTEMY a écrit :
>     >
>     >
>     >
>     > > the only little mis-interpretation is that it is a pure build
>     >
>     > information,
>     >
>     > > then I don't see why this property would appear in a consumer POM
>     >
>     >
>     >
>     > Thank you for the clarification, that makes perfectly sense. And I
>     > now
>     > see the benefit of using a property that can be inherited. In a multi
>     > modules project it's only necessary to define the timestamp once in
>     > the
>     > root pom. Parent poms deployed to Maven Central will never include a
>     > timestamp and there is no risk of affecting other projects deriving
>     > from
>     > the pom.
>     >
>     >
>     >
>     > Emmanuel Bourg
>     >
>     >
>     >
>     > ---------------------------------------------------------------------
>     > To unsubscribe, e-mail: [hidden email]
>     > For additional commands, e-mail: [hidden email]
>     >
>     >
>     >
>
>    
>





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]