Re: last review of Reproducible Builds proposal

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Michael Osipov-2
Am 2019-10-06 um 12:25 schrieb Tibor Digana:
>   ISO format was often discussed and this was found as problematic format
> because you cannot always compute it to UTC due to GMT offset. The offset
> is not enough. What is required for EXACT computing to UTC is Time zome
> name but this format does not support it. It is exactly the same problem in
> XML.

I don't understand that and do not agree. Of course, you can do
normalization. Can you please elaborate?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Hervé BOUTEMY
Le dimanche 6 octobre 2019, 16:41:27 CEST Emmanuel Bourg a écrit :

> Le 06/10/2019 à 09:43, Hervé BOUTEMY a écrit :
> > Notice that you can also express a timezone (as digits), as seen in the
> > unit tests.
> I know but that's not desirable, otherwise the formatted timestamp would
> depend on the timezone of the developer and that would harm the
> reproducibility of the pom (I assume the timestamp is added
> automatically to the pom during the build, is that right?).
>
> >> Would it be possible to prevent this property from being inherited?
> >
> > AFAIK no.
> > And I find that default inheritance from parent to child is a nice
> > feature.
>
> Why do you think this is a nice feature? I can only see the downside of
> child projects having a timestamp clamped in the past and leaving the
> developers clueless about the reason. This would be especially bad if a
> widely used parent pom like org.sonatype.oss:oss-parent or
> org.apache:apache had it.
choosing to add Reproducible Builds configuration (ie output timestamp) in a
parent is a strong choice, because it will inherit to every child.
Looks like the current fear of many things means we won't put the configuration
in ASF parent for now until parent learn by real experience how it works well:
next ASF parent POM release will use plugins versions that are able to support
Reproducible Builds, but won't configure a timestamp to activate the feature.
It will be an opt-in option for projects that like this idea.

Perhaps in a later version, experience will be sufficient to configure the
timestamp by default, and always keep the opt-out option when some project has
issues

>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Michael Osipov-2
In reply to this post by Michael Osipov-2
Am 2019-10-06 um 20:35 schrieb Hervé BOUTEMY:

> Le dimanche 6 octobre 2019 20:29:46 CEST, vous avez écrit :
>> Am 2019-10-06 um 20:21 schrieb Hervé BOUTEMY:> Le dimanche 6 octobre
>>
>> 2019 12:24:57 CEST, vous avez écrit :
>>   >> Am 2019-10-06 um 09:35 schrieb Hervé BOUTEMY:
>>   >>> Le samedi 5 octobre 2019, 22:46:20 CEST Michael Osipov a écrit :
>>   >>>> Am 2019-10-05 um 22:10 schrieb Hervé BOUTEMY:
>>   >>>>> Le samedi 5 octobre 2019 20:41:40 CEST, vous avez écrit :
>>   >>>>>> Am 2019-10-05 um 19:52 schrieb Hervé BOUTEMY:
>>   >>>>>>> based on the feedback I got, I updated the proposal:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682
>>
>>   >>>>>>> 31
>>   >>>>>>> 8
>>   >>>>>>>
>>   >>>>>>> The archives entries timestamp is now configured with
>>   >>>>>>> project.build.outputTimestamp property, in ISO-8601 format
>>
>> <project>
>>
>>   >>>>>>>       <properties>
>>
>> <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.bu
>>
>>   >>>>>>>         ild
>>   >>>>>>>         .ou
>>   >>>>>>>         tputTimestamp>>
>>   >>>>>>>
>>   >>>>>>>       </properties>
>>   >>>>>>>
>>   >>>>>>> </project>
>>   >>>>>>>
>>   >>>>>>> The shared components, plugins, parent poms and Maven core branches
>>   >>>>>>> have
>>   >>>>>>> been updated to match this new proposal
>>   >>>>>>>
>>   >>>>>>>
>>   >>>>>>> If no one objects, next week-end, I'll start the (heavy)
>>
>> release train
>>
>>   >>>>>>> to
>>   >>>>>>> bring (binary) Reproducible Builds plugins to general availability
>>   >>>>>>
>>   >>>>>> Really, really nice work..
>>   >>>>>
>>   >>>>> thank you, it required a lot of energy for a long period of time...
>>   >>>>>
>>   >>>>>> I do like <project.build.outputTimestamp>. Why did you make it a
>>   >>>>>> String?
>>   >>>>>> Why not go directly with Instant? It gives your ISO 8601 for free
>>   >>>>>
>>   >>>>> I tried to explain it in
>>   >>>>> https://issues.apache.org/jira/browse/MSHARED-837
>>   >>>>> Plexus Date injection support is really limited: could not match the
>>   >>>>> requirements
>>   >>>>
>>   >>>> yyyy-MM-dd'T'HH:mm:ssXXX and SimpleDateFormat will do the trick.
>>
>> It also
>>
>>   >>>> will require to change the converter of course.
>>   >>>>
>>   >>>> Alternatively, you could try
>>
>> https://commons.apache.org/proper/commons-lang/javadocs/api-release/src-h
>>
>>   >>>> tml /org/apache/commons/lang3/time/DateFormatUtils.html#line.72
>>
>> which I
>>
>>   >>>> have added long time ago.
>>   >>>
>>   >>> the question is not about coding the date format: it's already done in
>>   >>> maven-archiver
>>
>> https://github.com/apache/maven-archiver/commit/5f07f227aa89e0bb4163c125a
>>
>>   >>> 46fbd4c78445301
>>   >>
>>   >> The commit contains some flaws. I will leave comments there.
>>   >>
>>   >>> the question, as you perfectly asked initially, is: do you have an
>>
>> idea on
>>
>>   >>> how to have the direct Date injection, instead of having a String
>>
>> in the
>>
>>   >>> plugin parameter?
>>   >>
>>   >> I assumed the DateConverter will do as soon as you say:
>>   >> @Parameter
>>   >> Date myDate;
>>   >>
>>   >> Won't it?
>>   >
>>   > yes, it will
>>   > and if you look at the SimpleDateFormat value, you'll see that it
>>
>> does not
>>
>>   > permit any timezone, then uses the local developper timezone, then is not
>>   > reproducible
>>
>> No, it won't if you supply and enforce XXX => yyyy-MM-dd'T'HH:mm:ssXXX.
>> The input has to have the tz offset, thus you'll have it properly
>> converted. It would perform the very same conversion and your code
>> currently does in Maven Archiver. You should give it a try.
> DateConverter is part of plexus-containers and Sisu: I'm not modifying them

Why don't you say some upfront? I assumed you want to change it.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Emmanuel Bourg
In reply to this post by Michael Osipov-2
Le 06/10/2019 à 20:13, Hervé BOUTEMY a écrit :

> no, it does not add any dependency on developer configuration:
> 2019-10-05T18:37:42Z == 2019-10-05T20:37:42+02:00 == 2019-10-05T16:37:42-02:00

yes but:

 "2019-10-05T18:37:42Z" != "2019-10-05T20:37:42+02:00" !=
"2019-10-05T16:37:42-02:00"

The point is, two developers may generate a different pom if the local
timezone is used. A fixed timezone is necessary to ensure the pom itself
is reproducible.


> when will this value be written in the pom.xml is independant: currently, in
> my PoC, I wrote the values by hand. In the future, it will probably be updated
> by maven-release-plugin, and we'll have to choose if the timestamp is written
> in Z or if it is written in local timezone with its offset: both ways of
> expressing the timestamp are valid and will give reproducible result

The jar generated is reproducible, but not the pom. I suspect that if
the jar includes the pom it'll break its reproducibility too (this is
the default for maven-jar-plugin, but I don't know if it embeds the
original pom without the timestamp, or the generated pom with the
timestamp).


> once again, war files taken apart for web servers, who looks at timestamp in
> zip files?

archive timestamps are just the tip of the iceberg. There are more
visible timestamps elsewhere, for example in the javadoc headers, in
.properties files, in OSGi attributes, sometimes in the source files...

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: last review of Reproducible Builds proposal

Romain Manni-Bucau
Small reminder: if you want to be reproducible you must fix the timestamp
so whatever zone, whatever format works. It is common to use new Date(1000)
in utc but not important at the end.

Side note: same applies for most of the env though (locale for ex.).

Le dim. 6 oct. 2019 à 22:57, Tibor Digana <[hidden email]> a écrit :

> Hi Emmanuel,
>
> >> The point is, two developers may generate a different pom if the local
> timezone is used.
>
> very well explained! It could not be better: utc time is the same however
> the text is different and that's important for the stream identity.
> Karl had a proposal with additional property "format" as a compromise.
> Is the geography a localization important information for the behavior of
> reproducible build? IMO, it is not!
> So why not to have Karl's format NULL by default, means the UTC.
>
> btw, there is whole bunch of problems with time with/out TZ in webapps and
> rdbms databases and cloud system and distributed HTTP clients but that's
> another story.
>
>
> >> I suspect that if the jar includes the pom it'll break its
> reproducibility too
>
> Actually, there is "pom.properties" in META-INF folder of the JAR archive
> artifact and this file containes the comment having the system time
> generated by JDK.
> There's a PR for this issue. The POM and the properties file are both
> embedded in META-INF.
>
>
> Cheers
> Tibor17
>
> On Sun, Oct 6, 2019 at 10:19 PM Emmanuel Bourg <[hidden email]> wrote:
>
> > Le 06/10/2019 à 20:13, Hervé BOUTEMY a écrit :
> >
> > > no, it does not add any dependency on developer configuration:
> > > 2019-10-05T18:37:42Z == 2019-10-05T20:37:42+02:00 ==
> > 2019-10-05T16:37:42-02:00
> >
> > yes but:
> >
> >  "2019-10-05T18:37:42Z" != "2019-10-05T20:37:42+02:00" !=
> > "2019-10-05T16:37:42-02:00"
> >
> > The point is, two developers may generate a different pom if the local
> > timezone is used. A fixed timezone is necessary to ensure the pom itself
> > is reproducible.
> >
> >
> > > when will this value be written in the pom.xml is independant:
> > currently, in
> > > my PoC, I wrote the values by hand. In the future, it will probably be
> > updated
> > > by maven-release-plugin, and we'll have to choose if the timestamp is
> > written
> > > in Z or if it is written in local timezone with its offset: both ways
> of
> > > expressing the timestamp are valid and will give reproducible result
> >
> > The jar generated is reproducible, but not the pom. I suspect that if
> > the jar includes the pom it'll break its reproducibility too (this is
> > the default for maven-jar-plugin, but I don't know if it embeds the
> > original pom without the timestamp, or the generated pom with the
> > timestamp).
> >
> >
> > > once again, war files taken apart for web servers, who looks at
> > timestamp in
> > > zip files?
> >
> > archive timestamps are just the tip of the iceberg. There are more
> > visible timestamps elsewhere, for example in the javadoc headers, in
> > .properties files, in OSGi attributes, sometimes in the source files...
> >
> > Emmanuel Bourg
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>