[DISCUSS] checking reproducible builds

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] checking reproducible builds

Hervé BOUTEMY
Hi,

Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository.

For a live example, see the last paragraph of Maven Site Plugin vote that just started [1].

Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]:
1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository
2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference.

Now I want to discuss: is it clear? can you test and report, please?

If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository.

Thanks for your feedback

Regards,

Hervé

[1] https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E

[2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Karl Heinz Marbaise-3
Hi Hervé,

I've tried to check my release via the suggested recipe...


Downloaded the maven-studies repo and build the following commit:
90b426758363123af6fcc9aa7190b837c0551359 (mvn clean install)

Downloaded the source package

curl -O
https://repository.apache.org/content/repositories/maven-1555/org/apache/maven/plugins/maven-dependency-plugin/3.1.2/maven-dependency-plugin-3.1.2-source-release.zip

unzip maven-dependency-plugin-3.1.2-source-release.zip

cd maven-dependency-plugin-3.1.2 and tried to run the following:

mvn -Papache-release verify buildinfo:save -Dgpg.skip
-Dreference.repo=https://repository.apache.org/content/repositories/maven-1555/

and got the following:


[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save
(default-cli) on project maven-dependency-plugin: Error resolving
reference artifact
org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2: Could
not transfer artifact
org.apache.maven.plugins:maven-dependency-plugin:buildinfo:3.1.2 from/to
reference
(https://repository.apache.org/content/repositories/maven-1555/): Cannot
access https://repository.apache.org/content/repositories/maven-1555/
with type  using the available connector factories:
BasicRepositoryConnectorFactory: Cannot access
https://repository.apache.org/content/repositories/maven-1555/ with type
  using the available layout factories: Maven2RepositoryLayoutFactory:
Unsupported repository layout -> [Help 1]
[ERROR]



Kind regards
Karl Heinz Marbaise

On 07.03.20 11:36, Hervé BOUTEMY wrote:

> Hi,
>
> Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository.
>
> For a live example, see the last paragraph of Maven Site Plugin vote that just started [1].
>
> Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]:
> 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository
> 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference.
>
> Now I want to discuss: is it clear? can you test and report, please?
>
> If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository.
>
> Thanks for your feedback
>
> Regards,
>
> Hervé
>
> [1] https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E
>
> [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
In reply to this post by Hervé BOUTEMY
Am 2020-03-07 um 13:45 schrieb Michael Osipov:

> Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:
>> Hi,
>>
>> Yesterday, I made a key step forward for Reproducible Builds with
>> Maven: I wrote code to easily check that your local build produces the
>> same binaries as the reference binaries published either to staging or
>> to Central repository.
>>
>> For a live example, see the last paragraph of Maven Site Plugin vote
>> that just started [1].
>>
>> Process to check build output is based on a single plugin goal,
>> currently named buildinfo:save [2]:
>> 1. it creates a buildinfo file during build recording output
>> fingerprints, that will eventually in the future be published to
>> Central repository
>> 2. it downloads reference artifacts and/or reference buildinfo and
>> checks that the output of the local build is the same as the reference.
>>
>> Now I want to discuss: is it clear? can you test and report, please?
>>
>> If the feedback is positive, the next question will be: in which
>> plugin should we put this goal to make a release and add it to our
>> parent pom during release, so we publish reference buildinfo along our
>> reference binaries to Central repository.
>
> Fails for me with:
>> osipovmi@deblndw011x:~/var/Projekte/maven-site-plugin
>> ((maven-site-plugin-3.9.0)
>> $ ~/apache-maven-3.7.0-SNAPSHOT/bin/mvn -v
>> Apache Maven 3.7.0-SNAPSHOT (f2e9afd788de919646717532d26eca38826e9924)
>> Maven home: /net/home/osipovmi/apache-maven-3.7.0-SNAPSHOT
>> Java version: 1.8.0_242, vendor: Oracle Corporation, runtime:
>> /usr/local/openjdk8/jre
>> Default locale: de_DE, platform encoding: UTF-8
>> OS name: "freebsd", version: "12.1-stable", arch: "amd64", family: "unix"
>
> The build completely stalls at
>> [INFO] Replacing
>> /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.jar
>> with
>> /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0-shaded.jar
>>
>> [INFO] Dependency-reduced POM written at:
>> /var/osipovmi/Projekte/maven-site-plugin/dependency-reduced-pom.xml
>
> CPU time is consumed like hell, I killed the process after 10 min.
>
> Looking at it with JConsole shows that main thread is heavy working on
>
>> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:317)
>>
>> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
>>
>> org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:340)
>>
>> org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:203)
>>
>> org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.resolveDependencies(Maven31DependencyGraphBuilder.java:124)
>>
>> org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.buildDependencyGraph(Maven31DependencyGraphBuilder.java:110)
>>
>> org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:98)
>>
>> org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:67
>>
>> org.apache.maven.plugins.shade.mojo.ShadeMojo.updateExcludesInDeps(ShadeMojo.java:1266)
>>
>> org.apache.maven.plugins.shade.mojo.ShadeMojo.rewriteDependencyReducedPomIfWeHaveReduction(ShadeMojo.java:1188)
>>
>> org.apache.maven.plugins.shade.mojo.ShadeMojo.createDependencyReducedPom(ShadeMojo.java:1098)
>>
>> org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:599)
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPlug
>>
>
> This is a complete contrast to Maven 3.5.4 and not related to this new
> plugin. A mere "mvn clean verify" on MSITE stalls completely during
> shade. Need to test more.

OK, found it:

> 716cc1fe02661897232a7cc3e4c1bb3b3df3b832 is the first bad commit
> commit 716cc1fe02661897232a7cc3e4c1bb3b3df3b832
> Author: rfscholte <[hidden email]>
> Date:   Wed Jan 29 21:18:42 2020 +0100
>
>     [MNG-5669] same pom.xml is read multiple times
>
>  .../java/org/apache/maven/building/FileSource.java |  31 ++++
>  .../org/apache/maven/building/StringSource.java    |  33 +++-
>  .../java/org/apache/maven/building/UrlSource.java  |  32 +++-
>  .../apache/maven/project/ReactorModelCache.java    |  78 +++++++-
>  .../maven/model/building/ArtifactModelSource.java  |  59 ++++++
>  .../maven/model/building/DefaultModelBuilder.java  | 206 ++++++++++++++++-----
>  .../maven/model/building/FileModelSource.java      |   9 +-
>  .../apache/maven/model/building/ModelCache.java    |  29 +++
>  .../apache/maven/model/building/ModelCacheTag.java |  26 +++
>  .../model/superpom/DefaultSuperPomProvider.java    |   2 +-
>  .../internal/DefaultArtifactDescriptorReader.java  |   7 +-
>  .../repository/internal/DefaultModelResolver.java  |   7 +-
>  12 files changed, 451 insertions(+), 68 deletions(-)
>  create mode 100644 maven-model-builder/src/main/java/org/apache/maven/model/b


@Robert, do you want to revert? This requires more testing obviously.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
In reply to this post by Hervé BOUTEMY
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:

> Hi,
>
> Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository.
>
> For a live example, see the last paragraph of Maven Site Plugin vote that just started [1].
>
> Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]:
> 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository
> 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference.
>
> Now I want to discuss: is it clear? can you test and report, please?
>
> If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository.

After even reverting the offending commit from Maven master, I still get:

> [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @ maven-site-plugin ---
> [INFO] Saved info on build to /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo
> [INFO] Checking against reference build from https://repository.apache.org/content/repositories/maven-1554/...
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time:  01:12 min
> [INFO] Finished at: 2020-03-07T14:16:18+01:00
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) on project maven-site-plugin: Error resolving reference artifact org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0: Could not transfer artifact org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0 from/to reference (https://repository.apache.org/content/repositories/maven-1554/): Cannot access https://repository.apache.org/content/repositories/maven-1554/ with type  using the available connector factories: BasicRepositoryConnectorFactory: Cannot access https://repository.apache.org/content/repositories/maven-1554/ with type  using the available layout factories: Maven2RepositoryLayoutFactory: Unsupported repository layout -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Karl Heinz Marbaise-3
Hi,

On 07.03.20 14:19, Michael Osipov wrote:

> Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:
>> Hi,
>>
>> Yesterday, I made a key step forward for Reproducible Builds with
>> Maven: I wrote code to easily check that your local build produces the
>> same binaries as the reference binaries published either to staging or
>> to Central repository.
>>
>> For a live example, see the last paragraph of Maven Site Plugin vote
>> that just started [1].
>>
>> Process to check build output is based on a single plugin goal,
>> currently named buildinfo:save [2]:
>> 1. it creates a buildinfo file during build recording output
>> fingerprints, that will eventually in the future be published to
>> Central repository
>> 2. it downloads reference artifacts and/or reference buildinfo and
>> checks that the output of the local build is the same as the reference.
>>
>> Now I want to discuss: is it clear? can you test and report, please?
>>
>> If the feedback is positive, the next question will be: in which
>> plugin should we put this goal to make a release and add it to our
>> parent pom during release, so we publish reference buildinfo along our
>> reference binaries to Central repository.
>
> After even reverting the offending commit from Maven master, I still get:
>
>> [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @
>> maven-site-plugin ---
>> [INFO] Saved info on build to
>> /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo
>>
>> [INFO] Checking against reference build from
>> https://repository.apache.org/content/repositories/maven-1554/...
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time:  01:12 min
>> [INFO] Finished at: 2020-03-07T14:16:18+01:00
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-buildinfo-plugin:1.0-SNAPSHOT:save
>> (default-cli) on project maven-site-plugin: Error resolving reference
>> artifact org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0:
>> Could not transfer artifact
>> org.apache.maven.plugins:maven-site-plugin:buildinfo:3.9.0 from/to
>> reference
>> (https://repository.apache.org/content/repositories/maven-1554/):
>> Cannot access
>> https://repository.apache.org/content/repositories/maven-1554/ with
>> type  using the available connector factories:
>> BasicRepositoryConnectorFactory: Cannot access
>> https://repository.apache.org/content/repositories/maven-1554/ with
>> type  using the available layout factories:
>> Maven2RepositoryLayoutFactory: Unsupported repository layout -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with
>> the -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>

That's exactly the same issue I have reported with Maven 3.6.3 ...

Kind regards
Karl Heinz Marbaise

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
In reply to this post by Hervé BOUTEMY
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:

> Hi,
>
> Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository.
>
> For a live example, see the last paragraph of Maven Site Plugin vote that just started [1].
>
> Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]:
> 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository
> 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference.
>
> Now I want to discuss: is it clear? can you test and report, please?
>
> If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository.

Made some progress:

> [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @ maven-site-plugin ---
> [INFO] Saved info on build to /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo
> [INFO] Checking against reference build from https://repository.apache.org/content/repositories/maven-1554/...
> [WARNING] Reference buildinfo file not found: it will be generated from downloaded reference artifacts
> [INFO] Minimal buildinfo generated from downloaded artifacts: /var/osipovmi/Projekte/maven-site-plugin/target/reference/maven-site-plugin-3.9.0.buildinfo
> [WARNING] size mismatch maven-site-plugin-3.9.0.jar: diffoscope target/reference/maven-site-plugin-3.9.0.jar target/maven-site-plugin-3.9.0.jar
> [WARNING] size mismatch maven-site-plugin-3.9.0-sources.jar: diffoscope target/reference/maven-site-plugin-3.9.0-sources.jar target/maven-site-plugin-3.9.0-sources.jar
> [WARNING] size mismatch maven-site-plugin-3.9.0-source-release.zip: diffoscope target/reference/maven-site-plugin-3.9.0-source-release.zip target/maven-site-plugin-3.9.0-source-release.zip
> [WARNING] Reproducible Build output summary: 0 files ok, 3 different, 0 missing
> [WARNING] diff target/reference/maven-site-plugin-3.9.0.buildinfo target/maven-site-plugin-3.9.0.buildinfo

This is expected because I am on 1.8.0_242. I don't have Java 7
installed anymore on the server.

As note, reproducibility after some time is not always possible if
nessary compilers/tools aren't available anymore -- as you can see.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Hervé BOUTEMY
Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> This is expected because I am on 1.8.0_242. I don't have Java 7
> installed anymore on the server.
for the discussion I wanted us to have, just being able to test and see how we
detect issues, this is perfect, isn't it?
how did you find the experience? any improvement proposal?
and any idea on where to put this goal in the future?

>
> As note, reproducibility after some time is not always possible if
> nessary compilers/tools aren't available anymore -- as you can see.
when we absolutely want to rebuild, this is where containers can ease the job

Regards,

Hervé

>
> Michael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
In reply to this post by Hervé BOUTEMY
Diff on OpenJDK 11:

> ├── META-INF/MANIFEST.MF
> │ @@ -1,10 +1,10 @@
> │  Manifest-Version: 1.0
> │ +Implementation-Vendor: The Apache Software Foundation^M
> │ +Implementation-Title: Apache Maven Site Plugin^M
> │ +Implementation-Version: 3.9.0^M
> │ +Build-Jdk-Spec: 1.7^M
> │ +Specification-Vendor: The Apache Software Foundation^M
> │  Created-By: Maven Jar Plugin 3.2.0
> │ -Build-Jdk-Spec: 11^M
> │  Specification-Title: Apache Maven Site Plugin
> │  Specification-Version: 3.9
> │ -Specification-Vendor: The Apache Software Foundation^M
> │ -Implementation-Title: Apache Maven Site Plugin^M
> │ -Implementation-Version: 3.9.0^M
> │ -Implementation-Vendor: The Apache Software Foundation^M

It seems like the hash implementation differs from version to version...


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Elliotte Rusty Harold
In reply to this post by michaelo
On Sat, Mar 7, 2020 at 11:39 AM Michael Osipov <[hidden email]> wrote:
>

> As note, reproducibility after some time is not always possible if
> nessary compilers/tools aren't available anymore -- as you can see.
>

That's an important point. Some organizations archive their entire
build chain including compilers and other tools in the source
repository.

I haven't seen it done, but I imagine you could go further using
Docker images as the source of the reproducible build.


--
Elliotte Rusty Harold
[hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> > Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> >> This is expected because I am on 1.8.0_242. I don't have Java 7
> >> installed anymore on the server.
> >
> > for the discussion I wanted us to have, just being able to test and see
> > how we detect issues, this is perfect, isn't it?
>
> This is really nice. Here is the diffoscope output:
you're discovering the wonders of diffoscope :)

> > --- maven-site-plugin-3.9.0.jar
> > +++ reference/maven-site-plugin-3.9.0.jar
> > ├── zipinfo {}
> > │ @@ -1,8 +1,8 @@
[...]

> > META-INF/MANIFEST.MF
> > │ @@ -1,10 +1,10 @@
> > │  Manifest-Version: 1.0
> > │ +Implementation-Vendor: The Apache Software Foundation^M
> > │  Implementation-Title: Apache Maven Site Plugin
> > │  Implementation-Version: 3.9.0
> > │ +Build-Jdk-Spec: 1.7^M
> > │  Specification-Vendor: The Apache Software Foundation
> > │ -Specification-Title: Apache Maven Site Plugin^M
> > │ -Build-Jdk-Spec: 1.8^M
> > │  Created-By: Maven Jar Plugin 3.2.0
> > │ +Specification-Title: Apache Maven Site Plugin^M
> > │  Specification-Version: 3.9
> > │ -Implementation-Vendor: The Apache Software Foundation^M
>
> I wonder where the CRs code from...this could be the default
> serialization format on every platform.
FYI I don't have such CRs in output on my Linux box

>
> > how did you find the experience? any improvement proposal?
> > and any idea on where to put this goal in the future?
>
> There is room for improvement when I quickly read the code. I will write
> separately on this.
sure, code can be improved: don't hesitate
but I was not asking yet for code improvement (I'm confident, it will happen)
but *experience* improvement

> I'd leave as a plugin for now.
you mean a separate plugin? same "buildinfo" name as current? "save" goal
name?

> At least in 3.7.x.
3.7.x as Maven 3.7.x?
does that mean that you think it should be one day integrated into Maven core?
what's the rationale?

Regards,

Hervé

>
> M





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:

> Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
>> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
>>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
>>>> This is expected because I am on 1.8.0_242. I don't have Java 7
>>>> installed anymore on the server.
>>>
>>> for the discussion I wanted us to have, just being able to test and see
>>> how we detect issues, this is perfect, isn't it?
>>
>> This is really nice. Here is the diffoscope output:
> you're discovering the wonders of diffoscope :)
>
>>> --- maven-site-plugin-3.9.0.jar
>>> +++ reference/maven-site-plugin-3.9.0.jar
>>> ├── zipinfo {}
>>> │ @@ -1,8 +1,8 @@
> [...]
>>> META-INF/MANIFEST.MF
>>> │ @@ -1,10 +1,10 @@
>>> │  Manifest-Version: 1.0
>>> │ +Implementation-Vendor: The Apache Software Foundation^M
>>> │  Implementation-Title: Apache Maven Site Plugin
>>> │  Implementation-Version: 3.9.0
>>> │ +Build-Jdk-Spec: 1.7^M
>>> │  Specification-Vendor: The Apache Software Foundation
>>> │ -Specification-Title: Apache Maven Site Plugin^M
>>> │ -Build-Jdk-Spec: 1.8^M
>>> │  Created-By: Maven Jar Plugin 3.2.0
>>> │ +Specification-Title: Apache Maven Site Plugin^M
>>> │  Specification-Version: 3.9
>>> │ -Implementation-Vendor: The Apache Software Foundation^M
>>
>> I wonder where the CRs code from...this could be the default
>> serialization format on every platform.
> FYI I don't have such CRs in output on my Linux box

This cannot be. See
https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/share/classes/java/util/jar/Manifest.java 
and search for \r\n. Old Sun code uses *always* CRLF. Plase recheck on
your side and run a hexdump on the Manifest file.

>>> how did you find the experience? any improvement proposal?
>>> and any idea on where to put this goal in the future?
>>
>> There is room for improvement when I quickly read the code. I will write
>> separately on this.
> sure, code can be improved: don't hesitate
> but I was not asking yet for code improvement (I'm confident, it will happen)
> but *experience* improvement
>
>> I'd leave as a plugin for now.
> you mean a separate plugin? same "buildinfo" name as current? "save" goal
> name?

OK, let's talk about experience:

* buildinfo may be changed to broader name, e.g.,
maven-reproducibility-plugin. Explanain follows
* 'save' does too much. It should save only and not compare. Save should
either run at initialize or at build-resources phase, imho
* Add a 'compare' goal, not phase bound. It performs the actual comparsion.

Strictly speaking if the plugin is called buildinfo it should handle the
buildinfo files only.

>> At least in 3.7.x.
> 3.7.x as Maven 3.7.x?
> does that mean that you think it should be one day integrated into Maven core?
> what's the rationale?

Not really, but if this happens, not before 4.x. I don't have any
rationale or entry point for this yet.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

rfscholte
unpack and get (without transitive dependencies) are candidates to me. 
Having extra goals makes the plugin more interesting.

Robert
On 8-3-2020 23:25:11, Hervé BOUTEMY <[hidden email]> wrote:
clearly, save goal is not a good choice: buildinfo would be better

I know buildinfo is not a usual term, but it's widely used in Reproducible
Builds [1] & [2], then it would be nice us Maven not to reinvent a wheel that
has already been invented

on separating checking, I really don't see how this improves experience

I love this idea of maven-artifact-plugin, but I don't see which goals od
maven-dependency-plugin could go in:
https://maven.apache.org/plugins/maven-dependency-plugin/

Regards,

Hervé

[1] https://reproducible-builds.org/docs/jvm/

[2] https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles

Le dimanche 8 mars 2020, 21:04:56 CET Robert Scholte a écrit :

> I'm thinking of maven-artifact-plugin, having goals related to artifacts.
> That implies that the save goal should be renamed.
> A couple of goals of the maven-dependency-plugin are actually more
> artifact-related are might be worth moving.
>
> Robert
>
> On 8-3-2020 13:44:07, Michael Osipov wrote:
>
> Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> >>>> This is expected because I am on 1.8.0_242. I don't have Java 7
> >>>> installed anymore on the server.
> >>>
> >>> for the discussion I wanted us to have, just being able to test and see
> >>> how we detect issues, this is perfect, isn't it?
> >>
> >> This is really nice. Here is the diffoscope output:
> > you're discovering the wonders of diffoscope :)
> >
> >>> --- maven-site-plugin-3.9.0.jar
> >>> +++ reference/maven-site-plugin-3.9.0.jar
> >>> ├── zipinfo {}
> >>> │ @@ -1,8 +1,8 @@
> >
> > [...]
> >
> >>> META-INF/MANIFEST.MF
> >>> │ @@ -1,10 +1,10 @@
> >>> │ Manifest-Version: 1.0
> >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> >>> │ Implementation-Title: Apache Maven Site Plugin
> >>> │ Implementation-Version: 3.9.0
> >>> │ +Build-Jdk-Spec: 1.7^M
> >>> │ Specification-Vendor: The Apache Software Foundation
> >>> │ -Specification-Title: Apache Maven Site Plugin^M
> >>> │ -Build-Jdk-Spec: 1.8^M
> >>> │ Created-By: Maven Jar Plugin 3.2.0
> >>> │ +Specification-Title: Apache Maven Site Plugin^M
> >>> │ Specification-Version: 3.9
> >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> >>
> >> I wonder where the CRs code from...this could be the default
> >> serialization format on every platform.
> >
> > FYI I don't have such CRs in output on my Linux box
>
> This cannot be. See
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/sha
> re/classes/java/util/jar/Manifest.java and search for \r\n. Old Sun code
> uses *always* CRLF. Plase recheck on your side and run a hexdump on the
> Manifest file.
>
> >>> how did you find the experience? any improvement proposal?
> >>> and any idea on where to put this goal in the future?
> >>
> >> There is room for improvement when I quickly read the code. I will write
> >> separately on this.
> >
> > sure, code can be improved: don't hesitate
> > but I was not asking yet for code improvement (I'm confident, it will
> > happen) but *experience* improvement
> >
> >> I'd leave as a plugin for now.
> >
> > you mean a separate plugin? same "buildinfo" name as current? "save" goal
> > name?
>
> OK, let's talk about experience:
>
> * buildinfo may be changed to broader name, e.g.,
> maven-reproducibility-plugin. Explanain follows
> * 'save' does too much. It should save only and not compare. Save should
> either run at initialize or at build-resources phase, imho
> * Add a 'compare' goal, not phase bound. It performs the actual comparsion.
>
> Strictly speaking if the plugin is called buildinfo it should handle the
> buildinfo files only.
>
> >> At least in 3.7.x.
> >
> > 3.7.x as Maven 3.7.x?
> > does that mean that you think it should be one day integrated into Maven
> > core? what's the rationale?
>
> Not really, but if this happens, not before 4.x. I don't have any
> rationale or entry point for this yet.
>
> Michael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
yes, I saw that the main artifact is reproducible, but there are more subtle
cases with attached artifacts (-sources.jar and -source-release.zip)

If you build with run-its profile, you'll see that the pom.xml injected into
these artifacts has less differences: there is still the current directory in
it :(
It seems it is caused by additional maven-invoker-plugin configuration done in
run-its profile, that seems to replace original pom.xml with something
generated from invoker: I did not investigate more yet, any help from maven-
invoker-plugin experts appreciated

FYI I tested current maven-dependency-plugin release and found that it does
not suffer from this issue.

Regards,

Hervé

Le mardi 10 mars 2020, 13:11:42 CET Michael Osipov a écrit :

> Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:
> > Hi,
> >
> > Yesterday, I made a key step forward for Reproducible Builds with Maven: I
> > wrote code to easily check that your local build produces the same
> > binaries as the reference binaries published either to staging or to
> > Central repository.
> >
> > For a live example, see the last paragraph of Maven Site Plugin vote that
> > just started [1].
> >
> > Process to check build output is based on a single plugin goal, currently
> > named buildinfo:save [2]: 1. it creates a buildinfo file during build
> > recording output fingerprints, that will eventually in the future be
> > published to Central repository 2. it downloads reference artifacts
> > and/or reference buildinfo and checks that the output of the local build
> > is the same as the reference.
> >
> > Now I want to discuss: is it clear? can you test and report, please?
> >
> > If the feedback is positive, the next question will be: in which plugin
> > should we put this goal to make a release and add it to our parent pom
> > during release, so we publish reference buildinfo along our reference
> > binaries to Central repository.
> >
> > Thanks for your feedback
> >
> > Regards,
> >
> > Hervé
> >
> > [1]
> > https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5
> > a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E
> >
> > [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin
>
> I have now installed latest OpenJDK 7 from AdoptOpenJDK source.
>
> > [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @
> > maven-site-plugin --- [INFO] Saved info on build to
> > /usr/home/mosipov/Projekte/maven-site-plugin/target/maven-site-plugin-3.9
> > .0.buildinfo [INFO] Checking against reference build from
> > https://repository.apache.org/content/repositories/maven-1554/...
> > [WARNING] Reference buildinfo file not found: it will be generated from
> > downloaded reference artifacts [INFO] Minimal buildinfo generated from
> > downloaded artifacts:
> > /usr/home/mosipov/Projekte/maven-site-plugin/target/reference/maven-site-
> > plugin-3.9.0.buildinfo [WARNING] size mismatch
> > maven-site-plugin-3.9.0-source-release.zip: diffoscope
> > target/reference/maven-site-plugin-3.9.0-source-release.zip
> > target/maven-site-plugin-3.9.0-source-release.zip [WARNING] size mismatch
> > maven-site-plugin-3.9.0-sources.jar: diffoscope
> > target/reference/maven-site-plugin-3.9.0-sources.jar
> > target/maven-site-plugin-3.9.0-sources.jar [WARNING] Reproducible Build
> > output summary: 1 files ok, 2 different, 0 missing [WARNING] diff
> > target/reference/maven-site-plugin-3.9.0.buildinfo
> > target/maven-site-plugin-3.9.0.buildinfo
> on
>
> > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> > Maven home: /usr/local/share/java/maven
> > Java version: 1.7.0_251, vendor: Oracle Corporation, runtime:
> > /usr/local/openjdk7/jre Default locale: de_DE, platform encoding: UTF-8
> > OS name: "freebsd", version: "11.3-release-p6", arch: "i386", family:
> > "unix"
> and
>
> > $ git branch
> > * (HEAD losgelöst bei maven-site-plugin-3.9.0)
> >
> >> diffoscope target/reference/maven-site-plugin-3.9.0-source-release.zip
> >> target/maven-site-plugin-3.9.0-source-release.zip>
> > There is a diff in maven-site-plugin-3.9.0/dependency-reduced-pom.xml
> >
> >> diffoscope target/reference/maven-site-plugin-3.9.0-sources.jar
> >> target/maven-site-plugin-3.9.0-sources.jar>
> > So is here diff in the pom.xml which is actually
> > dependency-reduced-pom.xml.
> >
> > ├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml
> > │ ├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml
> > │ │ @@ -243,100 +243,40 @@
> > │ │    <profiles>
> > │ │      <profile>
> > │ │        <id>run-its</id>
> > │ │        <build>
> > │ │          <plugins>
> > │ │            <plugin>
> > │ │              <artifactId>maven-invoker-plugin</artifactId>
> > │ │ -            <version>3.2.1</version>
> > │ │ -            <executions>
> > │ │ -              <execution>
> > │ │ -                <id>integration-test</id>
> > │ │ -                <goals>
> > │ │ -                  <goal>install</goal>
> > │ │ -                  <goal>integration-test</goal>
> > │ │ -                  <goal>verify</goal>
> > │ │ -                </goals>
> > │ │ -                <configuration>
> > │ │ -                
> > <projectsDirectory>src/it/projects</projectsDirectory> │ │ -            
> >     <settingsFile>src/it/mrm/settings.xml</settingsFile> │ │ -          
> >       <filterProperties>
> > │ │ -                  
> > <mrm.repository.url>${mrm.repository.url}</mrm.repository.url> │ │ -    
> >             </filterProperties>
> > │ │ -                  <goals>
> > │ │ -                    <goal>clean</goal>
> > │ │ -                  
> > <goal>org.apache.maven.plugins:maven-site-plugin:3.9.0:site</goal> │ │ -
> >                 </goals>
> > │ │ -                  <properties>
> > │ │ -                  
> > <maven.compiler.source>1.7</maven.compiler.source>
> > │ │ -                  
> > <maven.compiler.target>1.7</maven.compiler.target>
> > │ │ -                  
> > <https.protocols>TLSv1,TLSv1.1,TLSv1.2</https.protocols> │ │ -          
> >       </properties>
> > │ │ -                  <debug>true</debug>
> > │ │ -                
> > <cloneProjectsTo>/home/herve/projets/maven/sources/plugins/core/maven-sit
> > e-plugin/target/checkout/target/it</cloneProjectsTo> │ │ -                
> >  <preBuildHookScript>setup</preBuildHookScript> │ │ -                
> > <postBuildHookScript>verify</postBuildHookScript> │ │ -                
> > <localRepositoryPath>/home/herve/projets/maven/sources/plugins/core/maven
> > -site-plugin/target/checkout/target/local-repo</localRepositoryPath> │ │ -
> >                  <pomIncludes>
> > │ │ -                    <pomInclude>*/pom.xml</pomInclude>
> > │ │ -                  </pomIncludes>
> > │ │ -                  <ignoreFailures>false</ignoreFailures>
> > │ │ -                  <environmentVariables>
> > │ │ -                  
> > <JENKINS_MAVEN_AGENT_DISABLED>true</JENKINS_MAVEN_AGENT_DISABLED> │ │ -  
> >                </environmentVariables>
> > │ │ -                </configuration>
> > │ │ -              </execution>
> > │ │ -            </executions>
> > │ │              <configuration>
> > │ │                <projectsDirectory>src/it/projects</projectsDirectory>
> > │ │                <settingsFile>src/it/mrm/settings.xml</settingsFile>
> > │ │                <filterProperties>
> > │ │                
> > <mrm.repository.url>${mrm.repository.url}</mrm.repository.url> │ │      
> >         </filterProperties>
> > │ │                <goals>
> > │ │                  <goal>clean</goal>
> > │ │ -              
> > <goal>org.apache.maven.plugins:maven-site-plugin:3.9.0:site</goal> │ │ +
> >              
> > <goal>${project.groupId}:${project.artifactId}:${project.version}:site</g
> > oal> │ │                </goals>
> > │ │                <properties>
> > │ │ -                <maven.compiler.source>1.7</maven.compiler.source>
> > │ │ -                <maven.compiler.target>1.7</maven.compiler.target>
> > │ │ -              
> > <https.protocols>TLSv1,TLSv1.1,TLSv1.2</https.protocols> │ │ +          
> >    
> > <maven.compiler.source>${maven.compiler.source}</maven.compiler.source> │
> > │ +              
> > <maven.compiler.target>${maven.compiler.target}</maven.compiler.target> │
> > │                </properties>
> > │ │ -              <debug>true</debug>
> > │ │ -            
> > <cloneProjectsTo>/home/herve/projets/maven/sources/plugins/core/maven-sit
> > e-plugin/target/checkout/target/it</cloneProjectsTo> │ │ -            
> > <preBuildHookScript>setup</preBuildHookScript>
> > │ │ -              <postBuildHookScript>verify</postBuildHookScript>
> > │ │ -            
> > <localRepositoryPath>/home/herve/projets/maven/sources/plugins/core/maven
> > -site-plugin/target/checkout/target/local-repo</localRepositoryPath> │ │ -
> >              <pomIncludes>
> > │ │ -                <pomInclude>*/pom.xml</pomInclude>
> > │ │ -              </pomIncludes>
> > │ │ -              <ignoreFailures>false</ignoreFailures>
> > │ │ -              <environmentVariables>
> > │ │ -              
> > <JENKINS_MAVEN_AGENT_DISABLED>true</JENKINS_MAVEN_AGENT_DISABLED> │ │ -  
> >            </environmentVariables>
> > │ │              </configuration>
> > │ │            </plugin>
> > │ │            <plugin>
> > │ │              <groupId>org.codehaus.mojo</groupId>
> > │ │              <artifactId>mrm-maven-plugin</artifactId>
> > │ │              <version>1.2.0</version>
> > │ │              <executions>
> > │ │                <execution>
> > │ │                  <goals>
> > │ │                    <goal>start</goal>
> > │ │                    <goal>stop</goal>
> > │ │                  </goals>
> > │ │ -                <configuration>
> > │ │ -                  <repositories>
> > │ │ -                    <mockRepo>
> > │ │ -                      <source>src/it/mrm/repository</source>
> > │ │ -                    </mockRepo>
> > │ │ -                    <proxyRepo/>
> > │ │ -                  </repositories>
> > │ │ -                </configuration>
> > │ │                </execution>
> > │ │              </executions>
> > │ │              <configuration>
> > │ │                <repositories>
> > │ │                  <mockRepo>
> > │ │                    <source>src/it/mrm/repository</source>
> > │ │                  </mockRepo>
>
> Any idea why my POM differs from your one?
>
> Michael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]