Re: [DISCUSS] checking reproducible builds

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:

> Hi,
>
> Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository.
>
> For a live example, see the last paragraph of Maven Site Plugin vote that just started [1].
>
> Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]:
> 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository
> 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference.
>
> Now I want to discuss: is it clear? can you test and report, please?
>
> If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository.

Fails for me with:
> osipovmi@deblndw011x:~/var/Projekte/maven-site-plugin ((maven-site-plugin-3.9.0)
> $ ~/apache-maven-3.7.0-SNAPSHOT/bin/mvn -v
> Apache Maven 3.7.0-SNAPSHOT (f2e9afd788de919646717532d26eca38826e9924)
> Maven home: /net/home/osipovmi/apache-maven-3.7.0-SNAPSHOT
> Java version: 1.8.0_242, vendor: Oracle Corporation, runtime: /usr/local/openjdk8/jre
> Default locale: de_DE, platform encoding: UTF-8
> OS name: "freebsd", version: "12.1-stable", arch: "amd64", family: "unix"

The build completely stalls at
> [INFO] Replacing /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.jar with /var/osipovmi/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0-shaded.jar
> [INFO] Dependency-reduced POM written at: /var/osipovmi/Projekte/maven-site-plugin/dependency-reduced-pom.xml

CPU time is consumed like hell, I killed the process after 10 min.

Looking at it with JConsole shows that main thread is heavy working on

> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:317)
> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
> org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:340)
> org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:203)
> org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.resolveDependencies(Maven31DependencyGraphBuilder.java:124)
> org.apache.maven.shared.dependency.graph.internal.Maven31DependencyGraphBuilder.buildDependencyGraph(Maven31DependencyGraphBuilder.java:110)
> org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:98)
> org.apache.maven.shared.dependency.graph.internal.DefaultDependencyGraphBuilder.buildDependencyGraph(DefaultDependencyGraphBuilder.java:67
> org.apache.maven.plugins.shade.mojo.ShadeMojo.updateExcludesInDeps(ShadeMojo.java:1266)
> org.apache.maven.plugins.shade.mojo.ShadeMojo.rewriteDependencyReducedPomIfWeHaveReduction(ShadeMojo.java:1188)
> org.apache.maven.plugins.shade.mojo.ShadeMojo.createDependencyReducedPom(ShadeMojo.java:1098)
> org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:599)
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPlug

This is a complete contrast to Maven 3.5.4 and not related to this new
plugin. A mere "mvn clean verify" on MSITE stalls completely during
shade. Need to test more.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

Enrico Olivelli
Il Dom 8 Mar 2020, 13:44 Michael Osipov <[hidden email]> ha scritto:

> Am 2020-03-08 um 12:48 schrieb Hervé BOUTEMY:
> > Le dimanche 8 mars 2020, 00:31:07 CET Michael Osipov a écrit :
> >> Am 2020-03-07 um 19:04 schrieb Hervé BOUTEMY:
> >>> Le samedi 7 mars 2020, 17:39:20 CET Michael Osipov a écrit :
> >>>> This is expected because I am on 1.8.0_242. I don't have Java 7
> >>>> installed anymore on the server.
> >>>
> >>> for the discussion I wanted us to have, just being able to test and see
> >>> how we detect issues, this is perfect, isn't it?
> >>
> >> This is really nice. Here is the diffoscope output:
> > you're discovering the wonders of diffoscope :)
> >
> >>> --- maven-site-plugin-3.9.0.jar
> >>> +++ reference/maven-site-plugin-3.9.0.jar
> >>> ├── zipinfo {}
> >>> │ @@ -1,8 +1,8 @@
> > [...]
> >>> META-INF/MANIFEST.MF
> >>> │ @@ -1,10 +1,10 @@
> >>> │  Manifest-Version: 1.0
> >>> │ +Implementation-Vendor: The Apache Software Foundation^M
> >>> │  Implementation-Title: Apache Maven Site Plugin
> >>> │  Implementation-Version: 3.9.0
> >>> │ +Build-Jdk-Spec: 1.7^M
> >>> │  Specification-Vendor: The Apache Software Foundation
> >>> │ -Specification-Title: Apache Maven Site Plugin^M
> >>> │ -Build-Jdk-Spec: 1.8^M
> >>> │  Created-By: Maven Jar Plugin 3.2.0
> >>> │ +Specification-Title: Apache Maven Site Plugin^M
> >>> │  Specification-Version: 3.9
> >>> │ -Implementation-Vendor: The Apache Software Foundation^M
> >>
> >> I wonder where the CRs code from...this could be the default
> >> serialization format on every platform.
> > FYI I don't have such CRs in output on my Linux box
>
> This cannot be. See
>
> https://github.com/AdoptOpenJDK/openjdk-jdk11u/blob/master/src/java.base/share/classes/java/util/jar/Manifest.java
> and search for \r\n. Old Sun code uses *always* CRLF. Plase recheck on
> your side and run a hexdump on the Manifest file.
>
> >>> how did you find the experience? any improvement proposal?
> >>> and any idea on where to put this goal in the future?
> >>
> >> There is room for improvement when I quickly read the code. I will write
> >> separately on this.
> > sure, code can be improved: don't hesitate
> > but I was not asking yet for code improvement (I'm confident, it will
> happen)
> > but *experience* improvement
> >
> >> I'd leave as a plugin for now.
> > you mean a separate plugin? same "buildinfo" name as current? "save" goal
> > name?
>
> OK, let's talk about experience:
>
> * buildinfo may be changed to broader name, e.g.,
> maven-reproducibility-plugin.


+1

> Explanain follows
> * 'save' does too much. It should save only and not compare. Save should
> either run at initialize or at build-resources phase, imho
> * Add a 'compare' goal, not phase bound. It performs the actual comparsion.
>

+1 for splitting this way

Enrico


> Strictly speaking if the plugin is called buildinfo it should handle the
> buildinfo files only.
>
> >> At least in 3.7.x.
> > 3.7.x as Maven 3.7.x?
> > does that mean that you think it should be one day integrated into Maven
> core?
> > what's the rationale?
>
> Not really, but if this happens, not before 4.x. I don't have any
> rationale or entry point for this yet.
>
> Michael
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] checking reproducible builds

michaelo
In reply to this post by michaelo
Am 2020-03-07 um 11:36 schrieb Hervé BOUTEMY:

> Hi,
>
> Yesterday, I made a key step forward for Reproducible Builds with Maven: I wrote code to easily check that your local build produces the same binaries as the reference binaries published either to staging or to Central repository.
>
> For a live example, see the last paragraph of Maven Site Plugin vote that just started [1].
>
> Process to check build output is based on a single plugin goal, currently named buildinfo:save [2]:
> 1. it creates a buildinfo file during build recording output fingerprints, that will eventually in the future be published to Central repository
> 2. it downloads reference artifacts and/or reference buildinfo and checks that the output of the local build is the same as the reference.
>
> Now I want to discuss: is it clear? can you test and report, please?
>
> If the feedback is positive, the next question will be: in which plugin should we put this goal to make a release and add it to our parent pom during release, so we publish reference buildinfo along our reference binaries to Central repository.
>
> Thanks for your feedback
>
> Regards,
>
> Hervé
>
> [1] https://lists.apache.org/thread.html/rd3af15d383ddceeb950cd90569e3dcdd6e5a0f5d3cd653ec534b0609%40%3Cdev.maven.apache.org%3E
>
> [2] https://github.com/apache/maven-studies/tree/maven-buildinfo-plugin

I have now installed latest OpenJDK 7 from AdoptOpenJDK source.

> [INFO] --- maven-buildinfo-plugin:1.0-SNAPSHOT:save (default-cli) @ maven-site-plugin ---
> [INFO] Saved info on build to /usr/home/mosipov/Projekte/maven-site-plugin/target/maven-site-plugin-3.9.0.buildinfo
> [INFO] Checking against reference build from https://repository.apache.org/content/repositories/maven-1554/...
> [WARNING] Reference buildinfo file not found: it will be generated from downloaded reference artifacts
> [INFO] Minimal buildinfo generated from downloaded artifacts: /usr/home/mosipov/Projekte/maven-site-plugin/target/reference/maven-site-plugin-3.9.0.buildinfo
> [WARNING] size mismatch maven-site-plugin-3.9.0-source-release.zip: diffoscope target/reference/maven-site-plugin-3.9.0-source-release.zip target/maven-site-plugin-3.9.0-source-release.zip
> [WARNING] size mismatch maven-site-plugin-3.9.0-sources.jar: diffoscope target/reference/maven-site-plugin-3.9.0-sources.jar target/maven-site-plugin-3.9.0-sources.jar
> [WARNING] Reproducible Build output summary: 1 files ok, 2 different, 0 missing
> [WARNING] diff target/reference/maven-site-plugin-3.9.0.buildinfo target/maven-site-plugin-3.9.0.buildinfo

on
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Maven home: /usr/local/share/java/maven
> Java version: 1.7.0_251, vendor: Oracle Corporation, runtime: /usr/local/openjdk7/jre
> Default locale: de_DE, platform encoding: UTF-8
> OS name: "freebsd", version: "11.3-release-p6", arch: "i386", family: "unix"

and

> $ git branch
> * (HEAD losgelöst bei maven-site-plugin-3.9.0)

>> diffoscope target/reference/maven-site-plugin-3.9.0-source-release.zip target/maven-site-plugin-3.9.0-source-release.zip
>
> There is a diff in maven-site-plugin-3.9.0/dependency-reduced-pom.xml
>
>> diffoscope target/reference/maven-site-plugin-3.9.0-sources.jar target/maven-site-plugin-3.9.0-sources.jar
>
> So is here diff in the pom.xml which is actually dependency-reduced-pom.xml.

> ├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml
> │ ├── META-INF/maven/org.apache.maven.plugins/maven-site-plugin/pom.xml
> │ │ @@ -243,100 +243,40 @@
> │ │    <profiles>
> │ │      <profile>
> │ │        <id>run-its</id>
> │ │        <build>
> │ │          <plugins>
> │ │            <plugin>
> │ │              <artifactId>maven-invoker-plugin</artifactId>
> │ │ -            <version>3.2.1</version>
> │ │ -            <executions>
> │ │ -              <execution>
> │ │ -                <id>integration-test</id>
> │ │ -                <goals>
> │ │ -                  <goal>install</goal>
> │ │ -                  <goal>integration-test</goal>
> │ │ -                  <goal>verify</goal>
> │ │ -                </goals>
> │ │ -                <configuration>
> │ │ -                  <projectsDirectory>src/it/projects</projectsDirectory>
> │ │ -                  <settingsFile>src/it/mrm/settings.xml</settingsFile>
> │ │ -                  <filterProperties>
> │ │ -                    <mrm.repository.url>${mrm.repository.url}</mrm.repository.url>
> │ │ -                  </filterProperties>
> │ │ -                  <goals>
> │ │ -                    <goal>clean</goal>
> │ │ -                    <goal>org.apache.maven.plugins:maven-site-plugin:3.9.0:site</goal>
> │ │ -                  </goals>
> │ │ -                  <properties>
> │ │ -                    <maven.compiler.source>1.7</maven.compiler.source>
> │ │ -                    <maven.compiler.target>1.7</maven.compiler.target>
> │ │ -                    <https.protocols>TLSv1,TLSv1.1,TLSv1.2</https.protocols>
> │ │ -                  </properties>
> │ │ -                  <debug>true</debug>
> │ │ -                  <cloneProjectsTo>/home/herve/projets/maven/sources/plugins/core/maven-site-plugin/target/checkout/target/it</cloneProjectsTo>
> │ │ -                  <preBuildHookScript>setup</preBuildHookScript>
> │ │ -                  <postBuildHookScript>verify</postBuildHookScript>
> │ │ -                  <localRepositoryPath>/home/herve/projets/maven/sources/plugins/core/maven-site-plugin/target/checkout/target/local-repo</localRepositoryPath>
> │ │ -                  <pomIncludes>
> │ │ -                    <pomInclude>*/pom.xml</pomInclude>
> │ │ -                  </pomIncludes>
> │ │ -                  <ignoreFailures>false</ignoreFailures>
> │ │ -                  <environmentVariables>
> │ │ -                    <JENKINS_MAVEN_AGENT_DISABLED>true</JENKINS_MAVEN_AGENT_DISABLED>
> │ │ -                  </environmentVariables>
> │ │ -                </configuration>
> │ │ -              </execution>
> │ │ -            </executions>
> │ │              <configuration>
> │ │                <projectsDirectory>src/it/projects</projectsDirectory>
> │ │                <settingsFile>src/it/mrm/settings.xml</settingsFile>
> │ │                <filterProperties>
> │ │                  <mrm.repository.url>${mrm.repository.url}</mrm.repository.url>
> │ │                </filterProperties>
> │ │                <goals>
> │ │                  <goal>clean</goal>
> │ │ -                <goal>org.apache.maven.plugins:maven-site-plugin:3.9.0:site</goal>
> │ │ +                <goal>${project.groupId}:${project.artifactId}:${project.version}:site</goal>
> │ │                </goals>
> │ │                <properties>
> │ │ -                <maven.compiler.source>1.7</maven.compiler.source>
> │ │ -                <maven.compiler.target>1.7</maven.compiler.target>
> │ │ -                <https.protocols>TLSv1,TLSv1.1,TLSv1.2</https.protocols>
> │ │ +                <maven.compiler.source>${maven.compiler.source}</maven.compiler.source>
> │ │ +                <maven.compiler.target>${maven.compiler.target}</maven.compiler.target>
> │ │                </properties>
> │ │ -              <debug>true</debug>
> │ │ -              <cloneProjectsTo>/home/herve/projets/maven/sources/plugins/core/maven-site-plugin/target/checkout/target/it</cloneProjectsTo>
> │ │ -              <preBuildHookScript>setup</preBuildHookScript>
> │ │ -              <postBuildHookScript>verify</postBuildHookScript>
> │ │ -              <localRepositoryPath>/home/herve/projets/maven/sources/plugins/core/maven-site-plugin/target/checkout/target/local-repo</localRepositoryPath>
> │ │ -              <pomIncludes>
> │ │ -                <pomInclude>*/pom.xml</pomInclude>
> │ │ -              </pomIncludes>
> │ │ -              <ignoreFailures>false</ignoreFailures>
> │ │ -              <environmentVariables>
> │ │ -                <JENKINS_MAVEN_AGENT_DISABLED>true</JENKINS_MAVEN_AGENT_DISABLED>
> │ │ -              </environmentVariables>
> │ │              </configuration>
> │ │            </plugin>
> │ │            <plugin>
> │ │              <groupId>org.codehaus.mojo</groupId>
> │ │              <artifactId>mrm-maven-plugin</artifactId>
> │ │              <version>1.2.0</version>
> │ │              <executions>
> │ │                <execution>
> │ │                  <goals>
> │ │                    <goal>start</goal>
> │ │                    <goal>stop</goal>
> │ │                  </goals>
> │ │ -                <configuration>
> │ │ -                  <repositories>
> │ │ -                    <mockRepo>
> │ │ -                      <source>src/it/mrm/repository</source>
> │ │ -                    </mockRepo>
> │ │ -                    <proxyRepo/>
> │ │ -                  </repositories>
> │ │ -                </configuration>
> │ │                </execution>
> │ │              </executions>
> │ │              <configuration>
> │ │                <repositories>
> │ │                  <mockRepo>
> │ │                    <source>src/it/mrm/repository</source>
> │ │                  </mockRepo>

Any idea why my POM differs from your one?

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]