Re: Talk: Bootstrapping the Java Ecosystem

Previous Topic Next Topic
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view

Re: Talk: Bootstrapping the Java Ecosystem

Björn Höfling
On Thu, 19 Nov 2020 09:15:07 +0100
Emmanuel Bourg <[hidden email]> wrote:

> Nice presentation, the packaging of Maven in Debian followed a similar
> path but we never documented the process. Did you go as far as
> recording the exact steps and build order required to build from
> scratch?

First of all, let my clarify that honor goes to Julien or the Guix
community or the Reproducible-Builds community. I'm merely
communicating the results here.

I'm not sure if this is what you mean by "recording the exact steps",
but you can use "guix graph maven" to get a (huge) dot/GraphViz
dependency tree. That is your record of the work.

> Spoiler for the next part of your quest toward packaging the Android
> SDK: Maven was the easy part, Gradle and Kotlin are many leagues above
> in term of circular dependencies and headache. We have been trying to
> bootstrap Kotlin for 2 years in Debian and haven't found the right
> path yet.

Yes, Android-SDK, Kotlin, Scala is much harder. Scala for example is
written in Scala. Julien started to build a Scala compiler in Guile,
I'm not sure what the current state is:


attachment0 (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view

Re: Talk: Bootstrapping the Java Ecosystem

Björn Höfling
On Thu, 19 Nov 2020 09:51:06 +0100
Tamás Cservenák <[hidden email]> wrote:

> Hi Bjorn and Emmanuel,
> Without starting any flame wars, am really curious: why are you
> repackaging Maven?

I also don't want any flame-wars :-) If this discussion goes to far
away from Maven, I would suggest to ask questions on the lists
guix-devel, guix-help or rb-general.

The intention of building Maven (and other programs) from source is the
old idea of software freedom: You as a user have the source code
available and have the rights to share and modify the software as you
like. For that, it is important that people really are able to build
the software from source, and also all dependencies and prerequisites
like compilers, instead of just downloading a binary or container and
run it. When software is packaged for Guix, you can effectively build
it (again) from source, with the exact same result, independent of the
time or machine you are on. You can even slightly modify the sources
and ask Guix to build it again for you.

> I'd understand for OS/distro native packages, but
> why do you rebuild JVM bytecode as well?

First, Guix is a package manager and is the main block of the
distribution "Guix System". For software freedom reasons, Guix is also
building JVM binaries from source.

> 1) Instead of rebuilding something (let's say LOC of 1x), you
> did that several, if not ten times (so LOC of 10x). So, you
> had to review the codebase 10x I guess? Otherwise you are not
> sure what you built - as I guess you build it from source
> to be sure and able to see (among others) what are you
> building. How long was the review process?
> 2) Similarly, in a process like this, how do you track
> vulnerabilities or any other outstanding bugs? Or the
> "intermediate" bootstrapped dependencies simply "does not
> matter"? Just the final "output" is what matters
> (let's say Maven)?
I would separate these topics: Guix (and other distros/package
managers) "just" provides a convenient, automated way to build and
package the final binary software from the sources.

The second thing is reviewing the software for security
vulnerabilities. As you said, this is a very time-consuming process
and not part of Guix. But having a reproducible build at hand you can
verify that a binary really was built from the sources someone claims
to stem from.

> 3) What are you really building? As in video, it is said
> several times that you "mutilate" some package to build
> it, then use it to "bootstrap" some other package, and finally
> you rebuild the target package. Given in the process there
> was once a "mutilated" tool, how are you certain, that output
> of the build is correct (I have no doubts about
> reproducibility)? How do you prove that output is what
> it is thought/assumed to be?

For example, to build the newest ant, Guix is first building jikes (an
old Java-compiler written in C for Java <1.6?), then it uses jikes to
build ant 1.8.4. That allows you to build icedtea and that allows you
to build the latest ant.

There is no guarantee that the final/latest ant that pops out of the
process would really work. You have to test that it really works as
expected. On the other hand, that is a which from package managers to
software developers (especially for languages/compilers): That they care
more about the bootstrapping process. That bootstrapping is defined and
as easy as possible. That maybe different ways of bootstrapping are
available and supported.
> 3) (Joker) What is the overall CO2 footprint of distros like
> these? I believe you did not use Apple M1 for this work... :)

As for other distributions, there is a "substitute server" available
that builds the software, you can just download the "official" distro's
binaries and don't need to compile everything for yourself.

Compared to all other computations we do these days, I would say it
doesn't count too much.


attachment0 (201 bytes) Download Attachment