Re: Talk: Bootstrapping the Java Ecosystem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Talk: Bootstrapping the Java Ecosystem

Tamás Cservenák
Thanks for the answers!

AFAIK, we in Apache as well "vote for source", while we provide binaries as
well.

Given the video mentions that Maven `-sources` artifacts are NOT buildable
(which is true, they are mainly used by IDEs to display library sources
while debug for example), am unsure -- at least for ASF artifacts -- why
not using then source release bundles instead? For example:
https://dist.apache.org/repos/dist/release/maven/maven-3/3.6.3/source/

Also, according to your explanation, the problem is now solved once for
all, right? You do have (those distros you mention, like Guix) Maven 3.6.3
built now, so you do not have to repeat this anymore?

My point is that Maven devs also use Maven 3.6.3 currently, and that
version will be used to build any future Maven release as well (ie. 3.6.4
or 4.0.0 and so on). So, you just had to "hop on" the bandwagon, do this
"dance" once, but from now on, all this work can be scraped, right?

Thanks
Tamas

On Fri, Nov 20, 2020 at 9:49 AM Emmanuel Bourg <[hidden email]> wrote:

> On 19/11/2020 09:51, Tamás Cservenák wrote:
>
> > Without starting any flame wars, am really curious: why are you
> > repackaging Maven?
> >
> > I'd understand for OS/distro native packages, but
> > why do you rebuild JVM bytecode as well?
> >
> > Again, am not to start any flame war, am just curious!
>
> Short answer: why not? This is an Open Source project, not an Open
> Binary project. Anyone should be able to rebuild the code, and in an
> ideal world where every project is reproducible, get byte identical
> binaries.
>
> Long answer: Debian, Fedora, and I assume Guix are "closed" ecosystems
> where you can rebuild every component from sources without needing tools
> or libraries outside of the distribution. If you were alone on a desert
> island with just a laptop, the sources and no internet connection, you
> would be able to rebuild any part of the distribution from scratch.
>
> This really goes to the roots of the open source philosophy, open source
> projects are meant to be built from sources, and if it's not possible
> then there is a problem somewhere. Assuming every project becomes
> reproducible at some point (see https://reproducible-builds.org for why
> it matters) the question of knowing who produced the binaries become
> irrelevant, because everyone get the exact same binaries.
>
>
> > 3) What are you really building? As in video, it is said
> > several times that you "mutilate" some package to build
> > it, then use it to "bootstrap" some other package, and finally
> > you rebuild the target package. Given in the process there
> > was once a "mutilated" tool, how are you certain, that output
> > of the build is correct (I have no doubts about
> > reproducibility)? How do you prove that output is what
> > it is thought/assumed to be?
>
> In Debian the Maven package we rebuild from sources is itself used to
> build all the other Maven based projects packaged in Debian (that's over
> 600 projects currently), so regressions are caught pretty quickly (it's
> rare but it happens sometimes when the binary compatibility is broken in
> a core library like maven-shared-utils).
>
>
> > 3) (Joker) What is the overall CO2 footprint of distros like
> > these? I believe you did not use Apple M1 for this work... :)
>
> Probably a tiny fraction of what bitcoin mining, Travis CI and
> Youtube/Netflix 4K videos generate ;)
>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Talk: Bootstrapping the Java Ecosystem

Tamás Cservenák
Sorry for being chatty :) but was curious, and looked at sources of "GNU
make".

And as I see, "GNU make" NEEDS "GNU make" to have it built :)
So, I guess this is a usual "thing" in the build tools realm...

Hence, same loops Julien had shown in his video stands for GNU make for
example, right?
(which we all know is not quite true: the loop is actually "project verX"
-uses-> "project verX-1" or alike)

And yes, with Maven (or, pretty much with JVM languages) you entered the
"Maven Artifact Universe", where you meet the same, if not deeper
dependency hierarchy as with deb/rpm or whatever other package system has
as well.

(Another joke) By the way, what do you do with npm packages? ;)

Thanks
Tamas


T

On Fri, Nov 20, 2020 at 10:13 AM Tamás Cservenák <[hidden email]>
wrote:

> Maybe it was not clear from my last statement: so you "hop on bandwagon",
> and you "joined" the branch of Maven releases, you have 3.6.3. Same version
> Maven devs use as well. So, you will be able to use it to rebuild 3.6.4,
> then use 3.6.4 to rebuild 3.6.5, and so on...
>
> Hence, no need to redo all this for EVERY maven version, right?
>
> T
>
> On Fri, Nov 20, 2020 at 10:06 AM Tamás Cservenák <[hidden email]>
> wrote:
>
>> Thanks for the answers!
>>
>> AFAIK, we in Apache as well "vote for source", while we provide binaries
>> as well.
>>
>> Given the video mentions that Maven `-sources` artifacts are NOT
>> buildable (which is true, they are mainly used by IDEs to display library
>> sources while debug for example), am unsure -- at least for ASF artifacts
>> -- why not using then source release bundles instead? For example:
>> https://dist.apache.org/repos/dist/release/maven/maven-3/3.6.3/source/
>>
>> Also, according to your explanation, the problem is now solved once for
>> all, right? You do have (those distros you mention, like Guix) Maven 3.6.3
>> built now, so you do not have to repeat this anymore?
>>
>> My point is that Maven devs also use Maven 3.6.3 currently, and that
>> version will be used to build any future Maven release as well (ie. 3.6.4
>> or 4.0.0 and so on). So, you just had to "hop on" the bandwagon, do this
>> "dance" once, but from now on, all this work can be scraped, right?
>>
>> Thanks
>> Tamas
>>
>> On Fri, Nov 20, 2020 at 9:49 AM Emmanuel Bourg <[hidden email]> wrote:
>>
>>> On 19/11/2020 09:51, Tamás Cservenák wrote:
>>>
>>> > Without starting any flame wars, am really curious: why are you
>>> > repackaging Maven?
>>> >
>>> > I'd understand for OS/distro native packages, but
>>> > why do you rebuild JVM bytecode as well?
>>> >
>>> > Again, am not to start any flame war, am just curious!
>>>
>>> Short answer: why not? This is an Open Source project, not an Open
>>> Binary project. Anyone should be able to rebuild the code, and in an
>>> ideal world where every project is reproducible, get byte identical
>>> binaries.
>>>
>>> Long answer: Debian, Fedora, and I assume Guix are "closed" ecosystems
>>> where you can rebuild every component from sources without needing tools
>>> or libraries outside of the distribution. If you were alone on a desert
>>> island with just a laptop, the sources and no internet connection, you
>>> would be able to rebuild any part of the distribution from scratch.
>>>
>>> This really goes to the roots of the open source philosophy, open source
>>> projects are meant to be built from sources, and if it's not possible
>>> then there is a problem somewhere. Assuming every project becomes
>>> reproducible at some point (see https://reproducible-builds.org for why
>>> it matters) the question of knowing who produced the binaries become
>>> irrelevant, because everyone get the exact same binaries.
>>>
>>>
>>> > 3) What are you really building? As in video, it is said
>>> > several times that you "mutilate" some package to build
>>> > it, then use it to "bootstrap" some other package, and finally
>>> > you rebuild the target package. Given in the process there
>>> > was once a "mutilated" tool, how are you certain, that output
>>> > of the build is correct (I have no doubts about
>>> > reproducibility)? How do you prove that output is what
>>> > it is thought/assumed to be?
>>>
>>> In Debian the Maven package we rebuild from sources is itself used to
>>> build all the other Maven based projects packaged in Debian (that's over
>>> 600 projects currently), so regressions are caught pretty quickly (it's
>>> rare but it happens sometimes when the binary compatibility is broken in
>>> a core library like maven-shared-utils).
>>>
>>>
>>> > 3) (Joker) What is the overall CO2 footprint of distros like
>>> > these? I believe you did not use Apple M1 for this work... :)
>>>
>>> Probably a tiny fraction of what bitcoin mining, Travis CI and
>>> Youtube/Netflix 4K videos generate ;)
>>>
>>> Emmanuel Bourg
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: Talk: Bootstrapping the Java Ecosystem

Emmanuel Bourg
In reply to this post by Tamás Cservenák
Le 20/11/2020 à 10:06, Tamás Cservenák a écrit :

> Also, according to your explanation, the problem is now solved once for
> all, right? You do have (those distros you mention, like Guix) Maven 3.6.3
> built now, so you do not have to repeat this anymore?

That's correct, in Debian this was done only once, future versions of
Maven will be built with the version 3.6.3 currently packaged.

The bootstrapping is really useful to get an initial package in a
distribution, or to provide an irrefutable proof that a set of sources
generate a given binary when you care about reproducibility.

> (Another joke) By the way, what do you do with npm packages?

I leave this mess to the JS folks :)

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Talk: Bootstrapping the Java Ecosystem

Björn Höfling
In reply to this post by Tamás Cservenák
On Fri, 20 Nov 2020 10:06:28 +0100
Tamás Cservenák <[hidden email]> wrote:

> Thanks for the answers!
>
> AFAIK, we in Apache as well "vote for source", while we provide
> binaries as well.
>
> Given the video mentions that Maven `-sources` artifacts are NOT
> buildable (which is true, they are mainly used by IDEs to display
> library sources while debug for example), am unsure -- at least for
> ASF artifacts -- why not using then source release bundles instead?
> For example:
> https://dist.apache.org/repos/dist/release/maven/maven-3/3.6.3/source/
Guix is trying to use these original source code repositories. But
there is no direct connection between the pom.xml and the original
source code. The problem and concern here is that Maven central is a
long-term store for the pom.xml, the binary .jar and the IDE-only
sources.jar. That works very well for rebuilding from binary
dependencies: If my company/project set up an Artifactory cache ten
years ago, I'm still able to rebuild my Maven-based software today by
using the then-downloaded binary dependencies.

But the source code repositories are eliminated quicker. For example,
I'm searching the sources for org.marlin:marlin:0.7.5-Unsafe, a
dependency of the less than 3 years old GeoServer 2.12.2. The source
code is buried with Boundless' death. If I would have at least a hash
sum in the pom.xml at Maven central, I might be able to download it
from softwareheritage.org.

> Also, according to your explanation, the problem is now solved once
> for all, right? You do have (those distros you mention, like Guix)
> Maven 3.6.3 built now, so you do not have to repeat this anymore?
>
> My point is that Maven devs also use Maven 3.6.3 currently, and that
> version will be used to build any future Maven release as well (ie.
> 3.6.4 or 4.0.0 and so on). So, you just had to "hop on" the
> bandwagon, do this "dance" once, but from now on, all this work can
> be scraped, right?

Yes. It is done "once forever". For new versions it might be
necessary to add some additional dependencies or update to newer
versions. This is not always trivial. From a practical point, it would
be nice if you could build Maven 4.0, 4.1, 4.2, etc all from 3.6.3.
Contrary the JDK has a linear dependency chain:

1.9->10->11->12->...

If you now update for example the make-package, you have to rebuild the
full chain.

Concerning make: There is also a bootstrapping-chain starting with
gnu-make-mesboot0 which is built only with tcc, a small C-compiler, which ...
... and at the end, there is some small binary bootstrap kernel, but
this is getting smaller and smaller.

Björn

attachment0 (201 bytes) Download Attachment