Re: Fwd: FOSDEM 19 Debian Java talk

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: FOSDEM 19 Debian Java talk

rfscholte
On Tue, 12 Feb 2019 12:34:56 +0100, Markus Koschany <[hidden email]> wrote:

> Hello,
>
> Dalibor Topic (Oracle) and Robert Scholte (Apache Maven) contacted me
> and were so kind to agree to make this discussion public, so that others
> can chime in too. I would like to use the opportunity to answer the
> initial question "what we are interested in seeing better supported from
> build tools" and give some general feedback about integrating Java into
> Debian.
>
>
> First of all Ant and Maven are most likely the best supported build
> systems at the moment. We carry only two patches for Maven, one because
> we use a newer version of SLF4j [1] and the second one is to make Maven
> builds reproducible. [2] It looks like [1] has been already merged
> upstream but [2] has not been forwarded yet. It would be great of
> course, if Maven builds would be reproducible out-of-the-box. In general
> I would like to see reproducible builds everywhere.

Hi Markus,

first of all thanks for the insights, it is important for us to know how  
Maven is used and in which way we can improve that way-of-work. Hervé is  
already working hard on the reproducible builds specs with your team in  
order to find out how we can improve our maven-plugins to get reproducible  
artifacts.

Maven itself is not 100% reproducible. We've learned that some Linux  
vendors rebuild Maven and the presentation confirmed that Debian is one of  
those vendors. What we've seen in the past is that sometimes people are  
having issues with Maven and after a while we discovered that they were  
not using the official Apache Maven distribution[1]. For us it is quite  
easy to say: sorry, not our official distribution, please contact your  
Linux distributor.
In such case we have 3 losers: the user, the Apache Maven project and the  
Linux Distributor. If only the official Maven distribution was used, then  
we would have had 3 winners.

When you decide to rebuild Maven, you're also taking all related  
responsibilities. I'm also wondering how you build Maven, since Maven is  
being built with Maven. That should be a challenge to also rebuild all  
plugins, etc.
And how do you test this and confirm that it works as the official  
distribution?
Sure, *IF* Maven is 100% reproducible then you can rely on our test-infra,  
but that's not the situation.

So here are my main questions:
- Are you making clear that you're not using the official Maven  
distribution, e.g. by adjust the info from 'mvn --version'?
- What is the optimum way of distributing Maven sources? For example: also  
providing compile and package scripts (instead of calling maven-plugins)?

[1] https://maven.apache.org/download.cgi

>
> Otherwise we have two build tools / Maven plugins called
> maven-debian-helper [3] and maven-repo-helper [4] that do all the Debian
> specific operations. Maven already supports local repositories and
> offline mode but I would like to see native support for unversioned jar
> files and dependencies too. At the moment we create our own local
> repository in /usr/share/maven-repo and in addition to the normal
> version, we have a so called "debian" version. Other Java projects in
> Debian only reference the debian version, so that we have to maintain
> only one library or application and when we decide to upgrade a package,
> reverse-depdencies continue to work because they use the unversioned
> "debian" instead of a specific version. In my experience other languages
> like Perl or Python, which are less version-centric, support this use
> case better.
>
> Regarding javadoc generation we would like to see an option that
> basically reverts to pre OpenJDK8 and simply is less strict than the
> current implementation. We currently use the undocumented and
> unsupported --ignore-source-errors option when we build javadoc. It is
> not feasible for us to fix all those syntax errors ourselves and we will
> rather ditch our documentation packages should --ignore-source-errors go
> away.
>
> Our biggest challenge is Gradle. If Robert wants to help us then he
> should never rewrite parts of Maven in Kotlin or encourage developers to
> use a specific prebuilt version of Maven and to ship a script in every
> project that downloads it from the internet (gradle-wrapper). ;)

Interesting :) We've been discussing how we could get more contributors  
and Kotlin was one idea, but that one didn't make it.
Even though we as Maven developers don't like the wrapper, the community  
is asking for it, so we're seriously considering to add it as part of  
Maven Core.

regards,
Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

AW: Fwd: FOSDEM 19 Debian Java talk

Bernd Eckenfels
Hello,

according to the Apache Release Policy a release is the source and while it allows and defines convinience binaries there is not really a Notion of „official binaries“ from the ASF Point of view. So Maybe the new property should be something like „binary Vendor“ or „packager“ (similiar what many package Managers have?)

I think the number of additional support Problems because of distribution specific packages and the number of solved Problems by distributions doing a good Integration Job can not be clearly tallied.

And therefore I would stay away from language like „modified“, „official“ to avoid those Groups to feel unwelcome (especially in the ASF spirit of open SOURCE).

Gruss
Bernd
--
http://bernd.eckenfels.net

Von: Michael Osipov
Gesendet: Mittwoch, 13. Februar 2019 19:36
An: Maven Developers List; Robert Scholte; Dalibor Topic; Markus Koschany
Cc: [hidden email]; Matthias Klose
Betreff: Re: Fwd: FOSDEM 19 Debian Java talk

Am 2019-02-12 um 20:09 schrieb Robert Scholte:

> On Tue, 12 Feb 2019 12:34:56 +0100, Markus Koschany <[hidden email]> wrote:
>
>> Hello,
>>
>> Dalibor Topic (Oracle) and Robert Scholte (Apache Maven) contacted me
>> and were so kind to agree to make this discussion public, so that others
>> can chime in too. I would like to use the opportunity to answer the
>> initial question "what we are interested in seeing better supported from
>> build tools" and give some general feedback about integrating Java into
>> Debian.
>>
>>
>> First of all Ant and Maven are most likely the best supported build
>> systems at the moment. We carry only two patches for Maven, one because
>> we use a newer version of SLF4j [1] and the second one is to make Maven
>> builds reproducible. [2] It looks like [1] has been already merged
>> upstream but [2] has not been forwarded yet. It would be great of
>> course, if Maven builds would be reproducible out-of-the-box. In general
>> I would like to see reproducible builds everywhere.
>
> Hi Markus,
>
> first of all thanks for the insights, it is important for us to know how
> Maven is used and in which way we can improve that way-of-work. Hervé is
> already working hard on the reproducible builds specs with your team in
> order to find out how we can improve our maven-plugins to get
> reproducible artifacts.
>
> Maven itself is not 100% reproducible. We've learned that some Linux
> vendors rebuild Maven and the presentation confirmed that Debian is one
> of those vendors. What we've seen in the past is that sometimes people
> are having issues with Maven and after a while we discovered that they
> were not using the official Apache Maven distribution[1]. For us it is
> quite easy to say: sorry, not our official distribution, please contact
> your Linux distributor.
> In such case we have 3 losers: the user, the Apache Maven project and
> the Linux Distributor. If only the official Maven distribution was used,
> then we would have had 3 winners.
>
> When you decide to rebuild Maven, you're also taking all related
> responsibilities. I'm also wondering how you build Maven, since Maven is
> being built with Maven. That should be a challenge to also rebuild all
> plugins, etc.
> And how do you test this and confirm that it works as the official
> distribution?
> Sure, *IF* Maven is 100% reproducible then you can rely on our
> test-infra, but that's not the situation.
>
> So here are my main questions:
> - Are you making clear that you're not using the official Maven
> distribution, e.g. by adjust the info from 'mvn --version'?

I expressed my proposal to Hervé that we need a new property:
maven.vendor. Our official distribution will carry the value: ASF.
Everyone else who modifies the content must change the value in the
build.properties. Thus, we will quickly know that this distro has been
modified by someone else.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Re: Fwd: FOSDEM 19 Debian Java talk

Michael Osipov-3
In reply to this post by rfscholte
I have proposed a similar approach with maven.vendor in the build.properties.

> Gesendet: Donnerstag, 14. Februar 2019 um 14:24 Uhr
> Von: "Gary Gregory" <[hidden email]>
> An: "Maven Developers List" <[hidden email]>
> Cc: "Robert Scholte" <[hidden email]>, "Dalibor Topic" <[hidden email]>, [hidden email], "Matthias Klose" <[hidden email]>
> Betreff: Re: Fwd: FOSDEM 19 Debian Java talk
>
> Jar manifest files carry data like "built-by" and implementation
> information:
>
> https://docs.oracle.com/javase/tutorial/deployment/jar/packageman.html
>
> Why not reuse "Implementation-Vendor" or invent a new entry and put
> "Debian" in there. Maven can display this additional information on "mvn
> -version".
>
> Gary
>
> On Thu, Feb 14, 2019 at 4:56 AM Markus Koschany <[hidden email]> wrote:
>
> > Hi Robert,
> >
> > Am 12.02.19 um 20:09 schrieb Robert Scholte:
> > [...]
> > > Hi Markus,
> > >
> > > first of all thanks for the insights, it is important for us to know how
> > > Maven is used and in which way we can improve that way-of-work. Hervé is
> > > already working hard on the reproducible builds specs with your team in
> > > order to find out how we can improve our maven-plugins to get
> > > reproducible artifacts.
> > >
> > > Maven itself is not 100% reproducible. We've learned that some Linux
> > > vendors rebuild Maven and the presentation confirmed that Debian is one
> > > of those vendors. What we've seen in the past is that sometimes people
> > > are having issues with Maven and after a while we discovered that they
> > > were not using the official Apache Maven distribution[1]. For us it is
> > > quite easy to say: sorry, not our official distribution, please contact
> > > your Linux distributor.
> > > In such case we have 3 losers: the user, the Apache Maven project and
> > > the Linux Distributor. If only the official Maven distribution was used,
> > > then we would have had 3 winners.
> > >
> > > When you decide to rebuild Maven, you're also taking all related
> > > responsibilities.
> >
> > We hear this a lot and it seems to be more common in the Java community.
> > From Debian's point of view (other distributions like Fedora share the
> > same view) it is essential being able to rebuild software from source.
> > The prerequisite is the availability of source code of course. Most of
> > us find it even strange when upstream developers recommend to use their
> > prebuilt versions. Do they have something to hide? Why can't they just
> > make building from source easy and trivial?
> >
> > We believe that every user should have the ability to modify software
> > but this is only possible if she can build it from source. We go to
> > great lengths to ensure that all software complies with Debian's free
> > software guidelines. For the enduser the language and build tools of a
> > certain piece of software almost become meaningless. They just type "apt
> > source maven", change into the maven directory and rebuild the software
> > with another one-liner.
> >
> > In case of Maven I don't see where our release differs fundamentally
> > from your binary releases. As I said there is only one reproducibility
> > patch left that doesn't change the functionality at all. So what we do
> > is grab your sources from https://github.com/apache/maven/releases or
> > maven.apache.org. In our opinion, without making any significant
> > changes, it should just behave as your binary release when we build it
> > from source. Otherwise there is source code missing or different.
> >
> > > I'm also wondering how you build Maven, since Maven is
> > > being built with Maven. That should be a challenge to also rebuild all
> > > plugins, etc.
> > > And how do you test this and confirm that it works as the official
> > > distribution?
> > > Sure, *IF* Maven is 100% reproducible then you can rely on our
> > > test-infra, but that's not the situation.
> >
> > It is a challenge to build Maven from source. We solved the
> > bootstrapping problem and now we can use Debian's Maven version to build
> > newer versions but we have to follow a certain build order when we make
> > an update.
> >
> > > So here are my main questions:
> > > - Are you making clear that you're not using the official Maven
> > > distribution, e.g. by adjust the info from 'mvn --version'?
> >
> > This is how it looks on Debian 9 "Stretch" at the moment.
> >
> > Apache Maven 3.3.9
> > Maven home: /usr/share/maven
> > Java version: 1.8.0_181, vendor: Oracle Corporation
> > Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
> > Default locale: de_DE, platform encoding: UTF-8
> > OS name: "linux", version: "4.9.0-8-amd64", arch: "amd64", family: "unix"
> >
> > It is obvious from Maven home I guess but there is no special emphasis
> > on Debian because when you install Maven from Debian you will never get
> > a prebuilt binary release, this is implicit for all software in Debian's
> > main distribution.
> >
> > > - What is the optimum way of distributing Maven sources? For example:
> > > also providing compile and package scripts (instead of calling
> > > maven-plugins)?
> >
> > The major headache for us is the initial bootstrapping of new compilers
> > or build tools. We often write our own Ant scripts to solve the chicken
> > and egg problem. For Maven this is currently solved but if I recall
> > correctly there are sometimes plugins that require a certain Maven
> > version which in turn requires a specific plugin version, a classic
> > dependency loop. So if there was a way to build Maven without Maven or
> > certain plugins that would obviously make our life easier.
> >
> > [...]
> > >> Our biggest challenge is Gradle. If Robert wants to help us then he
> > >> should never rewrite parts of Maven in Kotlin or encourage developers to
> > >> use a specific prebuilt version of Maven and to ship a script in every
> > >> project that downloads it from the internet (gradle-wrapper). ;)
> > >
> > > Interesting :) We've been discussing how we could get more contributors
> > > and Kotlin was one idea, but that one didn't make it.
> > > Even though we as Maven developers don't like the wrapper, the community
> > > is asking for it, so we're seriously considering to add it as part of
> > > Maven Core.
> >
> > Uh, don't give in to those developers. :) This is really a bad habit in
> > the Java community. A build system should be merely a tool to build your
> > software and maybe for simplifying project management but imagine all
> > C/C++/Python and Perl developers would be like that. What a nightmare.
> >
> > Regards,
> >
> > Markus
> >
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Fwd: FOSDEM 19 Debian Java talk

Hervé BOUTEMY
In reply to this post by rfscholte
Le samedi 16 février 2019, 11:38:59 CET Emmanuel Bourg a écrit :

> Le 15/02/2019 à 17:34, Hervé BOUTEMY a écrit :
> > IIUC you don't use the same versions of dependencies (because of
> > uniqueness of each lib in Debian [1]) as those used in a "normal
> > unconstrained" build from source [2]
> >
> > I don't yet understand the details of Debian build system, if the symlinks
> > are used at build time or only at runtime, but even if you the resulting
> > jars of the Debian build from Maven source are the same binaries, the
> > execution of Maven does not run with the same dependencies, then there
> > may be subtle differences during the run (for example from jansi, or
> > guice, or commons-cli or ...)
> I try to stick as much as possible to the same versions of the
> dependencies, especially the sensitive ones like Guice or Aether, but
> there are still some divergences. I've reviewed the dependencies
> installed in the /usr/share/maven/lib directory by the maven/3.6.0
> package in Debian unstable and we currently have the following differences:
>
>                       Maven    Debian
>   commons-lang3       3.8.1    3.8
>   commons-io          2.5      2.6
>   guava               25.1     19.0
>   jansi               1.17     1.17.1
>   jansi-native        1.17     1.18
>   maven-shared-utils  3.2.1    3.3.0
>   plexus-utils        3.1.0    3.1.1
>   wagon               3.2.0    3.3.1
>
> The versions are slightly ahead to the ones in Maven 3.6.0,
ahead does not always mean better :)
but it's good to see a precise measure of the intentionally managed difference
between unconstrained Maven distribution (which will become in the future a
reproducible build) and Debian choice = where subtle differences in behaviour
can happen for users = what should be clearly advertised to users

> except for
> Guava. Guava is only used by Guice for Preconditions, immutable
> collections and other basic operations. I don't think the difference is
> significant. At least, I've verified that the Guice 4.2.1 tests pass
> with Guava 19.0.
>
> > IIUC, that uniqueness of every library is a Debian principle, then a Maven
> > distribution from Debian cannot be equivalent to a "normal" distribution
> > where library versions are purely selected by Maven source's pom.xml,
> > without adding any constraint This is why having Debian's distribution
> > display its name in "mvn -v" would make that more clear.
> >
> > Let's discuss a little bit about plugins: when Debian launches the "mvn"
> > command to build some Debian package, does it use Maven Central like
> > anybody?  = where do the Maven plugins used during build come from?
> There are two different cases:
> 1. What Maven does when launched by a user
> 2. What Maven does when it's used in a Debian package build
>
>
> 1. When the user launches "mvn" to build a project it behaves as usual
> by downloading the dependencies from Maven Central or the other
> repositories defined in the pom. The Maven plugins also available as
> Debian packages are never used. The only code compiled by Debian that is
> executed is the set of dependencies found in the Maven lib directory
> (/usr/share/maven/lib).
>
> The default plugin versions are also exactly identical. Before Maven
> 3.5.0 we patched default-bindings.xml to align them with the versions
> packaged in Debian. We stopped to preserve the same default behavior.
> This is new in Debian 10 "Buster" (and the latest Ubuntu 18.04 LTS), the
> current stable version Debian 9 "Stretch" has Maven 3.3.9 with different
> default plugins.
for users, depending on default plugins versions injected by default biondings
is a bad practice: in Maven 3.7.0, MNG-6562 will provide a warning to help
users avoid that

https://issues.apache.org/jira/browse/MNG-6562

>
>
> 2. Debian builds are offline, completely disconnected from the network.
> It's impossible to download dependencies or plugins from Maven Central.
> That's why we recreate a Maven repository in /usr/share/maven-repo/ and
> instruct Maven to fetch the dependencies there instead of Maven Central
> when building a Debian package. This repository is populated by
> installing the packaged plugins and dependencies.
great, this is the key strategy that I wanted to discover

>
>
> If there are Maven developers interested I can do a live demo showing
> how Maven projects are built into Debian packages.
yes, I'm really interested, because I have so many more detailed questions

if we do that face to face in a Paris bar, I'll pay you a few beers :)
but if many people are interested, having a remote session will be better:
we'll have to wait for the beers...

Regards,

Hervé

>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]