Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Paul Hammant
Article updated to eliminate misunderstandings and talk about a different
index for 'maven central' too.

- ph

On Sat, May 13, 2017 at 3:04 PM, Paul Hammant <[hidden email]> wrote:

> I was discussing this of the list today, and it may interest people on dev@
>
> https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/
>
>      "Maven Central as multiple Git repositories"
>
> Enjoy,
>
> - Paul
>
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Stian Soiland-Reyes-2
Interesting idea! All for experimenting with git!

You did not take into account the protocol differences, fetching a single
8MB  jar from Maven Central is easy, while fetching 6 MB via multiple git
HTTP resources (even assuming git packs there will be multiple http calls)
is probably more expensive, particularly as a typical project may have 50
dependencies.

Would each version be a new git repo? The binary class files are not really
suitable for "diffing" and would pretty sure be changed every release (even
if the SRC is the exact same).

It is possible in JAR to use jar200 packing, which can significantly reduce
the compressed size. Perhsps that is relevant here. See
http://docs.oracle.com/javase/7/docs/technotes/tools/share/pack200.html

Are the pom files within these git repositories or can they be prefetched
separately like today to speed up transitive dependency loading?



On 13 May 2017 8:04 pm, "Paul Hammant" <[hidden email]> wrote:

> I was discussing this of the list today, and it may interest people on dev@
>
> https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/
>
>      "Maven Central as multiple Git repositories"
>
> Enjoy,
>
> - Paul
>
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Paul Hammant
There's one repo per thing that is versioned. All these are separate repos

    g: com/thoughtworks/paranamer a: paranamer type: javadoc  is one repo
    g: com/thoughtworks/paranamer a: paranamer type: sources  is one repo
    g: com/thoughtworks/paranamer a: paranamer type: classes  is one repo
    g: com/thoughtworks/xstream a: xstream type: javadoc  is one repo
    g: com/thoughtworks/xstream a: xstream type: sources  is one repo
    g: com/thoughtworks/xstream a: xstream type: classes  is one repo

Or you could do it in less repos, I'm sure, and Git will still make a
compression saving.

And despite what you said, Git manages to find some commonality in *binary
classes *to make a saving. XStream's regular JARs:

Total size for *27* original versions 8.4MB
The .git folder afterwards 2.4MB
Raw/bare storage space saving 71.4%
Your point about the git:// protocol being being more chatty that http://
is true. That's probably more CPU on the server side. It is definitely more
time. See here ...

Git takes 1s to do:

git clone https://github.com/paul-hammant/mc-xs-classes --depth 1 --branch
1.4.3 --bare


Wget takes 0.5s to do:

wget http://central.maven.org/maven2/com/thoughtworks/
xstream/xstream/1.4.3/xstream-1.4.3.jar


On pom files - they are within the base Git repos, yes.  You can't get them
from the server to the client in one Git operation, without bringing at
least a whole tag. Maybe there could be a tag just for the POM file to make
it addressable remotely, and checkoutable locally. I'll test that later
today.

- Paul


On Mon, May 15, 2017 at 4:14 AM, Stian Soiland-Reyes <[hidden email]>
wrote:



> Interesting idea! All for experimenting with git!
>
> You did not take into account the protocol differences, fetching a single
> 8MB  jar from Maven Central is easy, while fetching 6 MB via multiple git
> HTTP resources (even assuming git packs there will be multiple http calls)
> is probably more expensive, particularly as a typical project may have 50
> dependencies.
>
> Would each version be a new git repo? The binary class files are not really
> suitable for "diffing" and would pretty sure be changed every release (even
> if the SRC is the exact same).
>
> It is possible in JAR to use jar200 packing, which can significantly reduce
> the compressed size. Perhsps that is relevant here. See
> http://docs.oracle.com/javase/7/docs/technotes/tools/share/pack200.html
>
> Are the pom files within these git repositories or can they be prefetched
> separately like today to speed up transitive dependency loading?
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Hervé BOUTEMY
this idea of putting everything in git is funny: not sure this will go very
far from this poc, but let's imagine...

on classes branch, splitting the jar into individual .class has IMHO a big
drawback: we loose original jar and its signature

On the other branches, the current poc shows commits for versions that are
perfectly linear: if there are multiple branches that are released in
parallel, the commit won't be so clean. I don't know if this will have an
impact on compression efficiency.

Another issue with this idea: during development, with SNAPSHOTs, the git repo
will be polluted: this idea IMHO could only be valid for releases

not to speak about read concurrency when one requires to use multiple versions
of a lib. And of course, write concurrency is even harder.


Definitely, the idea is funny, but I don't see how this could go very far than
this funny idea (in addition to the complexity for implementing this format in
tooling)

Regards,

Hervé

Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :

> One more repo:
>
> https://github.com/paul-hammant/mc-xs-all/
>
> One branch for each of classes, javadoc, sources, and poms
>
> 15 javadoc original versions: 24.1M
>
> 16 sources original versions: 4.9M
>
> 27 classes original versions: 8.4M
>
> Afterwards git work the bare .git folder is: 8.4M
>
> *77.5% saving on storage*
>
> Any artifact, *including the poms,* can be pulled down via a single git
> command
>
> git clone https://github.com/paul-hammant/mc-xs-classes --depth 1 --branch
> TAGNAME
>
> 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2, classes-1.1,
> classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2, classes-1.2.1,
> classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4, classes-1.4.1,
> classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5, classes-1.4.6,
> classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2, javadoc-1.2.1,
> javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4, javadoc-1.4.1,
> javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5, javadoc-1.4.6,
> javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1, pom-1.2.2,
> pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3, pom-1.4.4,
> pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9, sources-1.1.3,
> sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3, sources-1.3.1,
> sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3, sources-1.4.4,
> sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8, sources-1.4.9
>
>  - Paul



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Brian Fox-2
Even more than redefining what Central does, you're effectively describing
a new, unofficial java class packaging and distribution mechanism. This
seems like it will violate signatures etc and make tracking of what you
actually have a nightmare.

On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <[hidden email]>
wrote:

> this idea of putting everything in git is funny: not sure this will go very
> far from this poc, but let's imagine...
>
> on classes branch, splitting the jar into individual .class has IMHO a big
> drawback: we loose original jar and its signature
>
> On the other branches, the current poc shows commits for versions that are
> perfectly linear: if there are multiple branches that are released in
> parallel, the commit won't be so clean. I don't know if this will have an
> impact on compression efficiency.
>
> Another issue with this idea: during development, with SNAPSHOTs, the git
> repo
> will be polluted: this idea IMHO could only be valid for releases
>
> not to speak about read concurrency when one requires to use multiple
> versions
> of a lib. And of course, write concurrency is even harder.
>
>
> Definitely, the idea is funny, but I don't see how this could go very far
> than
> this funny idea (in addition to the complexity for implementing this
> format in
> tooling)
>
> Regards,
>
> Hervé
>
> Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > One more repo:
> >
> > https://github.com/paul-hammant/mc-xs-all/
> >
> > One branch for each of classes, javadoc, sources, and poms
> >
> > 15 javadoc original versions: 24.1M
> >
> > 16 sources original versions: 4.9M
> >
> > 27 classes original versions: 8.4M
> >
> > Afterwards git work the bare .git folder is: 8.4M
> >
> > *77.5% saving on storage*
> >
> > Any artifact, *including the poms,* can be pulled down via a single git
> > command
> >
> > git clone https://github.com/paul-hammant/mc-xs-classes --depth 1
> --branch
> > TAGNAME
> >
> > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2, classes-1.1,
> > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2, classes-1.2.1,
> > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4, classes-1.4.1,
> > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> classes-1.4.6,
> > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2, javadoc-1.2.1,
> > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4, javadoc-1.4.1,
> > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> javadoc-1.4.6,
> > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> pom-1.2.2,
> > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3, pom-1.4.4,
> > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9, sources-1.1.3,
> > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3, sources-1.3.1,
> > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3, sources-1.4.4,
> > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8, sources-1.4.9
> >
> >  - Paul
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Aldrin Leal
Still, once github gets an outage, our repositories are basically
'left-padded' (taken offline)

--
-- Aldrin Leal, <[hidden email]> / http://about.me/aldrinleal

On Wed, May 17, 2017 at 1:35 PM, Paul Hammant <[hidden email]> wrote:

> Aldrin - https://github.com/paul-hammant/mc-xs-all - no large files added
> to Git. Git makes 70% saving on bytes used ('bare' mode).
>
> On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal <[hidden email]> wrote:
>
> > Just a friendly reminder that git is not optimized for large files (for
> > this, they made git-lfs). Plus, when you do checkout a git repo, there's
> no
> > binary diffs - so if you've got plenty of releases, you'll be wasting a
> lot
> > of space/time in terms of transmission and storage.
> >
> > --
> > -- Aldrin Leal, <[hidden email]> / http://about.me/aldrinleal
> >
> > On Wed, May 17, 2017 at 9:37 AM, Paul Hammant <[hidden email]> wrote:
> >
> > > We can agree to differ on what I'm suggesting and what the impact of
> that
> > > would be :)
> > >
> > > On Wed, May 17, 2017 at 8:59 AM, Brian Fox <[hidden email]> wrote:
> > >
> > > > Even more than redefining what Central does, you're effectively
> > > describing
> > > > a new, unofficial java class packaging and distribution mechanism.
> This
> > > > seems like it will violate signatures etc and make tracking of what
> you
> > > > actually have a nightmare.
> > > >
> > > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > this idea of putting everything in git is funny: not sure this will
> > go
> > > > very
> > > > > far from this poc, but let's imagine...
> > > > >
> > > > > on classes branch, splitting the jar into individual .class has
> IMHO
> > a
> > > > big
> > > > > drawback: we loose original jar and its signature
> > > > >
> > > > > On the other branches, the current poc shows commits for versions
> > that
> > > > are
> > > > > perfectly linear: if there are multiple branches that are released
> in
> > > > > parallel, the commit won't be so clean. I don't know if this will
> > have
> > > an
> > > > > impact on compression efficiency.
> > > > >
> > > > > Another issue with this idea: during development, with SNAPSHOTs,
> the
> > > git
> > > > > repo
> > > > > will be polluted: this idea IMHO could only be valid for releases
> > > > >
> > > > > not to speak about read concurrency when one requires to use
> multiple
> > > > > versions
> > > > > of a lib. And of course, write concurrency is even harder.
> > > > >
> > > > >
> > > > > Definitely, the idea is funny, but I don't see how this could go
> very
> > > far
> > > > > than
> > > > > this funny idea (in addition to the complexity for implementing
> this
> > > > > format in
> > > > > tooling)
> > > > >
> > > > > Regards,
> > > > >
> > > > > Hervé
> > > > >
> > > > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > > > > One more repo:
> > > > > >
> > > > > > https://github.com/paul-hammant/mc-xs-all/
> > > > > >
> > > > > > One branch for each of classes, javadoc, sources, and poms
> > > > > >
> > > > > > 15 javadoc original versions: 24.1M
> > > > > >
> > > > > > 16 sources original versions: 4.9M
> > > > > >
> > > > > > 27 classes original versions: 8.4M
> > > > > >
> > > > > > Afterwards git work the bare .git folder is: 8.4M
> > > > > >
> > > > > > *77.5% saving on storage*
> > > > > >
> > > > > > Any artifact, *including the poms,* can be pulled down via a
> single
> > > git
> > > > > > command
> > > > > >
> > > > > > git clone https://github.com/paul-hammant/mc-xs-classes --depth
> 1
> > > > > --branch
> > > > > > TAGNAME
> > > > > >
> > > > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > > > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
> > classes-1.1,
> > > > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> > > > classes-1.2.1,
> > > > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
> > > classes-1.4.1,
> > > > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > > > > classes-1.4.6,
> > > > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> > > > javadoc-1.2.1,
> > > > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
> > > javadoc-1.4.1,
> > > > > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> > > > > javadoc-1.4.6,
> > > > > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> > > > > pom-1.2.2,
> > > > > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
> > > > pom-1.4.4,
> > > > > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9,
> > sources-1.1.3,
> > > > > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3,
> > > sources-1.3.1,
> > > > > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
> > > > sources-1.4.4,
> > > > > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
> > > > sources-1.4.9
> > > > > >
> > > > > >  - Paul
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------------------------------------
> > ---------
> > > > > To unsubscribe, e-mail: [hidden email]
> > > > > For additional commands, e-mail: [hidden email]
> > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Robert Scholte-6
In reply to this post by Brian Fox-2
On Wed, 17 May 2017 20:41:02 +0200, Paul Hammant <[hidden email]> wrote:

> I would agree that it has the potential to be a new repository
> implementation, Robert.
> But I am not sure I follow your second sentence.

So suppose I want to add xstream-1.4.9 as dependency to my project. How  
should Maven know it has to go to  
https://github.com/paul-hammant/mc-xs-classes?
You need some kind of mapping, and with this structure you have to do it  
for every git-stored dependency.
The most straight-forward solution would be to add repositories (which  
might fit in POM model 4.0.0), but I cannot imagine users want to do that  
for every dependency.

Robert


> Maybe I do. There is one
> Git repo per group/artifact. That's true for whether it is the principal
> artifact you're after or a transitive dep.
>
> 1. For https://github.com/paul-hammant/mc-xs-all I have the sources,
> classes, and javadoc as separate branches in one repo.
>
> 2. For https://github.com/paul-hammant/mc-xs-classes and
> https://github.com/paul-hammant/mc-xs-javadocs and
> https://github.com/paul-hammant/mc-xs-sources I have three Git repos per
> group/artifact.
>
> #1 and #2 are alternate coices. #1 has poms in their own branch in that  
> Git
> repo too - and they too are one-line retrievable from the remote.
>
> - Paul
>
> On Wed, May 17, 2017 at 1:27 PM, Robert Scholte <[hidden email]>
> wrote:
>
>> I consider this as a new repository implementation. But this also  
>> implies
>> that in your pom, for every dependency you have to add a  
>> repository-entry
>> as well, right?
>>
>> Robert
>>
>>
>> On Wed, 17 May 2017 17:10:49 +0200, Aldrin Leal <[hidden email]>
>> wrote:
>>
>> Just a friendly reminder that git is not optimized for large files (for
>>> this, they made git-lfs). Plus, when you do checkout a git repo,  
>>> there's
>>> no
>>> binary diffs - so if you've got plenty of releases, you'll be wasting a
>>> lot
>>> of space/time in terms of transmission and storage.
>>>
>>> --
>>> -- Aldrin Leal, <[hidden email]> / http://about.me/aldrinleal
>>>
>>> On Wed, May 17, 2017 at 9:37 AM, Paul Hammant <[hidden email]> wrote:
>>>
>>> We can agree to differ on what I'm suggesting and what the impact of  
>>> that
>>>> would be :)
>>>>
>>>> On Wed, May 17, 2017 at 8:59 AM, Brian Fox <[hidden email]> wrote:
>>>>
>>>> > Even more than redefining what Central does, you're effectively
>>>> describing
>>>> > a new, unofficial java class packaging and distribution mechanism.  
>>>> This
>>>> > seems like it will violate signatures etc and make tracking of what  
>>>> you
>>>> > actually have a nightmare.
>>>> >
>>>> > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY  
>>>> <[hidden email]>
>>>> > wrote:
>>>> >
>>>> > > this idea of putting everything in git is funny: not sure this  
>>>> will
>>>> go
>>>> > very
>>>> > > far from this poc, but let's imagine...
>>>> > >
>>>> > > on classes branch, splitting the jar into individual .class has  
>>>> IMHO
>>>> a
>>>> > big
>>>> > > drawback: we loose original jar and its signature
>>>> > >
>>>> > > On the other branches, the current poc shows commits for versions
>>>> that
>>>> > are
>>>> > > perfectly linear: if there are multiple branches that are  
>>>> released in
>>>> > > parallel, the commit won't be so clean. I don't know if this will
>>>> have
>>>> an
>>>> > > impact on compression efficiency.
>>>> > >
>>>> > > Another issue with this idea: during development, with SNAPSHOTs,  
>>>> the
>>>> git
>>>> > > repo
>>>> > > will be polluted: this idea IMHO could only be valid for releases
>>>> > >
>>>> > > not to speak about read concurrency when one requires to use  
>>>> multiple
>>>> > > versions
>>>> > > of a lib. And of course, write concurrency is even harder.
>>>> > >
>>>> > >
>>>> > > Definitely, the idea is funny, but I don't see how this could go  
>>>> very
>>>> far
>>>> > > than
>>>> > > this funny idea (in addition to the complexity for implementing  
>>>> this
>>>> > > format in
>>>> > > tooling)
>>>> > >
>>>> > > Regards,
>>>> > >
>>>> > > Hervé
>>>> > >
>>>> > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
>>>> > > > One more repo:
>>>> > > >
>>>> > > > https://github.com/paul-hammant/mc-xs-all/
>>>> > > >
>>>> > > > One branch for each of classes, javadoc, sources, and poms
>>>> > > >
>>>> > > > 15 javadoc original versions: 24.1M
>>>> > > >
>>>> > > > 16 sources original versions: 4.9M
>>>> > > >
>>>> > > > 27 classes original versions: 8.4M
>>>> > > >
>>>> > > > Afterwards git work the bare .git folder is: 8.4M
>>>> > > >
>>>> > > > *77.5% saving on storage*
>>>> > > >
>>>> > > > Any artifact, *including the poms,* can be pulled down via a  
>>>> single
>>>> git
>>>> > > > command
>>>> > > >
>>>> > > > git clone https://github.com/paul-hammant/mc-xs-classes --depth  
>>>> 1
>>>> > > --branch
>>>> > > > TAGNAME
>>>> > > >
>>>> > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
>>>> > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
>>>> classes-1.1,
>>>> > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
>>>> > classes-1.2.1,
>>>> > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
>>>> classes-1.4.1,
>>>> > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
>>>> > > classes-1.4.6,
>>>> > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
>>>> > javadoc-1.2.1,
>>>> > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
>>>> javadoc-1.4.1,
>>>> > > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
>>>> > > javadoc-1.4.6,
>>>> > > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
>>>> > > pom-1.2.2,
>>>> > > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
>>>> > pom-1.4.4,
>>>> > > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9,
>>>> sources-1.1.3,
>>>> > > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3,
>>>> sources-1.3.1,
>>>> > > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
>>>> > sources-1.4.4,
>>>> > > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
>>>> > sources-1.4.9
>>>> > > >
>>>> > > >  - Paul
>>>> > >
>>>> > >
>>>> > >
>>>> > > ------------------------------------------------------------
>>>> ---------
>>>> > > To unsubscribe, e-mail: [hidden email]
>>>> > > For additional commands, e-mail: [hidden email]
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Paul Hammant
Robert,

From the blog entry:


*Actually changing Maven Central to do this*
The maven ‘deploy’ workflow and plugin would invisibly do a commit to (or
create of) a dedicated Git repo up on central.

For XStream, a new deployment would not go into
http://central.maven.org/maven2/com/thoughtworks/xstream/xstream/ any more.
Instead they would go into (say) [hidden email]:
maven2/com/thoughtworks/xstream/xstream.git

The maintainer for the group:artifact would not have to do anything
different to exist in the new deploy world, all the heavy lifting is done
in Maven Central and that future version of the deploy plugin. They would
have to upgrade their project to that deploy plugin, of course.

It is not just Maven Central. It would be Artifactory, Nexus, Gradle (etc)
technologies too.


I just put it on Github as that is an easy place to do look-here-see-bro
discussions.

:)

- Paul


On Wed, May 17, 2017 at 3:29 PM, Robert Scholte <[hidden email]>
wrote:
>
> On Wed, 17 May 2017 20:41:02 +0200, Paul Hammant <[hidden email]> wrote:
>
>> I would agree that it has the potential to be a new repository
>> implementation, Robert.
>> But I am not sure I follow your second sentence.
>
>
> So suppose I want to add xstream-1.4.9 as dependency to my project. How
should Maven know it has to go to
https://github.com/paul-hammant/mc-xs-classes?
> You need some kind of mapping, and with this structure you have to do it
for every git-stored dependency.
> The most straight-forward solution would be to add repositories (which
might fit in POM model 4.0.0), but I cannot imagine users want to do that
for every dependency.

>
> Robert
>
>
>
>> Maybe I do. There is one
>> Git repo per group/artifact. That's true for whether it is the principal
>> artifact you're after or a transitive dep.
>>
>> 1. For https://github.com/paul-hammant/mc-xs-all I have the sources,
>> classes, and javadoc as separate branches in one repo.
>>
>> 2. For https://github.com/paul-hammant/mc-xs-classes and
>> https://github.com/paul-hammant/mc-xs-javadocs and
>> https://github.com/paul-hammant/mc-xs-sources I have three Git repos per
>> group/artifact.
>>
>> #1 and #2 are alternate coices. #1 has poms in their own branch in that
Git
>> repo too - and they too are one-line retrievable from the remote.
>>
>> - Paul
>>
>> On Wed, May 17, 2017 at 1:27 PM, Robert Scholte <[hidden email]>
>> wrote:
>>
>>> I consider this as a new repository implementation. But this also
implies
>>> that in your pom, for every dependency you have to add a
repository-entry

>>> as well, right?
>>>
>>> Robert
>>>
>>>
>>> On Wed, 17 May 2017 17:10:49 +0200, Aldrin Leal <[hidden email]>
>>> wrote:
>>>
>>> Just a friendly reminder that git is not optimized for large files (for
>>>>
>>>> this, they made git-lfs). Plus, when you do checkout a git repo,
there's

>>>> no
>>>> binary diffs - so if you've got plenty of releases, you'll be wasting a
>>>> lot
>>>> of space/time in terms of transmission and storage.
>>>>
>>>> --
>>>> -- Aldrin Leal, <[hidden email]> / http://about.me/aldrinleal
>>>>
>>>> On Wed, May 17, 2017 at 9:37 AM, Paul Hammant <[hidden email]> wrote:
>>>>
>>>> We can agree to differ on what I'm suggesting and what the impact of
that
>>>>>
>>>>> would be :)
>>>>>
>>>>> On Wed, May 17, 2017 at 8:59 AM, Brian Fox <[hidden email]> wrote:
>>>>>
>>>>> > Even more than redefining what Central does, you're effectively
>>>>> describing
>>>>> > a new, unofficial java class packaging and distribution mechanism.
This
>>>>> > seems like it will violate signatures etc and make tracking of what
you
>>>>> > actually have a nightmare.
>>>>> >
>>>>> > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
[hidden email]>
>>>>> > wrote:
>>>>> >
>>>>> > > this idea of putting everything in git is funny: not sure this
will
>>>>> go
>>>>> > very
>>>>> > > far from this poc, but let's imagine...
>>>>> > >
>>>>> > > on classes branch, splitting the jar into individual .class has
IMHO
>>>>> a
>>>>> > big
>>>>> > > drawback: we loose original jar and its signature
>>>>> > >
>>>>> > > On the other branches, the current poc shows commits for versions
>>>>> that
>>>>> > are
>>>>> > > perfectly linear: if there are multiple branches that are
released in
>>>>> > > parallel, the commit won't be so clean. I don't know if this will
>>>>> have
>>>>> an
>>>>> > > impact on compression efficiency.
>>>>> > >
>>>>> > > Another issue with this idea: during development, with SNAPSHOTs,
the
>>>>> git
>>>>> > > repo
>>>>> > > will be polluted: this idea IMHO could only be valid for releases
>>>>> > >
>>>>> > > not to speak about read concurrency when one requires to use
multiple
>>>>> > > versions
>>>>> > > of a lib. And of course, write concurrency is even harder.
>>>>> > >
>>>>> > >
>>>>> > > Definitely, the idea is funny, but I don't see how this could go
very
>>>>> far
>>>>> > > than
>>>>> > > this funny idea (in addition to the complexity for implementing
this

>>>>> > > format in
>>>>> > > tooling)
>>>>> > >
>>>>> > > Regards,
>>>>> > >
>>>>> > > Hervé
>>>>> > >
>>>>> > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
>>>>> > > > One more repo:
>>>>> > > >
>>>>> > > > https://github.com/paul-hammant/mc-xs-all/
>>>>> > > >
>>>>> > > > One branch for each of classes, javadoc, sources, and poms
>>>>> > > >
>>>>> > > > 15 javadoc original versions: 24.1M
>>>>> > > >
>>>>> > > > 16 sources original versions: 4.9M
>>>>> > > >
>>>>> > > > 27 classes original versions: 8.4M
>>>>> > > >
>>>>> > > > Afterwards git work the bare .git folder is: 8.4M
>>>>> > > >
>>>>> > > > *77.5% saving on storage*
>>>>> > > >
>>>>> > > > Any artifact, *including the poms,* can be pulled down via a
single
>>>>> git
>>>>> > > > command
>>>>> > > >
>>>>> > > > git clone https://github.com/paul-hammant/mc-xs-classes --depth
1

>>>>> > > --branch
>>>>> > > > TAGNAME
>>>>> > > >
>>>>> > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
>>>>> > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
>>>>> classes-1.1,
>>>>> > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
>>>>> > classes-1.2.1,
>>>>> > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
>>>>> classes-1.4.1,
>>>>> > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
>>>>> > > classes-1.4.6,
>>>>> > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
>>>>> > javadoc-1.2.1,
>>>>> > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
>>>>> javadoc-1.4.1,
>>>>> > > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
>>>>> > > javadoc-1.4.6,
>>>>> > > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
>>>>> > > pom-1.2.2,
>>>>> > > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
>>>>> > pom-1.4.4,
>>>>> > > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9,
>>>>> sources-1.1.3,
>>>>> > > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3,
>>>>> sources-1.3.1,
>>>>> > > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
>>>>> > sources-1.4.4,
>>>>> > > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
>>>>> > sources-1.4.9
>>>>> > > >
>>>>> > > >  - Paul
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > ------------------------------------------------------------
>>>>> ---------
>>>>> > > To unsubscribe, e-mail: [hidden email]
>>>>> > > For additional commands, e-mail: [hidden email]
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Paul Hammant
In reply to this post by Aldrin Leal
Actually I'm proposing a predictable structure on 'central :

 [hidden email]:
maven2/<group-name-with-slashes-for-dots</<artifact-name>.git

(one minor fix versus previous description of the git:// location)

Or for the three separate variant:

 [hidden email]:
maven2/<group-name-with-slashes-for-dots</<artifact-name>-classes.git
 [hidden email]:
maven2/<group-name-with-slashes-for-dots</<artifact-name>-javadocs.git
 [hidden email]:
maven2/<group-name-with-slashes-for-dots</<artifact-name>-sources.git

More likely though (as you mentioned in your opening line) is the way
homebrew works - you point at repos elsewhere, but control poms/shas etc on
'central.




On Wed, May 17, 2017 at 3:59 PM, Manfred Moser <[hidden email]>
wrote:

> Having worked with repository managers and the implementation for various
> formats on Nexus for years I think such a format is a bit like Bower. It is
> a registry format that in turn points to git repositories that have the
> content.
>
> From a corporate usage and implementation point of view this is a utter
> nightmare since you would have to open up your systems to all those
> different repositories and sites hosting them instead of just one.
>
> You also cant simply make a copy of the content or analyze it in the way
> it manifests as binaries.
>
> I am not sure what you are looking for as benefits but from my point of
> view this is maybe a fun experiment but not something that will ever take
> off..
>
> Manfred
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

Hervé BOUTEMY
In reply to this post by Hervé BOUTEMY
Le mercredi 17 mai 2017, 14:46:13 CEST Paul Hammant a écrit :
> > not to speak about read concurrency when one requires to use multiple
> > versions
> > of a lib. And of course, write concurrency is even harder.
>
> I'm not following, dude.
:)

I'm not sure I'm not mistaken for the read part: with the current repo format,
one thing that is super easy is that each artifact version is just a path that
you can use simultaneously in one build or in multiple builds
With the proposed git repos, I don't get how read access to a tag is done
without changing disk state, so you can access mutliple tags at the same time.


This "on read" question happens with "on write": but as you told the idea is
more to consume releases from a central repository, and not on the desktop
with SNAPSHOTs and so on, the write question is not really a question.

Regards,

Hervé

>
> - Paul



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]