Puzzling Aether behaviour

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Puzzling Aether behaviour

Zoran Regvart
Hi Mavenistas,
I was recently quite puzzled on why Maven is trying to download a
dependency already present in the local repository and I tracked it
down to this comment in EnhancedLocalRepositoryManager[1]:

"artifact downloaded from remote repository is accepted only
downloaded from request repositories"

So regardless of the artifact being present in the local repository an
attempt is made to download it again if the id of the repository
doesn't match the id of the repository stored in
`_remote.repositories` file.

I think that this will prevent anyone trying to reuse a local
repository with artifacts downloaded from unknown/unconfigured
repository in an offline build.

Can someone shed some light on the rationale behind this?

Thanks :)

zoran

[1] https://github.com/eclipse/aether-core/blob/4cf5f7a406b516a45d8bf15e7dfe3fb3849cb87b/aether-impl/src/main/java/org/eclipse/aether/internal/impl/EnhancedLocalRepositoryManager.java#L106-L107

--
Zoran Regvart

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Puzzling Aether behaviour

Robert Scholte-8
There are a couple of things happening here.

1. dependencies are checked even though they are downloaded
This ensures that the build doesn't rely on *your* *local* repository. At  
any time anybody should be able to build the project, it even should be  
possible that you remove your own local repo, so it is important that all  
dependencies are available via a remote repository.

2. dependencies are verified against their original remote repository.
Recently I read an article about an interesting case regarding downloads  
of multiple repositories. (sadly I can't find it anymore).
IIRC what happened is that they noticed a change in behavior of an  
application. After a while they discovered that some third party  
repository was used before Maven Central for downloading dependencies and  
one library was changed in this third party repository. The solution was  
simple: use Maven Central as the primary repository at all time.
This might explain why it is important to not just trust the  
groupId+artifactId+version. The (original) source/repository or additional  
kind of checksum is as important.

thanks,
Robert

On Fri, 21 Dec 2018 13:33:10 +0100, Zoran Regvart <[hidden email]>  
wrote:

> Hi Mavenistas,
> I was recently quite puzzled on why Maven is trying to download a
> dependency already present in the local repository and I tracked it
> down to this comment in EnhancedLocalRepositoryManager[1]:
>
> "artifact downloaded from remote repository is accepted only
> downloaded from request repositories"
>
> So regardless of the artifact being present in the local repository an
> attempt is made to download it again if the id of the repository
> doesn't match the id of the repository stored in
> `_remote.repositories` file.
>
> I think that this will prevent anyone trying to reuse a local
> repository with artifacts downloaded from unknown/unconfigured
> repository in an offline build.
>
> Can someone shed some light on the rationale behind this?
>
> Thanks :)
>
> zoran
>
> [1]  
> https://github.com/eclipse/aether-core/blob/4cf5f7a406b516a45d8bf15e7dfe3fb3849cb87b/aether-impl/src/main/java/org/eclipse/aether/internal/impl/EnhancedLocalRepositoryManager.java#L106-L107

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Puzzling Aether behaviour

JackZi
How many times are the dependencies checked, Robert? Just out of interest.



-----
Hi!
--
Sent from: http://maven.40175.n5.nabble.com/Maven-Users-f40176.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Puzzling Aether behaviour

Zoran Regvart
In reply to this post by Robert Scholte-8
Hi Robert,
thanks for taking the time to reply. Do you think it would make sense
to put the hash value of the artifact in `_remote.repositories`. For
example I see this for camel-core:

camel-core-2.23.0.jar>maven.central=

It can become:

camel-core-2.23.0.jar>maven.central=2f67a52f3c0aea7a8e2d53acde292569aa7b8087

That is equal to camel-core-2.23.0.jar.sha1[1]

Then instead of downloading the full artifact Aether could opt to
calculate the hash of the local artifact check with
`_remote.repositories` or the .sha1 from the remote repository.

That way locally downloaded artifact could be checked for consistency
without re-downloading it. For example I might have:

camel-core-2.23.0.jar>maven.central=2f67a52f3c0aea7a8e2d53acde292569aa7b8087
camel-core-2.23.0.jar>my-repo=2f67a52f3c0aea7a8e2d53acde292569aa7b8087

Where the value for `camel-core-2.23.0.jar>maven.central` would come
from .sha1 file.

Hope my ramblings make some sense :)

Happy holidays!

zoran

[1] http://central.maven.org/maven2/org/apache/camel/camel-core/2.23.0/camel-core-2.23.0.jar.sha1

On Fri, Dec 21, 2018 at 8:23 PM Robert Scholte <[hidden email]> wrote:

>
> There are a couple of things happening here.
>
> 1. dependencies are checked even though they are downloaded
> This ensures that the build doesn't rely on *your* *local* repository. At
> any time anybody should be able to build the project, it even should be
> possible that you remove your own local repo, so it is important that all
> dependencies are available via a remote repository.
>
> 2. dependencies are verified against their original remote repository.
> Recently I read an article about an interesting case regarding downloads
> of multiple repositories. (sadly I can't find it anymore).
> IIRC what happened is that they noticed a change in behavior of an
> application. After a while they discovered that some third party
> repository was used before Maven Central for downloading dependencies and
> one library was changed in this third party repository. The solution was
> simple: use Maven Central as the primary repository at all time.
> This might explain why it is important to not just trust the
> groupId+artifactId+version. The (original) source/repository or additional
> kind of checksum is as important.
>
> thanks,
> Robert
>
> On Fri, 21 Dec 2018 13:33:10 +0100, Zoran Regvart <[hidden email]>
> wrote:
>
> > Hi Mavenistas,
> > I was recently quite puzzled on why Maven is trying to download a
> > dependency already present in the local repository and I tracked it
> > down to this comment in EnhancedLocalRepositoryManager[1]:
> >
> > "artifact downloaded from remote repository is accepted only
> > downloaded from request repositories"
> >
> > So regardless of the artifact being present in the local repository an
> > attempt is made to download it again if the id of the repository
> > doesn't match the id of the repository stored in
> > `_remote.repositories` file.
> >
> > I think that this will prevent anyone trying to reuse a local
> > repository with artifacts downloaded from unknown/unconfigured
> > repository in an offline build.
> >
> > Can someone shed some light on the rationale behind this?
> >
> > Thanks :)
> >
> > zoran
> >
> > [1]
> > https://github.com/eclipse/aether-core/blob/4cf5f7a406b516a45d8bf15e7dfe3fb3849cb87b/aether-impl/src/main/java/org/eclipse/aether/internal/impl/EnhancedLocalRepositoryManager.java#L106-L107
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


--
Zoran Regvart

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Puzzling Aether behaviour

Thomas Broyer-2
In reply to this post by Robert Scholte-8
Le ven. 21 déc. 2018 20:23, Robert Scholte <[hidden email]> a écrit :

> There are a couple of things happening here.
>
> 1. dependencies are checked even though they are downloaded
> This ensures that the build doesn't rely on *your* *local* repository. At
> any time anybody should be able to build the project, it even should be
> possible that you remove your own local repo, so it is important that all
> dependencies are available via a remote repository.
>
> 2. dependencies are verified against their original remote repository.
> Recently I read an article about an interesting case regarding downloads
> of multiple repositories. (sadly I can't find it anymore).
> IIRC what happened is that they noticed a change in behavior of an
> application. After a while they discovered that some third party
> repository was used before Maven Central for downloading dependencies and
> one library was changed in this third party repository. The solution was
> simple: use Maven Central as the primary repository at all time.
>

This is likely the article you're talking about:
https://blog.autsoft.hu/a-confusing-dependency/

This might explain why it is important to not just trust the

> groupId+artifactId+version. The (original) source/repository or
> additional
> kind of checksum is as important.
>
> thanks,
> Robert
>
> On Fri, 21 Dec 2018 13:33:10 +0100, Zoran Regvart <[hidden email]>
> wrote:
>
> > Hi Mavenistas,
> > I was recently quite puzzled on why Maven is trying to download a
> > dependency already present in the local repository and I tracked it
> > down to this comment in EnhancedLocalRepositoryManager[1]:
> >
> > "artifact downloaded from remote repository is accepted only
> > downloaded from request repositories"
> >
> > So regardless of the artifact being present in the local repository an
> > attempt is made to download it again if the id of the repository
> > doesn't match the id of the repository stored in
> > `_remote.repositories` file.
> >
> > I think that this will prevent anyone trying to reuse a local
> > repository with artifacts downloaded from unknown/unconfigured
> > repository in an offline build.
> >
> > Can someone shed some light on the rationale behind this?
> >
> > Thanks :)
> >
> > zoran
> >
> > [1]
> >
> https://github.com/eclipse/aether-core/blob/4cf5f7a406b516a45d8bf15e7dfe3fb3849cb87b/aether-impl/src/main/java/org/eclipse/aether/internal/impl/EnhancedLocalRepositoryManager.java#L106-L107
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Puzzling Aether behaviour

Robert Scholte-8
On Sat, 22 Dec 2018 23:39:55 +0100, Thomas Broyer <[hidden email]>  
wrote:

> https://blog.autsoft.hu/a-confusing-dependency/

Yes, that was the one, thanks!

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Puzzling Aether behaviour

Robert Scholte-8
In reply to this post by Zoran Regvart
AFAIK once downloaded it is only checking its existence every next time.
I do think that things can be improved in the Maven Artifact Resolver[1]  
similar to this idea.
Although you probably still need to re-download the jar and calculate the  
checksum(s), trusting the remote camel-core-2.23.0.jar.sha1 is not good  
enough.

thanks,
Robert

[1] https://maven.apache.org/resolver/index.html


On Sat, 22 Dec 2018 23:28:26 +0100, Zoran Regvart <[hidden email]>  
wrote:

> Hi Robert,
> thanks for taking the time to reply. Do you think it would make sense
> to put the hash value of the artifact in `_remote.repositories`. For
> example I see this for camel-core:
>
> camel-core-2.23.0.jar>maven.central=
>
> It can become:
>
> camel-core-2.23.0.jar>maven.central=2f67a52f3c0aea7a8e2d53acde292569aa7b8087
>
> That is equal to camel-core-2.23.0.jar.sha1[1]
>
> Then instead of downloading the full artifact Aether could opt to
> calculate the hash of the local artifact check with
> `_remote.repositories` or the .sha1 from the remote repository.
>
> That way locally downloaded artifact could be checked for consistency
> without re-downloading it. For example I might have:
>
> camel-core-2.23.0.jar>maven.central=2f67a52f3c0aea7a8e2d53acde292569aa7b8087
> camel-core-2.23.0.jar>my-repo=2f67a52f3c0aea7a8e2d53acde292569aa7b8087
>
> Where the value for `camel-core-2.23.0.jar>maven.central` would come
> from .sha1 file.
>
> Hope my ramblings make some sense :)
>
> Happy holidays!
>
> zoran
>
> [1]  
> http://central.maven.org/maven2/org/apache/camel/camel-core/2.23.0/camel-core-2.23.0.jar.sha1
>
> On Fri, Dec 21, 2018 at 8:23 PM Robert Scholte <[hidden email]>  
> wrote:
>>
>> There are a couple of things happening here.
>>
>> 1. dependencies are checked even though they are downloaded
>> This ensures that the build doesn't rely on *your* *local* repository.  
>> At
>> any time anybody should be able to build the project, it even should be
>> possible that you remove your own local repo, so it is important that  
>> all
>> dependencies are available via a remote repository.
>>
>> 2. dependencies are verified against their original remote repository.
>> Recently I read an article about an interesting case regarding downloads
>> of multiple repositories. (sadly I can't find it anymore).
>> IIRC what happened is that they noticed a change in behavior of an
>> application. After a while they discovered that some third party
>> repository was used before Maven Central for downloading dependencies  
>> and
>> one library was changed in this third party repository. The solution was
>> simple: use Maven Central as the primary repository at all time.
>> This might explain why it is important to not just trust the
>> groupId+artifactId+version. The (original) source/repository or  
>> additional
>> kind of checksum is as important.
>>
>> thanks,
>> Robert
>>
>> On Fri, 21 Dec 2018 13:33:10 +0100, Zoran Regvart <[hidden email]>
>> wrote:
>>
>> > Hi Mavenistas,
>> > I was recently quite puzzled on why Maven is trying to download a
>> > dependency already present in the local repository and I tracked it
>> > down to this comment in EnhancedLocalRepositoryManager[1]:
>> >
>> > "artifact downloaded from remote repository is accepted only
>> > downloaded from request repositories"
>> >
>> > So regardless of the artifact being present in the local repository an
>> > attempt is made to download it again if the id of the repository
>> > doesn't match the id of the repository stored in
>> > `_remote.repositories` file.
>> >
>> > I think that this will prevent anyone trying to reuse a local
>> > repository with artifacts downloaded from unknown/unconfigured
>> > repository in an offline build.
>> >
>> > Can someone shed some light on the rationale behind this?
>> >
>> > Thanks :)
>> >
>> > zoran
>> >
>> > [1]
>> >  
>> https://github.com/eclipse/aether-core/blob/4cf5f7a406b516a45d8bf15e7dfe3fb3849cb87b/aether-impl/src/main/java/org/eclipse/aether/internal/impl/EnhancedLocalRepositoryManager.java#L106-L107
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]