Re: Docker image with initialized local cache saves 50 seconds in startup

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

stephenconnolly
On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]> wrote:

> It's the situation when you have maven plugins in repo and it means that
> all custom plugins/deps can be still downloaded as before.
> Nothing exists like this in the world and we are talking about the
> approaches.
>

Cough cough cough

https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae418ca3fd63c63038e896255c6fe8/Dockerfile#L50

There are issues, but it does shave a bit of time... though we end up
adding our common dependencies into the seed pom so that it is an effective
speed up


>
> I added Karl, Herve and Stephen in CC because we talked about this issue
> in ASF CON and Twitter.
>
> On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <[hidden email]>
> wrote:
>
>> Hi Tibor,
>>
>> It has two issues:
>>
>> 1. It will not be the right plugin versions in 90% of the cases (except
>> demo ;))
>> 2. It will miss all custom plugins
>>
>> Now question is: what happens if you mount your local repo when running
>> docker? It works as expected. Means we could use a custom entrypoint
>> printing a warning banner if it is not done probably instead of incrasing
>> the image size without being sure to reach the original goal.
>>
>> Wdyt?
>>
>> Romain
>>
>> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
>> écrit :
>>
>> > If you use Docker images with Maven with no mapping of cache to the
>> > volumes, you may notice that Maven downloads the plugins for the build
>> > lifecycle.
>> >
>> > This slows down the build because a lot of artifacts and plugins are
>> > initially downloaded.
>> > This takes 50 seconds which might be even longer than the productive
>> build
>> > itself (compiler, package, ...).
>> >
>> > We discussed this topic with Herve and Karl at the Apache CON 2019 the
>> last
>> > time.
>> >
>> > Sometime the presentations were funny because the audience had to wait a
>> > minute while the console was black where the Maven was downloading the
>> > plugins in the background.
>> > Nobody was sure what happened that time, whether the console hanged or
>> the
>> > Cloud server hanged, or another issue happened with the network.
>> >
>> > I made a test and triggered the default lifecycle on Maven and I
>> realized
>> > that the cache was really very little, cca 12 MB.
>> > So this little cache in the container would save 50 seconds which is the
>> > improvement we are discussing.
>> >
>> > From the use point of view, the user would use a new base image `
>> > 3.6.2-jdk-14-prefetched` which is my idea.
>> >
>> > There are multiple technical solutions (range of plugins, extra pom,
>> > internal Maven plugin versions, etc).
>> >
>> > We understood that the best idea would be to have the image with the
>> cache
>> > in new Docker images produced by Carloss Sanchez.
>> >
>> > We are discussing this topic in [1] but we do not have a consensus on
>> who
>> > will develop the Docker scripts and how.
>> >
>> > We can continue here and we can propose a solution.
>> >
>> > [1] https://github.com/carlossg/docker-maven/issues/130
>> >
>> >
>> > Cheers
>> > Tibor17
>> >
>>
> --
Sent from my phone
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Romain Manni-Bucau
Agree we should publish with an asf account and make it part of the release
process.
That said I still fail to see how you can add a relevant cache. Maybe take
the time to review plugin version in a few asf projects (let say
maven-surefire, geronimo-openapi and spark) and check out if it works to
cache anything.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le mer. 30 oct. 2019 à 14:31, Tibor Digana <[hidden email]> a
écrit :

> Stephen, yeah something like you do in your scrip but it must not be a
> personal owner. Even Carloss is person who makes this deployment to
> DockerHub but his images are used by the entire world and we should decide
> whether we would agree with him to have such images under his
> responsibility or our responsibility as the Apache group. Then the script
> would be official and we can cut the release of Maven and the Docker image
> will be ready for the downloads together with the Maven distribution. So
> the users will always know that it is consistent deployment and they
> wouldn't expect that the image is missing for the new version.
>
> So here we should decide on who will be the deployer of these images with
> the cache. And the technical solution is smaller problem I would say.
>
> On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> [hidden email]> wrote:
>
> > On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]>
> wrote:
> >
> > > It's the situation when you have maven plugins in repo and it means
> that
> > > all custom plugins/deps can be still downloaded as before.
> > > Nothing exists like this in the world and we are talking about the
> > > approaches.
> > >
> >
> > Cough cough cough
> >
> >
> >
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae418ca3fd63c63038e896255c6fe8/Dockerfile#L50
> >
> > There are issues, but it does shave a bit of time... though we end up
> > adding our common dependencies into the seed pom so that it is an
> effective
> > speed up
> >
> >
> > >
> > > I added Karl, Herve and Stephen in CC because we talked about this
> issue
> > > in ASF CON and Twitter.
> > >
> > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > [hidden email]>
> > > wrote:
> > >
> > >> Hi Tibor,
> > >>
> > >> It has two issues:
> > >>
> > >> 1. It will not be the right plugin versions in 90% of the cases
> (except
> > >> demo ;))
> > >> 2. It will miss all custom plugins
> > >>
> > >> Now question is: what happens if you mount your local repo when
> running
> > >> docker? It works as expected. Means we could use a custom entrypoint
> > >> printing a warning banner if it is not done probably instead of
> > incrasing
> > >> the image size without being sure to reach the original goal.
> > >>
> > >> Wdyt?
> > >>
> > >> Romain
> > >>
> > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
> > >> écrit :
> > >>
> > >> > If you use Docker images with Maven with no mapping of cache to the
> > >> > volumes, you may notice that Maven downloads the plugins for the
> build
> > >> > lifecycle.
> > >> >
> > >> > This slows down the build because a lot of artifacts and plugins are
> > >> > initially downloaded.
> > >> > This takes 50 seconds which might be even longer than the productive
> > >> build
> > >> > itself (compiler, package, ...).
> > >> >
> > >> > We discussed this topic with Herve and Karl at the Apache CON 2019
> the
> > >> last
> > >> > time.
> > >> >
> > >> > Sometime the presentations were funny because the audience had to
> > wait a
> > >> > minute while the console was black where the Maven was downloading
> the
> > >> > plugins in the background.
> > >> > Nobody was sure what happened that time, whether the console hanged
> or
> > >> the
> > >> > Cloud server hanged, or another issue happened with the network.
> > >> >
> > >> > I made a test and triggered the default lifecycle on Maven and I
> > >> realized
> > >> > that the cache was really very little, cca 12 MB.
> > >> > So this little cache in the container would save 50 seconds which is
> > the
> > >> > improvement we are discussing.
> > >> >
> > >> > From the use point of view, the user would use a new base image `
> > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > >> >
> > >> > There are multiple technical solutions (range of plugins, extra pom,
> > >> > internal Maven plugin versions, etc).
> > >> >
> > >> > We understood that the best idea would be to have the image with the
> > >> cache
> > >> > in new Docker images produced by Carloss Sanchez.
> > >> >
> > >> > We are discussing this topic in [1] but we do not have a consensus
> on
> > >> who
> > >> > will develop the Docker scripts and how.
> > >> >
> > >> > We can continue here and we can propose a solution.
> > >> >
> > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > >> >
> > >> >
> > >> > Cheers
> > >> > Tibor17
> > >> >
> > >>
> > > --
> > Sent from my phone
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Tibor Digana
Thx Herve for the hint. The size of the local cahe is 40 MB.

All core plugins [1] and Packaging (ear, ejb, jar, rar and war) are
included in the range [3.0, 4.0).
Two plugins (verifier:2.0+ and rar:1.0+) do not have this range.

[1] https://maven.apache.org/plugins/


On Wed, Oct 30, 2019 at 10:47 PM Hervé Boutemy <[hidden email]> wrote:

> Le mercredi 30 octobre 2019 14:55:46 CET, vous avez écrit :
> > Agree we should publish with an asf account and make it part of the
> release
> > process.
> > That said I still fail to see how you can add a relevant cache. Maybe
> take
> > the time to review plugin version in a few asf projects (let say
> > maven-surefire, geronimo-openapi and spark) and check out if it works to
> > cache anything.
> if we choose one precise version of each plugin, we will never get the
> version
> used by a project: IMHO, we need to choose a range for each plugin
> https://maven.apache.org/plugins/
> Something like dependency:resolve-plugins should be able to do the job.
>
> With a range per plugin, I'd love to see the size of the initial local
> repository
> At least, using such an image would require minimal efforts from users,
> particularly on CI servers
>
> Regards,
>
> Hervé
>
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau>
> > | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > >
> >
> >
> > Le mer. 30 oct. 2019 à 14:31, Tibor Digana <[hidden email]> a
> >
> > écrit :
> > > Stephen, yeah something like you do in your scrip but it must not be a
> > > personal owner. Even Carloss is person who makes this deployment to
> > > DockerHub but his images are used by the entire world and we should
> decide
> > > whether we would agree with him to have such images under his
> > > responsibility or our responsibility as the Apache group. Then the
> script
> > > would be official and we can cut the release of Maven and the Docker
> image
> > > will be ready for the downloads together with the Maven distribution.
> So
> > > the users will always know that it is consistent deployment and they
> > > wouldn't expect that the image is missing for the new version.
> > >
> > > So here we should decide on who will be the deployer of these images
> with
> > > the cache. And the technical solution is smaller problem I would say.
> > >
> > > On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> > >
> > > [hidden email]> wrote:
> > > > On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]>
> > >
> > > wrote:
> > > > > It's the situation when you have maven plugins in repo and it means
> > >
> > > that
> > >
> > > > > all custom plugins/deps can be still downloaded as before.
> > > > > Nothing exists like this in the world and we are talking about the
> > > > > approaches.
> > > >
> > > > Cough cough cough
> > >
> > >
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae41
> > > 8ca3fd63c63038e896255c6fe8/Dockerfile#L50>
> > > > There are issues, but it does shave a bit of time... though we end up
> > > > adding our common dependencies into the seed pom so that it is an
> > >
> > > effective
> > >
> > > > speed up
> > > >
> > > > > I added Karl, Herve and Stephen in CC because we talked about this
> > >
> > > issue
> > >
> > > > > in ASF CON and Twitter.
> > > > >
> > > > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > > >
> > > > [hidden email]>
> > > >
> > > > > wrote:
> > > > >> Hi Tibor,
> > > > >>
> > > > >> It has two issues:
> > > > >>
> > > > >> 1. It will not be the right plugin versions in 90% of the cases
> > >
> > > (except
> > >
> > > > >> demo ;))
> > > > >> 2. It will miss all custom plugins
> > > > >>
> > > > >> Now question is: what happens if you mount your local repo when
> > >
> > > running
> > >
> > > > >> docker? It works as expected. Means we could use a custom
> entrypoint
> > > > >> printing a warning banner if it is not done probably instead of
> > > >
> > > > incrasing
> > > >
> > > > >> the image size without being sure to reach the original goal.
> > > > >>
> > > > >> Wdyt?
> > > > >>
> > > > >> Romain
> > > > >>
> > > > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <
> [hidden email]> a
> > > > >>
> > > > >> écrit :
> > > > >> > If you use Docker images with Maven with no mapping of cache to
> the
> > > > >> > volumes, you may notice that Maven downloads the plugins for the
> > >
> > > build
> > >
> > > > >> > lifecycle.
> > > > >> >
> > > > >> > This slows down the build because a lot of artifacts and plugins
> > > > >> > are
> > > > >> > initially downloaded.
> > > > >> > This takes 50 seconds which might be even longer than the
> > > > >> > productive
> > > > >>
> > > > >> build
> > > > >>
> > > > >> > itself (compiler, package, ...).
> > > > >> >
> > > > >> > We discussed this topic with Herve and Karl at the Apache CON
> 2019
> > >
> > > the
> > >
> > > > >> last
> > > > >>
> > > > >> > time.
> > > > >> >
> > > > >> > Sometime the presentations were funny because the audience had
> to
> > > >
> > > > wait a
> > > >
> > > > >> > minute while the console was black where the Maven was
> downloading
> > >
> > > the
> > >
> > > > >> > plugins in the background.
> > > > >> > Nobody was sure what happened that time, whether the console
> hanged
> > >
> > > or
> > >
> > > > >> the
> > > > >>
> > > > >> > Cloud server hanged, or another issue happened with the network.
> > > > >> >
> > > > >> > I made a test and triggered the default lifecycle on Maven and I
> > > > >>
> > > > >> realized
> > > > >>
> > > > >> > that the cache was really very little, cca 12 MB.
> > > > >> > So this little cache in the container would save 50 seconds
> which
> > > > >> > is
> > > >
> > > > the
> > > >
> > > > >> > improvement we are discussing.
> > > > >> >
> > > > >> > From the use point of view, the user would use a new base image
> `
> > > > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > > > >> >
> > > > >> > There are multiple technical solutions (range of plugins, extra
> > > > >> > pom,
> > > > >> > internal Maven plugin versions, etc).
> > > > >> >
> > > > >> > We understood that the best idea would be to have the image with
> > > > >> > the
> > > > >>
> > > > >> cache
> > > > >>
> > > > >> > in new Docker images produced by Carloss Sanchez.
> > > > >> >
> > > > >> > We are discussing this topic in [1] but we do not have a
> consensus
> > >
> > > on
> > >
> > > > >> who
> > > > >>
> > > > >> > will develop the Docker scripts and how.
> > > > >> >
> > > > >> > We can continue here and we can propose a solution.
> > > > >> >
> > > > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > > > >> >
> > > > >> >
> > > > >> > Cheers
> > > > >> > Tibor17
> > > > >
> > > > > --
> > > >
> > > > Sent from my phone
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>