Re: Docker image with initialized local cache saves 50 seconds in startup

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

hboutemy
Le mercredi 30 octobre 2019 14:55:46 CET, vous avez écrit :
> Agree we should publish with an asf account and make it part of the release
> process.
> That said I still fail to see how you can add a relevant cache. Maybe take
> the time to review plugin version in a few asf projects (let say
> maven-surefire, geronimo-openapi and spark) and check out if it works to
> cache anything.
if we choose one precise version of each plugin, we will never get the version
used by a project: IMHO, we need to choose a range for each plugin
https://maven.apache.org/plugins/
Something like dependency:resolve-plugins should be able to do the job.

With a range per plugin, I'd love to see the size of the initial local
repository
At least, using such an image would require minimal efforts from users,
particularly on CI servers

Regards,

Hervé

>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau>
> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
>
>
> Le mer. 30 oct. 2019 à 14:31, Tibor Digana <[hidden email]> a
>
> écrit :
> > Stephen, yeah something like you do in your scrip but it must not be a
> > personal owner. Even Carloss is person who makes this deployment to
> > DockerHub but his images are used by the entire world and we should decide
> > whether we would agree with him to have such images under his
> > responsibility or our responsibility as the Apache group. Then the script
> > would be official and we can cut the release of Maven and the Docker image
> > will be ready for the downloads together with the Maven distribution. So
> > the users will always know that it is consistent deployment and they
> > wouldn't expect that the image is missing for the new version.
> >
> > So here we should decide on who will be the deployer of these images with
> > the cache. And the technical solution is smaller problem I would say.
> >
> > On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> >
> > [hidden email]> wrote:
> > > On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]>
> >
> > wrote:
> > > > It's the situation when you have maven plugins in repo and it means
> >
> > that
> >
> > > > all custom plugins/deps can be still downloaded as before.
> > > > Nothing exists like this in the world and we are talking about the
> > > > approaches.
> > >
> > > Cough cough cough
> >
> > https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae41
> > 8ca3fd63c63038e896255c6fe8/Dockerfile#L50>
> > > There are issues, but it does shave a bit of time... though we end up
> > > adding our common dependencies into the seed pom so that it is an
> >
> > effective
> >
> > > speed up
> > >
> > > > I added Karl, Herve and Stephen in CC because we talked about this
> >
> > issue
> >
> > > > in ASF CON and Twitter.
> > > >
> > > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > >
> > > [hidden email]>
> > >
> > > > wrote:
> > > >> Hi Tibor,
> > > >>
> > > >> It has two issues:
> > > >>
> > > >> 1. It will not be the right plugin versions in 90% of the cases
> >
> > (except
> >
> > > >> demo ;))
> > > >> 2. It will miss all custom plugins
> > > >>
> > > >> Now question is: what happens if you mount your local repo when
> >
> > running
> >
> > > >> docker? It works as expected. Means we could use a custom entrypoint
> > > >> printing a warning banner if it is not done probably instead of
> > >
> > > incrasing
> > >
> > > >> the image size without being sure to reach the original goal.
> > > >>
> > > >> Wdyt?
> > > >>
> > > >> Romain
> > > >>
> > > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
> > > >>
> > > >> écrit :
> > > >> > If you use Docker images with Maven with no mapping of cache to the
> > > >> > volumes, you may notice that Maven downloads the plugins for the
> >
> > build
> >
> > > >> > lifecycle.
> > > >> >
> > > >> > This slows down the build because a lot of artifacts and plugins
> > > >> > are
> > > >> > initially downloaded.
> > > >> > This takes 50 seconds which might be even longer than the
> > > >> > productive
> > > >>
> > > >> build
> > > >>
> > > >> > itself (compiler, package, ...).
> > > >> >
> > > >> > We discussed this topic with Herve and Karl at the Apache CON 2019
> >
> > the
> >
> > > >> last
> > > >>
> > > >> > time.
> > > >> >
> > > >> > Sometime the presentations were funny because the audience had to
> > >
> > > wait a
> > >
> > > >> > minute while the console was black where the Maven was downloading
> >
> > the
> >
> > > >> > plugins in the background.
> > > >> > Nobody was sure what happened that time, whether the console hanged
> >
> > or
> >
> > > >> the
> > > >>
> > > >> > Cloud server hanged, or another issue happened with the network.
> > > >> >
> > > >> > I made a test and triggered the default lifecycle on Maven and I
> > > >>
> > > >> realized
> > > >>
> > > >> > that the cache was really very little, cca 12 MB.
> > > >> > So this little cache in the container would save 50 seconds which
> > > >> > is
> > >
> > > the
> > >
> > > >> > improvement we are discussing.
> > > >> >
> > > >> > From the use point of view, the user would use a new base image `
> > > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > > >> >
> > > >> > There are multiple technical solutions (range of plugins, extra
> > > >> > pom,
> > > >> > internal Maven plugin versions, etc).
> > > >> >
> > > >> > We understood that the best idea would be to have the image with
> > > >> > the
> > > >>
> > > >> cache
> > > >>
> > > >> > in new Docker images produced by Carloss Sanchez.
> > > >> >
> > > >> > We are discussing this topic in [1] but we do not have a consensus
> >
> > on
> >
> > > >> who
> > > >>
> > > >> > will develop the Docker scripts and how.
> > > >> >
> > > >> > We can continue here and we can propose a solution.
> > > >> >
> > > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > > >> >
> > > >> >
> > > >> > Cheers
> > > >> > Tibor17
> > > >
> > > > --
> > >
> > > Sent from my phone





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Romain Manni-Bucau
What is the usage of this first group outside demos (it is trivial to build
a dedicated image for demos so i drop it from any asf work for now)?

Le mer. 30 oct. 2019 à 23:18, Tibor Digana <[hidden email]> a
écrit :

> Romain, of course we know this but there are two groups of users.
> First those who use the Maven in docker without CI.
> And second of those who integrate the docker container with the CI and they
> use the volumes.
>
> The first group of people may freely use the image we are discussing here
> because this makes the default build lifecycle faster for them.
> For us it means that we will not be seens as "slow Maven", you know what i
> mean.
>
> The second group already use the maven images from Carloss.
>
> On Wed, Oct 30, 2019 at 11:12 PM Romain Manni-Bucau <[hidden email]
> >
> wrote:
>
> > Ci servers have all cache so it is useless to make the docker pull slow
> > with useless versions to then download or not plugins IMHO.
> >
> > Le mer. 30 oct. 2019 à 22:47, Hervé Boutemy <[hidden email]> a
> écrit
> > :
> >
> > > Le mercredi 30 octobre 2019 14:55:46 CET, vous avez écrit :
> > > > Agree we should publish with an asf account and make it part of the
> > > release
> > > > process.
> > > > That said I still fail to see how you can add a relevant cache. Maybe
> > > take
> > > > the time to review plugin version in a few asf projects (let say
> > > > maven-surefire, geronimo-openapi and spark) and check out if it works
> > to
> > > > cache anything.
> > > if we choose one precise version of each plugin, we will never get the
> > > version
> > > used by a project: IMHO, we need to choose a range for each plugin
> > > https://maven.apache.org/plugins/
> > > Something like dependency:resolve-plugins should be able to do the job.
> > >
> > > With a range per plugin, I'd love to see the size of the initial local
> > > repository
> > > At least, using such an image would require minimal efforts from users,
> > > particularly on CI servers
> > >
> > > Regards,
> > >
> > > Hervé
> > >
> > > >
> > > > Romain Manni-Bucau
> > > > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > > > <https://rmannibucau.metawerx.net/> | Old Blog
> > > > <http://rmannibucau.wordpress.com> | Github <
> > > https://github.com/rmannibucau>
> > > > | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > > > <
> > >
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > > > >
> > > >
> > > >
> > > > Le mer. 30 oct. 2019 à 14:31, Tibor Digana <[hidden email]>
> a
> > > >
> > > > écrit :
> > > > > Stephen, yeah something like you do in your scrip but it must not
> be
> > a
> > > > > personal owner. Even Carloss is person who makes this deployment to
> > > > > DockerHub but his images are used by the entire world and we should
> > > decide
> > > > > whether we would agree with him to have such images under his
> > > > > responsibility or our responsibility as the Apache group. Then the
> > > script
> > > > > would be official and we can cut the release of Maven and the
> Docker
> > > image
> > > > > will be ready for the downloads together with the Maven
> distribution.
> > > So
> > > > > the users will always know that it is consistent deployment and
> they
> > > > > wouldn't expect that the image is missing for the new version.
> > > > >
> > > > > So here we should decide on who will be the deployer of these
> images
> > > with
> > > > > the cache. And the technical solution is smaller problem I would
> say.
> > > > >
> > > > > On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> > > > >
> > > > > [hidden email]> wrote:
> > > > > > On Wed 30 Oct 2019 at 08:21, Tibor Digana <
> [hidden email]>
> > > > >
> > > > > wrote:
> > > > > > > It's the situation when you have maven plugins in repo and it
> > means
> > > > >
> > > > > that
> > > > >
> > > > > > > all custom plugins/deps can be still downloaded as before.
> > > > > > > Nothing exists like this in the world and we are talking about
> > the
> > > > > > > approaches.
> > > > > >
> > > > > > Cough cough cough
> > > > >
> > > > >
> > >
> >
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae41
> > > > > 8ca3fd63c63038e896255c6fe8/Dockerfile#L50>
> > > > > > There are issues, but it does shave a bit of time... though we
> end
> > up
> > > > > > adding our common dependencies into the seed pom so that it is an
> > > > >
> > > > > effective
> > > > >
> > > > > > speed up
> > > > > >
> > > > > > > I added Karl, Herve and Stephen in CC because we talked about
> > this
> > > > >
> > > > > issue
> > > > >
> > > > > > > in ASF CON and Twitter.
> > > > > > >
> > > > > > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > > > > >
> > > > > > [hidden email]>
> > > > > >
> > > > > > > wrote:
> > > > > > >> Hi Tibor,
> > > > > > >>
> > > > > > >> It has two issues:
> > > > > > >>
> > > > > > >> 1. It will not be the right plugin versions in 90% of the
> cases
> > > > >
> > > > > (except
> > > > >
> > > > > > >> demo ;))
> > > > > > >> 2. It will miss all custom plugins
> > > > > > >>
> > > > > > >> Now question is: what happens if you mount your local repo
> when
> > > > >
> > > > > running
> > > > >
> > > > > > >> docker? It works as expected. Means we could use a custom
> > > entrypoint
> > > > > > >> printing a warning banner if it is not done probably instead
> of
> > > > > >
> > > > > > incrasing
> > > > > >
> > > > > > >> the image size without being sure to reach the original goal.
> > > > > > >>
> > > > > > >> Wdyt?
> > > > > > >>
> > > > > > >> Romain
> > > > > > >>
> > > > > > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <
> > > [hidden email]> a
> > > > > > >>
> > > > > > >> écrit :
> > > > > > >> > If you use Docker images with Maven with no mapping of cache
> > to
> > > the
> > > > > > >> > volumes, you may notice that Maven downloads the plugins for
> > the
> > > > >
> > > > > build
> > > > >
> > > > > > >> > lifecycle.
> > > > > > >> >
> > > > > > >> > This slows down the build because a lot of artifacts and
> > plugins
> > > > > > >> > are
> > > > > > >> > initially downloaded.
> > > > > > >> > This takes 50 seconds which might be even longer than the
> > > > > > >> > productive
> > > > > > >>
> > > > > > >> build
> > > > > > >>
> > > > > > >> > itself (compiler, package, ...).
> > > > > > >> >
> > > > > > >> > We discussed this topic with Herve and Karl at the Apache
> CON
> > > 2019
> > > > >
> > > > > the
> > > > >
> > > > > > >> last
> > > > > > >>
> > > > > > >> > time.
> > > > > > >> >
> > > > > > >> > Sometime the presentations were funny because the audience
> had
> > > to
> > > > > >
> > > > > > wait a
> > > > > >
> > > > > > >> > minute while the console was black where the Maven was
> > > downloading
> > > > >
> > > > > the
> > > > >
> > > > > > >> > plugins in the background.
> > > > > > >> > Nobody was sure what happened that time, whether the console
> > > hanged
> > > > >
> > > > > or
> > > > >
> > > > > > >> the
> > > > > > >>
> > > > > > >> > Cloud server hanged, or another issue happened with the
> > network.
> > > > > > >> >
> > > > > > >> > I made a test and triggered the default lifecycle on Maven
> > and I
> > > > > > >>
> > > > > > >> realized
> > > > > > >>
> > > > > > >> > that the cache was really very little, cca 12 MB.
> > > > > > >> > So this little cache in the container would save 50 seconds
> > > which
> > > > > > >> > is
> > > > > >
> > > > > > the
> > > > > >
> > > > > > >> > improvement we are discussing.
> > > > > > >> >
> > > > > > >> > From the use point of view, the user would use a new base
> > image
> > > `
> > > > > > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > > > > > >> >
> > > > > > >> > There are multiple technical solutions (range of plugins,
> > extra
> > > > > > >> > pom,
> > > > > > >> > internal Maven plugin versions, etc).
> > > > > > >> >
> > > > > > >> > We understood that the best idea would be to have the image
> > with
> > > > > > >> > the
> > > > > > >>
> > > > > > >> cache
> > > > > > >>
> > > > > > >> > in new Docker images produced by Carloss Sanchez.
> > > > > > >> >
> > > > > > >> > We are discussing this topic in [1] but we do not have a
> > > consensus
> > > > >
> > > > > on
> > > > >
> > > > > > >> who
> > > > > > >>
> > > > > > >> > will develop the Docker scripts and how.
> > > > > > >> >
> > > > > > >> > We can continue here and we can propose a solution.
> > > > > > >> >
> > > > > > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Cheers
> > > > > > >> > Tibor17
> > > > > > >
> > > > > > > --
> > > > > >
> > > > > > Sent from my phone
> > >
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> > >
> >
>