Re: Docker image with initialized local cache saves 50 seconds in startup

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Romain Manni-Bucau
Hi Tibor,

It has two issues:

1. It will not be the right plugin versions in 90% of the cases (except
demo ;))
2. It will miss all custom plugins

Now question is: what happens if you mount your local repo when running
docker? It works as expected. Means we could use a custom entrypoint
printing a warning banner if it is not done probably instead of incrasing
the image size without being sure to reach the original goal.

Wdyt?

Romain

Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
écrit :

> If you use Docker images with Maven with no mapping of cache to the
> volumes, you may notice that Maven downloads the plugins for the build
> lifecycle.
>
> This slows down the build because a lot of artifacts and plugins are
> initially downloaded.
> This takes 50 seconds which might be even longer than the productive build
> itself (compiler, package, ...).
>
> We discussed this topic with Herve and Karl at the Apache CON 2019 the last
> time.
>
> Sometime the presentations were funny because the audience had to wait a
> minute while the console was black where the Maven was downloading the
> plugins in the background.
> Nobody was sure what happened that time, whether the console hanged or the
> Cloud server hanged, or another issue happened with the network.
>
> I made a test and triggered the default lifecycle on Maven and I realized
> that the cache was really very little, cca 12 MB.
> So this little cache in the container would save 50 seconds which is the
> improvement we are discussing.
>
> From the use point of view, the user would use a new base image `
> 3.6.2-jdk-14-prefetched` which is my idea.
>
> There are multiple technical solutions (range of plugins, extra pom,
> internal Maven plugin versions, etc).
>
> We understood that the best idea would be to have the image with the cache
> in new Docker images produced by Carloss Sanchez.
>
> We are discussing this topic in [1] but we do not have a consensus on who
> will develop the Docker scripts and how.
>
> We can continue here and we can propose a solution.
>
> [1] https://github.com/carlossg/docker-maven/issues/130
>
>
> Cheers
> Tibor17
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Tibor Digana
It's the situation when you have maven plugins in repo and it means that
all custom plugins/deps can be still downloaded as before.
Nothing exists like this in the world and we are talking about the
approaches.

I added Karl, Herve and Stephen in CC because we talked about this issue in
ASF CON and Twitter.

On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <[hidden email]>
wrote:

> Hi Tibor,
>
> It has two issues:
>
> 1. It will not be the right plugin versions in 90% of the cases (except
> demo ;))
> 2. It will miss all custom plugins
>
> Now question is: what happens if you mount your local repo when running
> docker? It works as expected. Means we could use a custom entrypoint
> printing a warning banner if it is not done probably instead of incrasing
> the image size without being sure to reach the original goal.
>
> Wdyt?
>
> Romain
>
> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
> écrit :
>
> > If you use Docker images with Maven with no mapping of cache to the
> > volumes, you may notice that Maven downloads the plugins for the build
> > lifecycle.
> >
> > This slows down the build because a lot of artifacts and plugins are
> > initially downloaded.
> > This takes 50 seconds which might be even longer than the productive
> build
> > itself (compiler, package, ...).
> >
> > We discussed this topic with Herve and Karl at the Apache CON 2019 the
> last
> > time.
> >
> > Sometime the presentations were funny because the audience had to wait a
> > minute while the console was black where the Maven was downloading the
> > plugins in the background.
> > Nobody was sure what happened that time, whether the console hanged or
> the
> > Cloud server hanged, or another issue happened with the network.
> >
> > I made a test and triggered the default lifecycle on Maven and I realized
> > that the cache was really very little, cca 12 MB.
> > So this little cache in the container would save 50 seconds which is the
> > improvement we are discussing.
> >
> > From the use point of view, the user would use a new base image `
> > 3.6.2-jdk-14-prefetched` which is my idea.
> >
> > There are multiple technical solutions (range of plugins, extra pom,
> > internal Maven plugin versions, etc).
> >
> > We understood that the best idea would be to have the image with the
> cache
> > in new Docker images produced by Carloss Sanchez.
> >
> > We are discussing this topic in [1] but we do not have a consensus on who
> > will develop the Docker scripts and how.
> >
> > We can continue here and we can propose a solution.
> >
> > [1] https://github.com/carlossg/docker-maven/issues/130
> >
> >
> > Cheers
> > Tibor17
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Tibor Digana
I don't want your dependencies of course because it is user specific issue.
I only want to trigger "mvn clean install" or deploy and this will fetch
the plugins, not the dependencies from a dummy empty pom. (don't mean
user's pom). Finally, the user will have plugins in the image which come
from the definition of the bindings for particular Maven dist. In 3.7.0 we
will have new plugins in the bindings and so the dummy POM will not
necessarily specify any. That's my point. I think it is very simple but the
problem is if our group would deploy Docker images with default plugins in
the cache or Carloss will do it.

On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
[hidden email]> wrote:

> On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]> wrote:
>
> > It's the situation when you have maven plugins in repo and it means that
> > all custom plugins/deps can be still downloaded as before.
> > Nothing exists like this in the world and we are talking about the
> > approaches.
> >
>
> Cough cough cough
>
>
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae418ca3fd63c63038e896255c6fe8/Dockerfile#L50
>
> There are issues, but it does shave a bit of time... though we end up
> adding our common dependencies into the seed pom so that it is an effective
> speed up
>
>
> >
> > I added Karl, Herve and Stephen in CC because we talked about this issue
> > in ASF CON and Twitter.
> >
> > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> [hidden email]>
> > wrote:
> >
> >> Hi Tibor,
> >>
> >> It has two issues:
> >>
> >> 1. It will not be the right plugin versions in 90% of the cases (except
> >> demo ;))
> >> 2. It will miss all custom plugins
> >>
> >> Now question is: what happens if you mount your local repo when running
> >> docker? It works as expected. Means we could use a custom entrypoint
> >> printing a warning banner if it is not done probably instead of
> incrasing
> >> the image size without being sure to reach the original goal.
> >>
> >> Wdyt?
> >>
> >> Romain
> >>
> >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
> >> écrit :
> >>
> >> > If you use Docker images with Maven with no mapping of cache to the
> >> > volumes, you may notice that Maven downloads the plugins for the build
> >> > lifecycle.
> >> >
> >> > This slows down the build because a lot of artifacts and plugins are
> >> > initially downloaded.
> >> > This takes 50 seconds which might be even longer than the productive
> >> build
> >> > itself (compiler, package, ...).
> >> >
> >> > We discussed this topic with Herve and Karl at the Apache CON 2019 the
> >> last
> >> > time.
> >> >
> >> > Sometime the presentations were funny because the audience had to
> wait a
> >> > minute while the console was black where the Maven was downloading the
> >> > plugins in the background.
> >> > Nobody was sure what happened that time, whether the console hanged or
> >> the
> >> > Cloud server hanged, or another issue happened with the network.
> >> >
> >> > I made a test and triggered the default lifecycle on Maven and I
> >> realized
> >> > that the cache was really very little, cca 12 MB.
> >> > So this little cache in the container would save 50 seconds which is
> the
> >> > improvement we are discussing.
> >> >
> >> > From the use point of view, the user would use a new base image `
> >> > 3.6.2-jdk-14-prefetched` which is my idea.
> >> >
> >> > There are multiple technical solutions (range of plugins, extra pom,
> >> > internal Maven plugin versions, etc).
> >> >
> >> > We understood that the best idea would be to have the image with the
> >> cache
> >> > in new Docker images produced by Carloss Sanchez.
> >> >
> >> > We are discussing this topic in [1] but we do not have a consensus on
> >> who
> >> > will develop the Docker scripts and how.
> >> >
> >> > We can continue here and we can propose a solution.
> >> >
> >> > [1] https://github.com/carlossg/docker-maven/issues/130
> >> >
> >> >
> >> > Cheers
> >> > Tibor17
> >> >
> >>
> > --
> Sent from my phone
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Tibor Digana
In reply to this post by Tibor Digana
Stephen, yeah something like you do in your scrip but it must not be a
personal owner. Even Carloss is person who makes this deployment to
DockerHub but his images are used by the entire world and we should decide
whether we would agree with him to have such images under his
responsibility or our responsibility as the Apache group. Then the script
would be official and we can cut the release of Maven and the Docker image
will be ready for the downloads together with the Maven distribution. So
the users will always know that it is consistent deployment and they
wouldn't expect that the image is missing for the new version.

So here we should decide on who will be the deployer of these images with
the cache. And the technical solution is smaller problem I would say.

On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
[hidden email]> wrote:

> On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]> wrote:
>
> > It's the situation when you have maven plugins in repo and it means that
> > all custom plugins/deps can be still downloaded as before.
> > Nothing exists like this in the world and we are talking about the
> > approaches.
> >
>
> Cough cough cough
>
>
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae418ca3fd63c63038e896255c6fe8/Dockerfile#L50
>
> There are issues, but it does shave a bit of time... though we end up
> adding our common dependencies into the seed pom so that it is an effective
> speed up
>
>
> >
> > I added Karl, Herve and Stephen in CC because we talked about this issue
> > in ASF CON and Twitter.
> >
> > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> [hidden email]>
> > wrote:
> >
> >> Hi Tibor,
> >>
> >> It has two issues:
> >>
> >> 1. It will not be the right plugin versions in 90% of the cases (except
> >> demo ;))
> >> 2. It will miss all custom plugins
> >>
> >> Now question is: what happens if you mount your local repo when running
> >> docker? It works as expected. Means we could use a custom entrypoint
> >> printing a warning banner if it is not done probably instead of
> incrasing
> >> the image size without being sure to reach the original goal.
> >>
> >> Wdyt?
> >>
> >> Romain
> >>
> >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
> >> écrit :
> >>
> >> > If you use Docker images with Maven with no mapping of cache to the
> >> > volumes, you may notice that Maven downloads the plugins for the build
> >> > lifecycle.
> >> >
> >> > This slows down the build because a lot of artifacts and plugins are
> >> > initially downloaded.
> >> > This takes 50 seconds which might be even longer than the productive
> >> build
> >> > itself (compiler, package, ...).
> >> >
> >> > We discussed this topic with Herve and Karl at the Apache CON 2019 the
> >> last
> >> > time.
> >> >
> >> > Sometime the presentations were funny because the audience had to
> wait a
> >> > minute while the console was black where the Maven was downloading the
> >> > plugins in the background.
> >> > Nobody was sure what happened that time, whether the console hanged or
> >> the
> >> > Cloud server hanged, or another issue happened with the network.
> >> >
> >> > I made a test and triggered the default lifecycle on Maven and I
> >> realized
> >> > that the cache was really very little, cca 12 MB.
> >> > So this little cache in the container would save 50 seconds which is
> the
> >> > improvement we are discussing.
> >> >
> >> > From the use point of view, the user would use a new base image `
> >> > 3.6.2-jdk-14-prefetched` which is my idea.
> >> >
> >> > There are multiple technical solutions (range of plugins, extra pom,
> >> > internal Maven plugin versions, etc).
> >> >
> >> > We understood that the best idea would be to have the image with the
> >> cache
> >> > in new Docker images produced by Carloss Sanchez.
> >> >
> >> > We are discussing this topic in [1] but we do not have a consensus on
> >> who
> >> > will develop the Docker scripts and how.
> >> >
> >> > We can continue here and we can propose a solution.
> >> >
> >> > [1] https://github.com/carlossg/docker-maven/issues/130
> >> >
> >> >
> >> > Cheers
> >> > Tibor17
> >> >
> >>
> > --
> Sent from my phone
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

stephenconnolly
My script is for my cases. I'm just saying it's not rocket science, others
are doing it already

On Wed, 30 Oct 2019 at 13:31, Tibor Digana <[hidden email]> wrote:

> Stephen, yeah something like you do in your scrip but it must not be a
> personal owner. Even Carloss is person who makes this deployment to
> DockerHub but his images are used by the entire world and we should decide
> whether we would agree with him to have such images under his
> responsibility or our responsibility as the Apache group. Then the script
> would be official and we can cut the release of Maven and the Docker image
> will be ready for the downloads together with the Maven distribution. So
> the users will always know that it is consistent deployment and they
> wouldn't expect that the image is missing for the new version.
>
> So here we should decide on who will be the deployer of these images with
> the cache. And the technical solution is smaller problem I would say.
>
> On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> [hidden email]> wrote:
>
> > On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]>
> wrote:
> >
> > > It's the situation when you have maven plugins in repo and it means
> that
> > > all custom plugins/deps can be still downloaded as before.
> > > Nothing exists like this in the world and we are talking about the
> > > approaches.
> > >
> >
> > Cough cough cough
> >
> >
> >
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae418ca3fd63c63038e896255c6fe8/Dockerfile#L50
> >
> > There are issues, but it does shave a bit of time... though we end up
> > adding our common dependencies into the seed pom so that it is an
> effective
> > speed up
> >
> >
> > >
> > > I added Karl, Herve and Stephen in CC because we talked about this
> issue
> > > in ASF CON and Twitter.
> > >
> > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > [hidden email]>
> > > wrote:
> > >
> > >> Hi Tibor,
> > >>
> > >> It has two issues:
> > >>
> > >> 1. It will not be the right plugin versions in 90% of the cases
> (except
> > >> demo ;))
> > >> 2. It will miss all custom plugins
> > >>
> > >> Now question is: what happens if you mount your local repo when
> running
> > >> docker? It works as expected. Means we could use a custom entrypoint
> > >> printing a warning banner if it is not done probably instead of
> > incrasing
> > >> the image size without being sure to reach the original goal.
> > >>
> > >> Wdyt?
> > >>
> > >> Romain
> > >>
> > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]> a
> > >> écrit :
> > >>
> > >> > If you use Docker images with Maven with no mapping of cache to the
> > >> > volumes, you may notice that Maven downloads the plugins for the
> build
> > >> > lifecycle.
> > >> >
> > >> > This slows down the build because a lot of artifacts and plugins are
> > >> > initially downloaded.
> > >> > This takes 50 seconds which might be even longer than the productive
> > >> build
> > >> > itself (compiler, package, ...).
> > >> >
> > >> > We discussed this topic with Herve and Karl at the Apache CON 2019
> the
> > >> last
> > >> > time.
> > >> >
> > >> > Sometime the presentations were funny because the audience had to
> > wait a
> > >> > minute while the console was black where the Maven was downloading
> the
> > >> > plugins in the background.
> > >> > Nobody was sure what happened that time, whether the console hanged
> or
> > >> the
> > >> > Cloud server hanged, or another issue happened with the network.
> > >> >
> > >> > I made a test and triggered the default lifecycle on Maven and I
> > >> realized
> > >> > that the cache was really very little, cca 12 MB.
> > >> > So this little cache in the container would save 50 seconds which is
> > the
> > >> > improvement we are discussing.
> > >> >
> > >> > From the use point of view, the user would use a new base image `
> > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > >> >
> > >> > There are multiple technical solutions (range of plugins, extra pom,
> > >> > internal Maven plugin versions, etc).
> > >> >
> > >> > We understood that the best idea would be to have the image with the
> > >> cache
> > >> > in new Docker images produced by Carloss Sanchez.
> > >> >
> > >> > We are discussing this topic in [1] but we do not have a consensus
> on
> > >> who
> > >> > will develop the Docker scripts and how.
> > >> >
> > >> > We can continue here and we can propose a solution.
> > >> >
> > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > >> >
> > >> >
> > >> > Cheers
> > >> > Tibor17
> > >> >
> > >>
> > > --
> > Sent from my phone
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Tibor Digana
Stephen, this is clear to me but we need to move ahead a bit.
Should I start the vote that we will have new project (
github.com/apache/maven-docker) for these purposes?
Or should we talk to Carloss and ask him if he would be fine with deploying
these images in DockerHub?

On Wed, Oct 30, 2019 at 3:21 PM Stephen Connolly <
[hidden email]> wrote:

> My script is for my cases. I'm just saying it's not rocket science, others
> are doing it already
>
> On Wed, 30 Oct 2019 at 13:31, Tibor Digana <[hidden email]> wrote:
>
> > Stephen, yeah something like you do in your scrip but it must not be a
> > personal owner. Even Carloss is person who makes this deployment to
> > DockerHub but his images are used by the entire world and we should
> decide
> > whether we would agree with him to have such images under his
> > responsibility or our responsibility as the Apache group. Then the script
> > would be official and we can cut the release of Maven and the Docker
> image
> > will be ready for the downloads together with the Maven distribution. So
> > the users will always know that it is consistent deployment and they
> > wouldn't expect that the image is missing for the new version.
> >
> > So here we should decide on who will be the deployer of these images with
> > the cache. And the technical solution is smaller problem I would say.
> >
> > On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> > [hidden email]> wrote:
> >
> > > On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]>
> > wrote:
> > >
> > > > It's the situation when you have maven plugins in repo and it means
> > that
> > > > all custom plugins/deps can be still downloaded as before.
> > > > Nothing exists like this in the world and we are talking about the
> > > > approaches.
> > > >
> > >
> > > Cough cough cough
> > >
> > >
> > >
> >
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae418ca3fd63c63038e896255c6fe8/Dockerfile#L50
> > >
> > > There are issues, but it does shave a bit of time... though we end up
> > > adding our common dependencies into the seed pom so that it is an
> > effective
> > > speed up
> > >
> > >
> > > >
> > > > I added Karl, Herve and Stephen in CC because we talked about this
> > issue
> > > > in ASF CON and Twitter.
> > > >
> > > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > >> Hi Tibor,
> > > >>
> > > >> It has two issues:
> > > >>
> > > >> 1. It will not be the right plugin versions in 90% of the cases
> > (except
> > > >> demo ;))
> > > >> 2. It will miss all custom plugins
> > > >>
> > > >> Now question is: what happens if you mount your local repo when
> > running
> > > >> docker? It works as expected. Means we could use a custom entrypoint
> > > >> printing a warning banner if it is not done probably instead of
> > > incrasing
> > > >> the image size without being sure to reach the original goal.
> > > >>
> > > >> Wdyt?
> > > >>
> > > >> Romain
> > > >>
> > > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <[hidden email]>
> a
> > > >> écrit :
> > > >>
> > > >> > If you use Docker images with Maven with no mapping of cache to
> the
> > > >> > volumes, you may notice that Maven downloads the plugins for the
> > build
> > > >> > lifecycle.
> > > >> >
> > > >> > This slows down the build because a lot of artifacts and plugins
> are
> > > >> > initially downloaded.
> > > >> > This takes 50 seconds which might be even longer than the
> productive
> > > >> build
> > > >> > itself (compiler, package, ...).
> > > >> >
> > > >> > We discussed this topic with Herve and Karl at the Apache CON 2019
> > the
> > > >> last
> > > >> > time.
> > > >> >
> > > >> > Sometime the presentations were funny because the audience had to
> > > wait a
> > > >> > minute while the console was black where the Maven was downloading
> > the
> > > >> > plugins in the background.
> > > >> > Nobody was sure what happened that time, whether the console
> hanged
> > or
> > > >> the
> > > >> > Cloud server hanged, or another issue happened with the network.
> > > >> >
> > > >> > I made a test and triggered the default lifecycle on Maven and I
> > > >> realized
> > > >> > that the cache was really very little, cca 12 MB.
> > > >> > So this little cache in the container would save 50 seconds which
> is
> > > the
> > > >> > improvement we are discussing.
> > > >> >
> > > >> > From the use point of view, the user would use a new base image `
> > > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > > >> >
> > > >> > There are multiple technical solutions (range of plugins, extra
> pom,
> > > >> > internal Maven plugin versions, etc).
> > > >> >
> > > >> > We understood that the best idea would be to have the image with
> the
> > > >> cache
> > > >> > in new Docker images produced by Carloss Sanchez.
> > > >> >
> > > >> > We are discussing this topic in [1] but we do not have a consensus
> > on
> > > >> who
> > > >> > will develop the Docker scripts and how.
> > > >> >
> > > >> > We can continue here and we can propose a solution.
> > > >> >
> > > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > > >> >
> > > >> >
> > > >> > Cheers
> > > >> > Tibor17
> > > >> >
> > > >>
> > > > --
> > > Sent from my phone
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Romain Manni-Bucau
In reply to this post by Tibor Digana
Ci servers have all cache so it is useless to make the docker pull slow
with useless versions to then download or not plugins IMHO.

Le mer. 30 oct. 2019 à 22:47, Hervé Boutemy <[hidden email]> a écrit :

> Le mercredi 30 octobre 2019 14:55:46 CET, vous avez écrit :
> > Agree we should publish with an asf account and make it part of the
> release
> > process.
> > That said I still fail to see how you can add a relevant cache. Maybe
> take
> > the time to review plugin version in a few asf projects (let say
> > maven-surefire, geronimo-openapi and spark) and check out if it works to
> > cache anything.
> if we choose one precise version of each plugin, we will never get the
> version
> used by a project: IMHO, we need to choose a range for each plugin
> https://maven.apache.org/plugins/
> Something like dependency:resolve-plugins should be able to do the job.
>
> With a range per plugin, I'd love to see the size of the initial local
> repository
> At least, using such an image would require minimal efforts from users,
> particularly on CI servers
>
> Regards,
>
> Hervé
>
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau>
> > | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > >
> >
> >
> > Le mer. 30 oct. 2019 à 14:31, Tibor Digana <[hidden email]> a
> >
> > écrit :
> > > Stephen, yeah something like you do in your scrip but it must not be a
> > > personal owner. Even Carloss is person who makes this deployment to
> > > DockerHub but his images are used by the entire world and we should
> decide
> > > whether we would agree with him to have such images under his
> > > responsibility or our responsibility as the Apache group. Then the
> script
> > > would be official and we can cut the release of Maven and the Docker
> image
> > > will be ready for the downloads together with the Maven distribution.
> So
> > > the users will always know that it is consistent deployment and they
> > > wouldn't expect that the image is missing for the new version.
> > >
> > > So here we should decide on who will be the deployer of these images
> with
> > > the cache. And the technical solution is smaller problem I would say.
> > >
> > > On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> > >
> > > [hidden email]> wrote:
> > > > On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]>
> > >
> > > wrote:
> > > > > It's the situation when you have maven plugins in repo and it means
> > >
> > > that
> > >
> > > > > all custom plugins/deps can be still downloaded as before.
> > > > > Nothing exists like this in the world and we are talking about the
> > > > > approaches.
> > > >
> > > > Cough cough cough
> > >
> > >
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae41
> > > 8ca3fd63c63038e896255c6fe8/Dockerfile#L50>
> > > > There are issues, but it does shave a bit of time... though we end up
> > > > adding our common dependencies into the seed pom so that it is an
> > >
> > > effective
> > >
> > > > speed up
> > > >
> > > > > I added Karl, Herve and Stephen in CC because we talked about this
> > >
> > > issue
> > >
> > > > > in ASF CON and Twitter.
> > > > >
> > > > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > > >
> > > > [hidden email]>
> > > >
> > > > > wrote:
> > > > >> Hi Tibor,
> > > > >>
> > > > >> It has two issues:
> > > > >>
> > > > >> 1. It will not be the right plugin versions in 90% of the cases
> > >
> > > (except
> > >
> > > > >> demo ;))
> > > > >> 2. It will miss all custom plugins
> > > > >>
> > > > >> Now question is: what happens if you mount your local repo when
> > >
> > > running
> > >
> > > > >> docker? It works as expected. Means we could use a custom
> entrypoint
> > > > >> printing a warning banner if it is not done probably instead of
> > > >
> > > > incrasing
> > > >
> > > > >> the image size without being sure to reach the original goal.
> > > > >>
> > > > >> Wdyt?
> > > > >>
> > > > >> Romain
> > > > >>
> > > > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <
> [hidden email]> a
> > > > >>
> > > > >> écrit :
> > > > >> > If you use Docker images with Maven with no mapping of cache to
> the
> > > > >> > volumes, you may notice that Maven downloads the plugins for the
> > >
> > > build
> > >
> > > > >> > lifecycle.
> > > > >> >
> > > > >> > This slows down the build because a lot of artifacts and plugins
> > > > >> > are
> > > > >> > initially downloaded.
> > > > >> > This takes 50 seconds which might be even longer than the
> > > > >> > productive
> > > > >>
> > > > >> build
> > > > >>
> > > > >> > itself (compiler, package, ...).
> > > > >> >
> > > > >> > We discussed this topic with Herve and Karl at the Apache CON
> 2019
> > >
> > > the
> > >
> > > > >> last
> > > > >>
> > > > >> > time.
> > > > >> >
> > > > >> > Sometime the presentations were funny because the audience had
> to
> > > >
> > > > wait a
> > > >
> > > > >> > minute while the console was black where the Maven was
> downloading
> > >
> > > the
> > >
> > > > >> > plugins in the background.
> > > > >> > Nobody was sure what happened that time, whether the console
> hanged
> > >
> > > or
> > >
> > > > >> the
> > > > >>
> > > > >> > Cloud server hanged, or another issue happened with the network.
> > > > >> >
> > > > >> > I made a test and triggered the default lifecycle on Maven and I
> > > > >>
> > > > >> realized
> > > > >>
> > > > >> > that the cache was really very little, cca 12 MB.
> > > > >> > So this little cache in the container would save 50 seconds
> which
> > > > >> > is
> > > >
> > > > the
> > > >
> > > > >> > improvement we are discussing.
> > > > >> >
> > > > >> > From the use point of view, the user would use a new base image
> `
> > > > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > > > >> >
> > > > >> > There are multiple technical solutions (range of plugins, extra
> > > > >> > pom,
> > > > >> > internal Maven plugin versions, etc).
> > > > >> >
> > > > >> > We understood that the best idea would be to have the image with
> > > > >> > the
> > > > >>
> > > > >> cache
> > > > >>
> > > > >> > in new Docker images produced by Carloss Sanchez.
> > > > >> >
> > > > >> > We are discussing this topic in [1] but we do not have a
> consensus
> > >
> > > on
> > >
> > > > >> who
> > > > >>
> > > > >> > will develop the Docker scripts and how.
> > > > >> >
> > > > >> > We can continue here and we can propose a solution.
> > > > >> >
> > > > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > > > >> >
> > > > >> >
> > > > >> > Cheers
> > > > >> > Tibor17
> > > > >
> > > > > --
> > > >
> > > > Sent from my phone
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker image with initialized local cache saves 50 seconds in startup

Tibor Digana
Romain, of course we know this but there are two groups of users.
First those who use the Maven in docker without CI.
And second of those who integrate the docker container with the CI and they
use the volumes.

The first group of people may freely use the image we are discussing here
because this makes the default build lifecycle faster for them.
For us it means that we will not be seens as "slow Maven", you know what i
mean.

The second group already use the maven images from Carloss.

On Wed, Oct 30, 2019 at 11:12 PM Romain Manni-Bucau <[hidden email]>
wrote:

> Ci servers have all cache so it is useless to make the docker pull slow
> with useless versions to then download or not plugins IMHO.
>
> Le mer. 30 oct. 2019 à 22:47, Hervé Boutemy <[hidden email]> a écrit
> :
>
> > Le mercredi 30 octobre 2019 14:55:46 CET, vous avez écrit :
> > > Agree we should publish with an asf account and make it part of the
> > release
> > > process.
> > > That said I still fail to see how you can add a relevant cache. Maybe
> > take
> > > the time to review plugin version in a few asf projects (let say
> > > maven-surefire, geronimo-openapi and spark) and check out if it works
> to
> > > cache anything.
> > if we choose one precise version of each plugin, we will never get the
> > version
> > used by a project: IMHO, we need to choose a range for each plugin
> > https://maven.apache.org/plugins/
> > Something like dependency:resolve-plugins should be able to do the job.
> >
> > With a range per plugin, I'd love to see the size of the initial local
> > repository
> > At least, using such an image would require minimal efforts from users,
> > particularly on CI servers
> >
> > Regards,
> >
> > Hervé
> >
> > >
> > > Romain Manni-Bucau
> > > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > > <https://rmannibucau.metawerx.net/> | Old Blog
> > > <http://rmannibucau.wordpress.com> | Github <
> > https://github.com/rmannibucau>
> > > | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > > <
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > > >
> > >
> > >
> > > Le mer. 30 oct. 2019 à 14:31, Tibor Digana <[hidden email]> a
> > >
> > > écrit :
> > > > Stephen, yeah something like you do in your scrip but it must not be
> a
> > > > personal owner. Even Carloss is person who makes this deployment to
> > > > DockerHub but his images are used by the entire world and we should
> > decide
> > > > whether we would agree with him to have such images under his
> > > > responsibility or our responsibility as the Apache group. Then the
> > script
> > > > would be official and we can cut the release of Maven and the Docker
> > image
> > > > will be ready for the downloads together with the Maven distribution.
> > So
> > > > the users will always know that it is consistent deployment and they
> > > > wouldn't expect that the image is missing for the new version.
> > > >
> > > > So here we should decide on who will be the deployer of these images
> > with
> > > > the cache. And the technical solution is smaller problem I would say.
> > > >
> > > > On Wed, Oct 30, 2019 at 10:28 AM Stephen Connolly <
> > > >
> > > > [hidden email]> wrote:
> > > > > On Wed 30 Oct 2019 at 08:21, Tibor Digana <[hidden email]>
> > > >
> > > > wrote:
> > > > > > It's the situation when you have maven plugins in repo and it
> means
> > > >
> > > > that
> > > >
> > > > > > all custom plugins/deps can be still downloaded as before.
> > > > > > Nothing exists like this in the world and we are talking about
> the
> > > > > > approaches.
> > > > >
> > > > > Cough cough cough
> > > >
> > > >
> >
> https://github.com/stephenc/docker-git-java8-maven-vim/blob/168b9968deae41
> > > > 8ca3fd63c63038e896255c6fe8/Dockerfile#L50>
> > > > > There are issues, but it does shave a bit of time... though we end
> up
> > > > > adding our common dependencies into the seed pom so that it is an
> > > >
> > > > effective
> > > >
> > > > > speed up
> > > > >
> > > > > > I added Karl, Herve and Stephen in CC because we talked about
> this
> > > >
> > > > issue
> > > >
> > > > > > in ASF CON and Twitter.
> > > > > >
> > > > > > On Wed, Oct 30, 2019 at 6:36 AM Romain Manni-Bucau <
> > > > >
> > > > > [hidden email]>
> > > > >
> > > > > > wrote:
> > > > > >> Hi Tibor,
> > > > > >>
> > > > > >> It has two issues:
> > > > > >>
> > > > > >> 1. It will not be the right plugin versions in 90% of the cases
> > > >
> > > > (except
> > > >
> > > > > >> demo ;))
> > > > > >> 2. It will miss all custom plugins
> > > > > >>
> > > > > >> Now question is: what happens if you mount your local repo when
> > > >
> > > > running
> > > >
> > > > > >> docker? It works as expected. Means we could use a custom
> > entrypoint
> > > > > >> printing a warning banner if it is not done probably instead of
> > > > >
> > > > > incrasing
> > > > >
> > > > > >> the image size without being sure to reach the original goal.
> > > > > >>
> > > > > >> Wdyt?
> > > > > >>
> > > > > >> Romain
> > > > > >>
> > > > > >> Le mer. 30 oct. 2019 à 02:03, Tibor Digana <
> > [hidden email]> a
> > > > > >>
> > > > > >> écrit :
> > > > > >> > If you use Docker images with Maven with no mapping of cache
> to
> > the
> > > > > >> > volumes, you may notice that Maven downloads the plugins for
> the
> > > >
> > > > build
> > > >
> > > > > >> > lifecycle.
> > > > > >> >
> > > > > >> > This slows down the build because a lot of artifacts and
> plugins
> > > > > >> > are
> > > > > >> > initially downloaded.
> > > > > >> > This takes 50 seconds which might be even longer than the
> > > > > >> > productive
> > > > > >>
> > > > > >> build
> > > > > >>
> > > > > >> > itself (compiler, package, ...).
> > > > > >> >
> > > > > >> > We discussed this topic with Herve and Karl at the Apache CON
> > 2019
> > > >
> > > > the
> > > >
> > > > > >> last
> > > > > >>
> > > > > >> > time.
> > > > > >> >
> > > > > >> > Sometime the presentations were funny because the audience had
> > to
> > > > >
> > > > > wait a
> > > > >
> > > > > >> > minute while the console was black where the Maven was
> > downloading
> > > >
> > > > the
> > > >
> > > > > >> > plugins in the background.
> > > > > >> > Nobody was sure what happened that time, whether the console
> > hanged
> > > >
> > > > or
> > > >
> > > > > >> the
> > > > > >>
> > > > > >> > Cloud server hanged, or another issue happened with the
> network.
> > > > > >> >
> > > > > >> > I made a test and triggered the default lifecycle on Maven
> and I
> > > > > >>
> > > > > >> realized
> > > > > >>
> > > > > >> > that the cache was really very little, cca 12 MB.
> > > > > >> > So this little cache in the container would save 50 seconds
> > which
> > > > > >> > is
> > > > >
> > > > > the
> > > > >
> > > > > >> > improvement we are discussing.
> > > > > >> >
> > > > > >> > From the use point of view, the user would use a new base
> image
> > `
> > > > > >> > 3.6.2-jdk-14-prefetched` which is my idea.
> > > > > >> >
> > > > > >> > There are multiple technical solutions (range of plugins,
> extra
> > > > > >> > pom,
> > > > > >> > internal Maven plugin versions, etc).
> > > > > >> >
> > > > > >> > We understood that the best idea would be to have the image
> with
> > > > > >> > the
> > > > > >>
> > > > > >> cache
> > > > > >>
> > > > > >> > in new Docker images produced by Carloss Sanchez.
> > > > > >> >
> > > > > >> > We are discussing this topic in [1] but we do not have a
> > consensus
> > > >
> > > > on
> > > >
> > > > > >> who
> > > > > >>
> > > > > >> > will develop the Docker scripts and how.
> > > > > >> >
> > > > > >> > We can continue here and we can propose a solution.
> > > > > >> >
> > > > > >> > [1] https://github.com/carlossg/docker-maven/issues/130
> > > > > >> >
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Tibor17
> > > > > >
> > > > > > --
> > > > >
> > > > > Sent from my phone
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>