Why no mvn daemon?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Why no mvn daemon?

Romain Manni-Bucau
Hi guys,

Wonder why maven doesnt have any daemon and din't find anything satisfying
to answer this question with our preferred search engine.

Idea would be - a bit like gradle I guess - to cache computations and
classloaders.

Asking cause:
- computing the reactor could be long on big projects and if the pom didnt'
change between 2 builds it is useless
- creating a classloader can be long but "filling" it - loadClass -  is
really longer for several plugins - big classloaders or dynamic languages -
and doing it again and again is slow

So the idea would be to run maven as a long running process (transparently
through a -D in mvn opts if possible) and:
- cache all is cacheable like the reactor etc while pom last modified date
didnt change (OS needs to support it or natives could be used for OS where
the JVM doesnt help)
- cache classloaders and reuse them when re-executing a mojo
- the communication between "mvn" which would then be a client to the
daemon could just use a socket

Of course if the pom changed the cache would just be invaildated but we
would still gain the maven boot itself and we can still keep the
classloader cache for plugins while their configuration is the sae
(dependencies).

Last thing missing in my description is two more configurations: a -Dxxx to
kill the daemon and one to skip it for one build.

Any technical/politics reason I miss to not have it?

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Tamás Cservenák
Maven Shell?

https://github.com/jdillon/mvnsh

On Thu, Mar 3, 2016 at 2:13 PM Romain Manni-Bucau <[hidden email]>
wrote:

> Hi guys,
>
> Wonder why maven doesnt have any daemon and din't find anything satisfying
> to answer this question with our preferred search engine.
>
> Idea would be - a bit like gradle I guess - to cache computations and
> classloaders.
>
> Asking cause:
> - computing the reactor could be long on big projects and if the pom didnt'
> change between 2 builds it is useless
> - creating a classloader can be long but "filling" it - loadClass -  is
> really longer for several plugins - big classloaders or dynamic languages -
> and doing it again and again is slow
>
> So the idea would be to run maven as a long running process (transparently
> through a -D in mvn opts if possible) and:
> - cache all is cacheable like the reactor etc while pom last modified date
> didnt change (OS needs to support it or natives could be used for OS where
> the JVM doesnt help)
> - cache classloaders and reuse them when re-executing a mojo
> - the communication between "mvn" which would then be a client to the
> daemon could just use a socket
>
> Of course if the pom changed the cache would just be invaildated but we
> would still gain the maven boot itself and we can still keep the
> classloader cache for plugins while their configuration is the sae
> (dependencies).
>
> Last thing missing in my description is two more configurations: a -Dxxx to
> kill the daemon and one to skip it for one build.
>
> Any technical/politics reason I miss to not have it?
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> <http://www.tomitribe.com>
>
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Romain Manni-Bucau
Good catch but something not changing user shell would be great and part of
maven distribution would be awesome.


Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>

2016-03-03 14:42 GMT+01:00 Tamás Cservenák <[hidden email]>:

> Maven Shell?
>
> https://github.com/jdillon/mvnsh
>
> On Thu, Mar 3, 2016 at 2:13 PM Romain Manni-Bucau <[hidden email]>
> wrote:
>
> > Hi guys,
> >
> > Wonder why maven doesnt have any daemon and din't find anything
> satisfying
> > to answer this question with our preferred search engine.
> >
> > Idea would be - a bit like gradle I guess - to cache computations and
> > classloaders.
> >
> > Asking cause:
> > - computing the reactor could be long on big projects and if the pom
> didnt'
> > change between 2 builds it is useless
> > - creating a classloader can be long but "filling" it - loadClass -  is
> > really longer for several plugins - big classloaders or dynamic
> languages -
> > and doing it again and again is slow
> >
> > So the idea would be to run maven as a long running process
> (transparently
> > through a -D in mvn opts if possible) and:
> > - cache all is cacheable like the reactor etc while pom last modified
> date
> > didnt change (OS needs to support it or natives could be used for OS
> where
> > the JVM doesnt help)
> > - cache classloaders and reuse them when re-executing a mojo
> > - the communication between "mvn" which would then be a client to the
> > daemon could just use a socket
> >
> > Of course if the pom changed the cache would just be invaildated but we
> > would still gain the maven boot itself and we can still keep the
> > classloader cache for plugins while their configuration is the sae
> > (dependencies).
> >
> > Last thing missing in my description is two more configurations: a -Dxxx
> to
> > kill the daemon and one to skip it for one build.
> >
> > Any technical/politics reason I miss to not have it?
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > <http://rmannibucau.wordpress.com> | Github <
> > https://github.com/rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> > <http://www.tomitribe.com>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Jason Dillon-2
In reply to this post by Tamás Cservenák
On March 3, 2016 at 5:42:56 AM, Tamás Cservenák ([hidden email]) wrote:
Maven Shell? 

https://github.com/jdillon/mvnsh 

Does anyone actually use this?

—jason

Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Jeff Jensen
Some months ago I was looking for something like this.  Due to this thread,
I will soon set it up and try it.  (Thanks for sharing Tamás!)


On Sat, Mar 5, 2016 at 10:31 PM, Jason Dillon <[hidden email]>
wrote:

> On March 3, 2016 at 5:42:56 AM, Tamás Cservenák ([hidden email])
> wrote:
> Maven Shell?
>
> https://github.com/jdillon/mvnsh
>
> Does anyone actually use this?
>
> —jason
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Jeff Jensen
FYI Just cloned and tried build but Jansi 1.2 is not in Central: "Could not
find artifact org.fusesource.jansi:jansi:jar:1.2".
http://search.maven.org/#search|gav|1|g%3A%22org.fusesource.jansi%22%20AND%20a%3A%22jansi%22

I noted the issue tracker no longer exists so replying here to start.

jansi is not declared in the POMs, so assuming transitive.  Found it is
only referenced by
"mvnsh-commands/mvnsh-maven/src/main/java/org/sonatype/maven/shell/commands/maven/ColorizingStream.java"
in mvnsh-commands/mvnsh-maven.
I'm out of time to look further at the moment.

Jason, if you have a built version, do you mind adding it as a download to
the release files?
Do you have interest in supporting further or prefer not to continue with
it?


On Sat, Mar 5, 2016 at 10:40 PM, Jeff Jensen <
[hidden email]> wrote:

> Some months ago I was looking for something like this.  Due to this
> thread, I will soon set it up and try it.  (Thanks for sharing Tamás!)
>
>
> On Sat, Mar 5, 2016 at 10:31 PM, Jason Dillon <[hidden email]>
> wrote:
>
>> On March 3, 2016 at 5:42:56 AM, Tamás Cservenák ([hidden email])
>> wrote:
>> Maven Shell?
>>
>> https://github.com/jdillon/mvnsh
>>
>> Does anyone actually use this?
>>
>> —jason
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Jason Dillon-2
On March 8, 2016 at 7:45:53 AM, Jeff Jensen ([hidden email]) wrote:
FYI Just cloned and tried build but Jansi 1.2 is not in Central: "Could not 
find artifact org.fusesource.jansi:jansi:jar:1.2". 
http://search.maven.org/#search|gav|1|g%3A%22org.fusesource.jansi%22%20AND%20a%3A%22jansi%22 

I noted the issue tracker no longer exists so replying here to start. 

jansi is not declared in the POMs, so assuming transitive. Found it is 
only referenced by 
"mvnsh-commands/mvnsh-maven/src/main/java/org/sonatype/maven/shell/commands/maven/ColorizingStream.java" 
in mvnsh-commands/mvnsh-maven. 
I'm out of time to look further at the moment. 
I’m aware of this.  I took a brief look at upgrading to the latest jline but its a bit more work.  There is another sonatype version of jline that may work better in the short-term until moving to the official jline2 stuff can be done.



Jason, if you have a built version, do you mind adding it as a download to 
the release files? 
I can make a binary of this, though I do plan on fixing it up so that folks can build it in the near future.


Do you have interest in supporting further or prefer not to continue with 
it? 
I’ve recently put in a bit more effort to bring this stuff up to speed w/changes to maven.  I’m not sure I’ll be spending a ton of effort around it but I’m not sure how much effort is required.  Certainly though if folks are actually using this then I’ll do what I can to support its continued use.

There is another maven shell which may have more legs in terms of longer term community support, but I’m not yet sure where that is in terms of public availability.

—jason
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Jason Dillon-2
Jason, if you have a built version, do you mind adding it as a download to 
the release files? 
I can make a binary of this, though I do plan on fixing it up so that folks can build it in the near future.

Build up here for the moment:

https://www.dropbox.com/sh/yvt1g43r2pr2scj/AADqM__VhJaFf_x57OiUEZXva?dl=0

gshell:master should be buildable with just central now, dangling ref to older version of jline for classifer=tests which was unused and polluting the build dependencies.

—jason


Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Jeff Jensen
In reply to this post by Jason Dillon-2
On Wed, Mar 9, 2016 at 3:59 PM, Jason Dillon <[hidden email]> wrote:

> On March 8, 2016 at 7:45:53 AM, Jeff Jensen (
> [hidden email]) wrote:
>
> FYI Just cloned and tried build but Jansi 1.2 is not in Central: "Could not
>
> find artifact org.fusesource.jansi:jansi:jar:1.2".
>
> http://search.maven.org/#search|gav|1|g%3A%22org.fusesource.jansi%22%20AND%20a%3A%22jansi%22
> <http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.fusesource.jansi%22%20AND%20a%3A%22jansi%22>
>
>
> I noted the issue tracker no longer exists so replying here to start.
>
> jansi is not declared in the POMs, so assuming transitive. Found it is
> only referenced by
>
> "mvnsh-commands/mvnsh-maven/src/main/java/org/sonatype/maven/shell/commands/maven/ColorizingStream.java"
>
> in mvnsh-commands/mvnsh-maven.
> I'm out of time to look further at the moment.
>
> I’m aware of this.  I took a brief look at upgrading to the latest jline
> but its a bit more work.  There is another sonatype version of jline that
> may work better in the short-term until moving to the official jline2 stuff
> can be done.
>
>
> Jason, if you have a built version, do you mind adding it as a download to
>
> the release files?
>
> I can make a binary of this, though I do plan on fixing it up so that
> folks can build it in the near future.
>
Thanks, it's nice to simply use vs build, from my "user perspective" :-)



> Do you have interest in supporting further or prefer not to continue with
> it?
>
> I’ve recently put in a bit more effort to bring this stuff up to speed
> w/changes to maven.  I’m not sure I’ll be spending a ton of effort around
> it but I’m not sure how much effort is required.  Certainly though if folks
> are actually using this then I’ll do what I can to support its continued
> use.
>
Thanks - my curiosity was around if this was "abandon-ware" or you may make
it buildable again and do a few updates over the years... if not, then I
would keep looking for one!


There is another maven shell which may have more legs in terms of longer
> term community support, but I’m not yet sure where that is in terms of
> public availability.
>

Interesting, is it an internal Sonatype one or from somewhere else?
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Jeff Jensen
In reply to this post by Jason Dillon-2
Thanks!  It's running just fine.  :-)  It's very cool.  Really nice
directory traversal and completion, colors, and commands/features I don't
know yet!

Very interesting timing diffs (for each casual test, I ran the command for
each multiple times to seed infra caches):
 * on some asciidoc gen with "mvn generate-resources", it was about the
same duration as CLI but after running each about 10 times, mvnshell saved
about 20% consistently (possibly due to JIT? besides directory traversal,
this was the first things I did).  This was my key use case for wanting a
"mvn server" - handling situations like this with repeated runs (asciidoc,
site, etc.).

 * on a simple "mvn clean", mvnshell was about 2x faster than CLI.

 * on a small module build, "mvn install" was about 20% faster over CLI
(after a mvn clean for each).

I look forward to trying more things.

Nice to have, thank you Jason!


On Wed, Mar 9, 2016 at 4:17 PM, Jason Dillon <[hidden email]> wrote:

> Jason, if you have a built version, do you mind adding it as a download to
>
> the release files?
>
> I can make a binary of this, though I do plan on fixing it up so that
> folks can build it in the near future.
>
> Build up here for the moment:
>
> https://www.dropbox.com/sh/yvt1g43r2pr2scj/AADqM__VhJaFf_x57OiUEZXva?dl=0
>
> gshell:master should be buildable with just central now, dangling ref to
> older version of jline for classifer=tests which was unused and polluting
> the build dependencies.
>
> —jason
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Hervé BOUTEMY
In reply to this post by Romain Manni-Bucau
I think mvnd is not so well known [1]: I did never test it, I should probably
do...

and as incremental compilation discussion have started recently, we'll need to
have a global discussion on evaluating and eventually mixing differents ways of
improving build performance

when evaluating, I think we'll need to keep in mind one key aspect: from a
user perspective, which deployment and usage complexities does every solution
bring for which performance improvement?

I feel that each solution that brings more expected performance improvement
comes with at least a deployment complexity, perhaps sometimes use complexity
also (per-project configuration, ...)

but currently mvnd is the only free solution publicly available: I should
definitely test it to better know it when we'll have more in depth discussions
in the future

Please keep posting your factual returns on experience: that's definitely
useful

Regards,

Hervé

[1] https://github.com/mvndaemon/mvnd

Le mardi 30 mars 2021, 22:32:37 CEST Romain Manni-Bucau a écrit :

> Le mar. 30 mars 2021 à 22:16, Jochen Wiedmann <[hidden email]> a
>
> écrit :
> > On Tue, Mar 30, 2021 at 7:58 PM Benjamin Marwell <[hidden email]>
> >
> > wrote:
> > > Other than that, I use mvnd at work and we never had problems on any
> > > project. We consistently save time without the need of adding the -T
> > > parameter manually. Another big question is probably: is there enough
> > > community demand to merge it into core?
> > >
> > > Tbh, I expected more mails on this thread.
> >
> > The question for me: Who's launching this mvnd? I wouldn't want to
> > have a dedicated server for that. However, if (for example) m2e would
> > launch it automatically when I open Eclipse, and perform a shutdown,
> > when Eclipse closes, then I'd be perfectly happy to use it.
>
> mvnd. As soon as env changes (you launch it from another project) daemon is
> killed.
> Indeed you can kill it manually too - which makes it saner than idea remote
> maven server or ide maven server which cant be killed generally and consume
> a lot of mem while ide is on.
>
> Indeed you can bind mvnd --stop on eclipse shutdown which would behave as
> you expect but working on the cli without any ide specific integ is maven
> scope, not idea integration IMO (even if a few code enable it but guess it
> is legacy now thanks the ioc mvn has).
>
> Feel free to give a try to mvnd, works very well everywhere ;).
>
> > Jochen
> >
> >
> >
> > --
> >
> > Look, that's why there's rules, understand? So that you think before
> > you break 'em.
> >
> >     -- (Terry Pratchett, Thief of Time)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Why no mvn daemon?

Romain Manni-Bucau
Le mer. 31 mars 2021 à 08:17, Hervé BOUTEMY <[hidden email]> a
écrit :

> I think mvnd is not so well known [1]: I did never test it, I should
> probably
> do...
>
> and as incremental compilation discussion have started recently, we'll
> need to
> have a global discussion on evaluating and eventually mixing differents
> ways of
> improving build performance
>

+100000, mvnd is awesome but solves the JVM pitfalls mainly, incremental
support is per mojo to be optimized and both would be awesome.
Basically optimizing the JVM + the runtime at the same time.


>
> when evaluating, I think we'll need to keep in mind one key aspect: from a
> user perspective, which deployment and usage complexities does every
> solution
> bring for which performance improvement?
>

I think the main one maven should target is the "local" env (this is why
mvnd fits so well) - by opposition of a remote server as gradle enterprise
one which requires some infra, typically maven does not produce nexus byt
stick on local software and I think it is sane to target it primarly.


>
> I feel that each solution that brings more expected performance
> improvement
> comes with at least a deployment complexity, perhaps sometimes use
> complexity
> also (per-project configuration, ...)
>

Hmm, here the only drawback of mvnd is to have the daemon running and
consume the related memory but it is also why you use it so it is almost
free for me - at least in usage. It is not like having to setup a server
and configure the client in maven since launching mvnd instead of mvn you
launch the mvnd client + the daemon if not already running so in terms of
user experience you can alias mvn=mvnd and you will not notice the
difference if you have enough memory.
My typicall pitfall with 16G of ram is to have idea+chrome+minikube
launched and mvd preventing a graalvm build (not enough memory)...but it is
not mainstream too as memory consumption.
Also there is a ticket about improving mvnd daemon auto-monitoring to kill
itself if machine memory is too low and it is not used (
https://github.com/mvndaemon/mvnd/issues/364) so this pitfall can disappear
too.


>
> but currently mvnd is the only free solution publicly available: I should
> definitely test it to better know it when we'll have more in depth
> discussions
> in the future
>
> Please keep posting your factual returns on experience: that's definitely
> useful
>
> Regards,
>
> Hervé
>
> [1] https://github.com/mvndaemon/mvnd
>
> Le mardi 30 mars 2021, 22:32:37 CEST Romain Manni-Bucau a écrit :
> > Le mar. 30 mars 2021 à 22:16, Jochen Wiedmann <[hidden email]>
> a
> >
> > écrit :
> > > On Tue, Mar 30, 2021 at 7:58 PM Benjamin Marwell <[hidden email]>
> > >
> > > wrote:
> > > > Other than that, I use mvnd at work and we never had problems on any
> > > > project. We consistently save time without the need of adding the -T
> > > > parameter manually. Another big question is probably: is there enough
> > > > community demand to merge it into core?
> > > >
> > > > Tbh, I expected more mails on this thread.
> > >
> > > The question for me: Who's launching this mvnd? I wouldn't want to
> > > have a dedicated server for that. However, if (for example) m2e would
> > > launch it automatically when I open Eclipse, and perform a shutdown,
> > > when Eclipse closes, then I'd be perfectly happy to use it.
> >
> > mvnd. As soon as env changes (you launch it from another project) daemon
> is
> > killed.
> > Indeed you can kill it manually too - which makes it saner than idea
> remote
> > maven server or ide maven server which cant be killed generally and
> consume
> > a lot of mem while ide is on.
> >
> > Indeed you can bind mvnd --stop on eclipse shutdown which would behave as
> > you expect but working on the cli without any ide specific integ is maven
> > scope, not idea integration IMO (even if a few code enable it but guess
> it
> > is legacy now thanks the ioc mvn has).
> >
> > Feel free to give a try to mvnd, works very well everywhere ;).
> >
> > > Jochen
> > >
> > >
> > >
> > > --
> > >
> > > Look, that's why there's rules, understand? So that you think before
> > > you break 'em.
> > >
> > >     -- (Terry Pratchett, Thief of Time)
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>