Re: parallelise not overlapping tasks

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

rfscholte
Hi Romain,

I don't think this is something for the pom.
It is the plugin which should know if it has effect on the build.
Or maybe more global: if we could define for the plugin what is input and  
what is output, then it should be possible to decide which parts could run  
in parallel.

thanks,
Robert

On Fri, 19 Jan 2018 14:20:50 +0100, Romain Manni-Bucau  
<[hidden email]> wrote:

> Hi guys,
>
> there is no way to parallelize not conflicting tasks for a same phase at
> the moment right? Any way it gets under the radar?
>
> A common example is to run all code analyzis concurrently (findbugs, pmd,
> checkstyle, ...) at the same time without waiting for one then the other
> etc since all can be very long.
>
> It could be nice an fancy to define part of the reactor parallelisablity  
> in
> the pom, like:
>
> <plan>
>     <parallel>
>         <phase>process-sources</phase>
>         <executions>
>             <execution>
>               <groupId>...</groupId>
>               <artifactId>...</artifactId>
>               <version>...</version>
>               <id>...</id>
>             <execution>
>             <execution>
>               <groupId>...</groupId>
>               <artifactId>...</artifactId>
>               <version>...</version>
>               <id>...</id>
>             <execution>
>         </executions>
>     </parallel>
> </plan>
>
> anything to enhance it?
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github  
> <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Tibor Digana
I am facing very slow Maven reporting/site.
I have optimized JavaDoc and avoided the javadoc plugin downloading
sources, Java API which has improved the performance but still Site takes 8
minutes at least to complete.
It would be worth to mark some reporting plugins running in parallel in
aggregator POM.

On Fri, Jan 19, 2018 at 2:20 PM, Romain Manni-Bucau <[hidden email]>
wrote:

> Hi guys,
>
> there is no way to parallelize not conflicting tasks for a same phase at
> the moment right? Any way it gets under the radar?
>
> A common example is to run all code analyzis concurrently (findbugs, pmd,
> checkstyle, ...) at the same time without waiting for one then the other
> etc since all can be very long.
>
> It could be nice an fancy to define part of the reactor parallelisablity in
> the pom, like:
>
> <plan>
>     <parallel>
>         <phase>process-sources</phase>
>         <executions>
>             <execution>
>               <groupId>...</groupId>
>               <artifactId>...</artifactId>
>               <version>...</version>
>               <id>...</id>
>             <execution>
>             <execution>
>               <groupId>...</groupId>
>               <artifactId>...</artifactId>
>               <version>...</version>
>               <id>...</id>
>             <execution>
>         </executions>
>     </parallel>
> </plan>
>
> anything to enhance it?
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/
> rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau>
>
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Mickael Istria-2
I think there is a difference between scheduling tasks (one is after the
other) and assuming that 2 tasks that are ready can run in parallel.
Adding scheduling data would be helpful and get rid of the concept of
phases, for sure; but we also -and more importantly- need the existing
mojos to be audited and be able to declare whether they're thread-safe
before running them in parallel, and this sounds like a gigantic work.
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Mickael Istria-2
Ok, my bad, thanks for the hint.
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Romain Manni-Bucau
In reply to this post by rfscholte
Using the scm is not enough or it is for single module projects.

You have to have a graph of dependencies (inputs/outputs) and save each
task state in target to have incremental support

But please note incremental build is != parallel build at mojo level.

This last one is easy to do and a quick win IMHO

Le sam. 8 déc. 2018 00:50, Tibor Digana <[hidden email]> a écrit :

> In my projects, the most plugins use single execution.
> External projects also have this kind of principle.
> Thus we should have a look in those possibilities where the most plugins
> can gain the performance.
> Usually the compiler and tests take long.
> I know that maven-compiler-plugin:3.8.1 will be incremental which is good
> of course but we should perhaps continue with gaining the build
> performance.
> If somebody has an idea on how to develop a compiler which partially
> compiles a module depending on SCM changes, feel free to bring it to our
> mailing list. The same with tests where the set of tests is changed
> depending on SCM changes.
>
> Cheers
> Tibor
>
>
> On Fri, Jan 19, 2018 at 2:21 PM Romain Manni-Bucau <[hidden email]>
> wrote:
>
> > Hi guys,
> >
> > there is no way to parallelize not conflicting tasks for a same phase at
> > the moment right? Any way it gets under the radar?
> >
> > A common example is to run all code analyzis concurrently (findbugs, pmd,
> > checkstyle, ...) at the same time without waiting for one then the other
> > etc since all can be very long.
> >
> > It could be nice an fancy to define part of the reactor parallelisablity
> in
> > the pom, like:
> >
> > <plan>
> >     <parallel>
> >         <phase>process-sources</phase>
> >         <executions>
> >             <execution>
> >               <groupId>...</groupId>
> >               <artifactId>...</artifactId>
> >               <version>...</version>
> >               <id>...</id>
> >             <execution>
> >             <execution>
> >               <groupId>...</groupId>
> >               <artifactId>...</artifactId>
> >               <version>...</version>
> >               <id>...</id>
> >             <execution>
> >         </executions>
> >     </parallel>
> > </plan>
> >
> > anything to enhance it?
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github <
> > https://github.com/rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau>
> >
>