parallelise not overlapping tasks

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

parallelise not overlapping tasks

Romain Manni-Bucau
Hi guys,

there is no way to parallelize not conflicting tasks for a same phase at
the moment right? Any way it gets under the radar?

A common example is to run all code analyzis concurrently (findbugs, pmd,
checkstyle, ...) at the same time without waiting for one then the other
etc since all can be very long.

It could be nice an fancy to define part of the reactor parallelisablity in
the pom, like:

<plan>
    <parallel>
        <phase>process-sources</phase>
        <executions>
            <execution>
              <groupId>...</groupId>
              <artifactId>...</artifactId>
              <version>...</version>
              <id>...</id>
            <execution>
            <execution>
              <groupId>...</groupId>
              <artifactId>...</artifactId>
              <version>...</version>
              <id>...</id>
            <execution>
        </executions>
    </parallel>
</plan>

anything to enhance it?

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau>
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Romain Manni-Bucau
Hey guys,

any way we move that topic forward beginning of next year?

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le lun. 22 janv. 2018 à 01:04, Tibor Digana <[hidden email]> a
écrit :

> I am facing very slow Maven reporting/site.
> I have optimized JavaDoc and avoided the javadoc plugin downloading
> sources, Java API which has improved the performance but still Site takes 8
> minutes at least to complete.
> It would be worth to mark some reporting plugins running in parallel in
> aggregator POM.
>
> On Fri, Jan 19, 2018 at 2:20 PM, Romain Manni-Bucau <[hidden email]
> >
> wrote:
>
> > Hi guys,
> >
> > there is no way to parallelize not conflicting tasks for a same phase at
> > the moment right? Any way it gets under the radar?
> >
> > A common example is to run all code analyzis concurrently (findbugs, pmd,
> > checkstyle, ...) at the same time without waiting for one then the other
> > etc since all can be very long.
> >
> > It could be nice an fancy to define part of the reactor parallelisablity
> in
> > the pom, like:
> >
> > <plan>
> >     <parallel>
> >         <phase>process-sources</phase>
> >         <executions>
> >             <execution>
> >               <groupId>...</groupId>
> >               <artifactId>...</artifactId>
> >               <version>...</version>
> >               <id>...</id>
> >             <execution>
> >             <execution>
> >               <groupId>...</groupId>
> >               <artifactId>...</artifactId>
> >               <version>...</version>
> >               <id>...</id>
> >             <execution>
> >         </executions>
> >     </parallel>
> > </plan>
> >
> > anything to enhance it?
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github <https://github.com/
> > rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Romain Manni-Bucau
Currently maven can't but I expect a way to do it, either in the next xsd
as originally proposed or, why not, with a naming convention in the id of
the execution (<execution>my-exec#after#other-exec</execution> or something
like that if we want it before maven 4)

The nice thing is that once done it makes phases pretty much useless (it is
just about making implicit these dependencies) and it makes the whole build
parallelizable and not just modules which will often find some bottleneck
modules in projects building a distribution.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le jeu. 6 déc. 2018 à 14:44, Mickael Istria <[hidden email]> a écrit :

> Hi,
>
> > any way we move that topic forward beginning of next year?
>
> I guess providing patches would be the best way ;)
>
> I'm coming late in this discussion and I'm a newcomer on that list, so I
> could miss context. This could relate to an effort we are doing in Eclipse
> IDE (and Eclipse m2e) to run module builds in parallel. One question I have
> is how do you know 2 tasks aren't conflicting? We didn't figure out a safe
> way to know that in m2e, maybe I missed something?
>
> Cheers,
>
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Romain Manni-Bucau
Mojos already have the ability to say if they are threadsafe, what do you
see missing?

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le jeu. 6 déc. 2018 à 15:23, Mickael Istria <[hidden email]> a écrit :

> I think there is a difference between scheduling tasks (one is after the
> other) and assuming that 2 tasks that are ready can run in parallel.
> Adding scheduling data would be helpful and get rid of the concept of
> phases, for sure; but we also -and more importantly- need the existing
> mojos to be audited and be able to declare whether they're thread-safe
> before running them in parallel, and this sounds like a gigantic work.
>
Reply | Threaded
Open this post in threaded view
|

Re: parallelise not overlapping tasks

Tibor Digana
In reply to this post by Romain Manni-Bucau
In my projects, the most plugins use single execution.
External projects also have this kind of principle.
Thus we should have a look in those possibilities where the most plugins
can gain the performance.
Usually the compiler and tests take long.
I know that maven-compiler-plugin:3.8.1 will be incremental which is good
of course but we should perhaps continue with gaining the build performance.
If somebody has an idea on how to develop a compiler which partially
compiles a module depending on SCM changes, feel free to bring it to our
mailing list. The same with tests where the set of tests is changed
depending on SCM changes.

Cheers
Tibor


On Fri, Jan 19, 2018 at 2:21 PM Romain Manni-Bucau <[hidden email]>
wrote:

> Hi guys,
>
> there is no way to parallelize not conflicting tasks for a same phase at
> the moment right? Any way it gets under the radar?
>
> A common example is to run all code analyzis concurrently (findbugs, pmd,
> checkstyle, ...) at the same time without waiting for one then the other
> etc since all can be very long.
>
> It could be nice an fancy to define part of the reactor parallelisablity in
> the pom, like:
>
> <plan>
>     <parallel>
>         <phase>process-sources</phase>
>         <executions>
>             <execution>
>               <groupId>...</groupId>
>               <artifactId>...</artifactId>
>               <version>...</version>
>               <id>...</id>
>             <execution>
>             <execution>
>               <groupId>...</groupId>
>               <artifactId>...</artifactId>
>               <version>...</version>
>               <id>...</id>
>             <execution>
>         </executions>
>     </parallel>
> </plan>
>
> anything to enhance it?
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau>
>