[Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

Massimiliano Dessì
Hi all,
as a part of my daily job in Red Hat
I've worked on a "customization" of Maven for our Kie Workbench used with
Drools, JBPM and Optaplanner.
The starting point was exports Objects created by Drools inside our Maven
plugin, but the features are growed a lot.

A brief and not exhaustive list of the features:
-From static file producer to in memory producer and exporter
-Reusable internal components to optimize memory footprint and time of
execution
-Maven like a builder Daemon
-Configurable behaviour using a pipeline of decorators
-Stateless (when possible)
-Concurrent builds and concurrent logs
-Cloud enabled
-Local and remote executions
-Plugins turned from FileSystem based to in memory based
-Async API to consume build result
-Incremental builds

The current version is based on the Maven 3.3.9 but could be easily updated
and is in our plans in the next months.
Currently we have four modules plus other testing module and the offprocess
module is under development.

core
service
maven-plugins (plugins turned from FS to in memory)
distribution

Preso with details:
https://www.slideshare.net/desmax74/when-old-meets-new-turning-maven-into-a-high-scalable-resource-efficient-cloud-ready-microservice

I'd like to contribute this code, but since isn't a simple patch I'd like
discuss about how to contribute,
code plus tests is around 18k loc.

Best
Max
--
http://twitter.com/desmax74
http://www.linkedin.com/in/desmax74
http://www.slideshare.net/desmax74
Author of: Spring 2.5 Aspect Oriented Programming
"http://www.packtpub.com/aspect-oriented-programming-with-spring-2-5/book"
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

Karl Heinz Marbaise-3
Hi,

On 06/08/18 11:16, Massimiliano Dessì wrote:

> Hi all,
> as a part of my daily job in Red Hat
> I've worked on a "customization" of Maven for our Kie Workbench used with
> Drools, JBPM and Optaplanner.
> The starting point was exports Objects created by Drools inside our Maven
> plugin, but the features are growed a lot.
>
> A brief and not exhaustive list of the features:
> -From static file producer to in memory producer and exporter
> -Reusable internal components to optimize memory footprint and time of
> execution
> -Maven like a builder Daemon
> -Configurable behaviour using a pipeline of decorators
> -Stateless (when possible)
> -Concurrent builds and concurrent logs
> -Cloud enabled
> -Local and remote executions
> -Plugins turned from FileSystem based to in memory based
> -Async API to consume build result
> -Incremental builds
>
> The current version is based on the Maven 3.3.9 but could be easily updated
> and is in our plans in the next months.
> Currently we have four modules plus other testing module and the offprocess
> module is under development.
>
> core
> service
> maven-plugins (plugins turned from FS to in memory)
> distribution
>
> Preso with details:
> https://www.slideshare.net/desmax74/when-old-meets-new-turning-maven-into-a-high-scalable-resource-efficient-cloud-ready-microservice
>
> I'd like to contribute this code, but since isn't a simple patch I'd like
> discuss about how to contribute,
> code plus tests is around 18k loc.
>
> Best
> Max
>

This sounds great...at first I would like to take a deeper look on it
and do of course some test?...

Kind regards
Karl Heinz Marbaise

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

Karl Heinz Marbaise-3
Hi,

is the code somewhere available? Usable ?


kind regards
Karl Heinz Marbaise

On 06/08/18 13:15, Karl Heinz Marbaise wrote:

> Hi,
>
> On 06/08/18 11:16, Massimiliano Dessì wrote:
>> Hi all,
>> as a part of my daily job in Red Hat
>> I've worked on a "customization" of Maven for our Kie Workbench used with
>> Drools, JBPM and Optaplanner.
>> The starting point was exports Objects created by Drools inside our Maven
>> plugin, but the features are growed a lot.
>>
>> A brief and not exhaustive list of the features:
>> -From static file producer to in memory producer and exporter
>> -Reusable internal components to optimize memory footprint and time of
>> execution
>> -Maven like a builder Daemon
>> -Configurable behaviour using a pipeline of decorators
>> -Stateless (when possible)
>> -Concurrent builds and concurrent logs
>> -Cloud enabled
>> -Local and remote executions
>> -Plugins turned from FileSystem based to in memory based
>> -Async API to consume build result
>> -Incremental builds
>>
>> The current version is based on the Maven 3.3.9 but could be easily
>> updated
>> and is in our plans in the next months.
>> Currently we have four modules plus other testing module and the
>> offprocess
>> module is under development.
>>
>> core
>> service
>> maven-plugins (plugins turned from FS to in memory)
>> distribution
>>
>> Preso with details:
>> https://www.slideshare.net/desmax74/when-old-meets-new-turning-maven-into-a-high-scalable-resource-efficient-cloud-ready-microservice 
>>
>>
>> I'd like to contribute this code, but since isn't a simple patch I'd like
>> discuss about how to contribute,
>> code plus tests is around 18k loc.
>>
>> Best
>> Max
>>
>
> This sounds great...at first I would like to take a deeper look on it
> and do of course some test?...
>
> Kind regards
> Karl Heinz Marbaise

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

Massimiliano Dessì
In reply to this post by Massimiliano Dessì
Hi Tibor
the pom is changed the rirst time the compiler discover the project,
it turning off the default Maven compiler and adding Takari,
I've contributed the failOnError flag in Takari projects because we need to
provide a real incremental behaviour
to the user, compile all the correct stuff and provide the error message
about the parts to fix,
instead of stop everything.

2018-08-06 13:23 GMT+02:00 Tibor Digana <[hidden email]>:

> I am interested in the incremental builds.
> Takari has some extension for incremental build, and there are also some
> extensions for children in Maven multi-module project when you update it
> via Git.
>
> On Mon, Aug 6, 2018 at 11:16 AM, Massimiliano Dessì <
> [hidden email]> wrote:
>
> > Hi all,
> > as a part of my daily job in Red Hat
> > I've worked on a "customization" of Maven for our Kie Workbench used with
> > Drools, JBPM and Optaplanner.
> > The starting point was exports Objects created by Drools inside our Maven
> > plugin, but the features are growed a lot.
> >
> > A brief and not exhaustive list of the features:
> > -From static file producer to in memory producer and exporter
> > -Reusable internal components to optimize memory footprint and time of
> > execution
> > -Maven like a builder Daemon
> > -Configurable behaviour using a pipeline of decorators
> > -Stateless (when possible)
> > -Concurrent builds and concurrent logs
> > -Cloud enabled
> > -Local and remote executions
> > -Plugins turned from FileSystem based to in memory based
> > -Async API to consume build result
> > -Incremental builds
> >
> > The current version is based on the Maven 3.3.9 but could be easily
> updated
> > and is in our plans in the next months.
> > Currently we have four modules plus other testing module and the
> offprocess
> > module is under development.
> >
> > core
> > service
> > maven-plugins (plugins turned from FS to in memory)
> > distribution
> >
> > Preso with details:
> > https://www.slideshare.net/desmax74/when-old-meets-new-
> > turning-maven-into-a-high-scalable-resource-efficient-
> > cloud-ready-microservice
> >
> > I'd like to contribute this code, but since isn't a simple patch I'd like
> > discuss about how to contribute,
> > code plus tests is around 18k loc.
> >
> > Best
> > Max
> > --
> > http://twitter.com/desmax74
> > http://www.linkedin.com/in/desmax74
> > http://www.slideshare.net/desmax74
> > Author of: Spring 2.5 Aspect Oriented Programming
> > "http://www.packtpub.com/aspect-oriented-programming-
> with-spring-2-5/book"
> >
>



--
http://twitter.com/desmax74
http://www.linkedin.com/in/desmax74
http://www.slideshare.net/desmax74
Author of: Spring 2.5 Aspect Oriented Programming
"http://www.packtpub.com/aspect-oriented-programming-with-spring-2-5/book"
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

Paul Hammant
In reply to this post by Massimiliano Dessì
Related to Incremental - idempotent builds would be good.

Having been exposed to Google's Blaze (now Bazel in OSSland) and directed
graph build system science (as opposed to recursive build systems) there's
a great efficiency from skipping parts of a build that don't need to be
done.

Also related to incremental -
https://github.com/khmarbaise/incremental-module-builder goes some way.  It
doesn't do changes in dep triggering a build (or not), I don't think. It
also can't work out the difference between the heavy phases of a module's
build that are independently skipable (compile, test compile, jar, shade,
etc).

- Paul
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

Alexandre Porcelli Bakos
In reply to this post by Massimiliano Dessì
Hi Robert,

 I'm Alex Porcelli and I work with Max. I can also create some jiras
about things that we couldn't do.

 Here's an example (likely my first JIRA):
 In our use case, all sources are stored in bare Git repositories and
we manipulate it using NIO2 API [1], if Maven accepted NIO2 Path
interface for example, instead of File, we didn't need to create a
temp folder to build. This would also require split the input
parameter to build, one for sources (that likely would just use
directly Git) and another Path for the target folder.

 This is one example out of others that we found during our journey.

Regards,
---
Alexandre Porcelli
Principal Software Engineer
Red Hat Business Systems and Intelligence Group
On Fri, Nov 2, 2018 at 5:57 AM Robert Scholte <[hidden email]> wrote:

>
> On Wed, 08 Aug 2018 09:49:17 +0200, Massimiliano Dessì
> <[hidden email]> wrote:
>
> > Hi Robert
> >
> > 2018-08-07 18:35 GMT+02:00 Robert Scholte <[hidden email]>:
> >
> >> Hi Max,
> >>
> >> regarding the contribution, I would like to propose the following:
> >> - At least create a JIRA issue for every feature.
>
> Can we start creating these JIRA issues? I'd like to work on a list of
> features we should embed in Maven 4 and some of these are worth adding.
> Would be nice to keep you as the reporter and that you can refer to the
> codechanges on your fork.
> All to make it easier to implement it in Maven Core.
>
> thanks,
> Robert
>
> >> - Due to the large amount of code I can imagine there are some
> >> dependencies between the features, i.e some features should be
> >> implemented
> >> first.
> >> - Based on that we can decide what we want to adopt, prioritize/order
> >> and
> >> start working on the implementations.
> >> - I assume CLA[1] is not an issue ;)
> >>
> >
> > +1
> >
> >
> >>
> >> Do you have statistics of the memory consumption?
> >>
> >
> > Nope,
> >
> >>
> >> It looks to me we need to introduce some sort of profiles, so people can
> >> choose the preferred setup.
> >> I see both the memory and concurrency as solutions that come with
> >> "costs"
> >> not everybody wants to pay.
> >> Did you solve this with a specific builder?
> >>
> >
> > I've created the entire "compiler" to use with only one instance or
> > multiple, but now if multiple running in the same time they working
> > correctly,
> > the two major problems arised during the concurrent builds was the random
> > build failed to changed order of the parameters and the output
> > interleaved
> > in the same output.
> >
> >
> >> Also, did you have to break APIs or are all current plugins and
> >> extensions
> >> still usable?
> >>
> >
> > No API breaks, two aspects was mandatory, easy (or at least with few
> > changes) update of Maven version
> > and the same behaviour compared to the MavenCLI,
> > for this reason the changes on Maven API are limited to few classes of
> > the
> > Maven embedder, some was suggested by the debugging experience,
> > everytime the major part of the time was spent on the creation of the
> > Plexus/Sisu Ioc Container and this suggest to reuse the Container and
> > some
> > internal objects if the pom isn't changed between builds,
> > because in our workbench the editing is on the Java files and on drools
> > rules mostly,
> > in this way after the first build, when the maven infrastructure is
> > created
> > and takari build all the buildable stuffs,
> > the others following builds are fast, the Maven infrastructure used by
> > the
> > embedder is ready and only the changes classes needs to be recompiled,
> > in our workbench we cache the compiler created in this way with the
> > project
> > associated, and this association is alive for the entire working session,
> > a step furter could be the changes of Maven internal but this mean breaks
> > the Maven API.
> > Then when we have writed some concurrent build test to see if the Maven
> > runtime was happy we have discovered some problems and fixed,
> > specific slf4j in memory appenders are used to write the specific build
> > output, since the embedder redirects the entire System.out to a single
> > file.
> >
> > Here the Mojo used to share the drools runtime objects with the client
> > caller, it's the culprit of all these stuffs :)
> > https://github.com/kiegroup/droolsjbpm-integration/blob/master/kie-maven-plugin/src/main/java/org/kie/maven/plugin/BuildMojo.java#L183
> >
> >
> >
> >>
> >> thanks,
> >> Robert
> >>
> >> [1] https://www.apache.org/dev/new-committers-guide
> >
> >
> > Best
> > Max
> >
> >
> >>
> >>
> >>
> >> On Mon, 06 Aug 2018 11:16:36 +0200, Massimiliano Dessì <
> >> [hidden email]> wrote:
> >>
> >> Hi all,
> >>> as a part of my daily job in Red Hat
> >>> I've worked on a "customization" of Maven for our Kie Workbench used
> >>> with
> >>> Drools, JBPM and Optaplanner.
> >>> The starting point was exports Objects created by Drools inside our
> >>> Maven
> >>> plugin, but the features are growed a lot.
> >>>
> >>> A brief and not exhaustive list of the features:
> >>> -From static file producer to in memory producer and exporter
> >>> -Reusable internal components to optimize memory footprint and time of
> >>> execution
> >>> -Maven like a builder Daemon
> >>> -Configurable behaviour using a pipeline of decorators
> >>> -Stateless (when possible)
> >>> -Concurrent builds and concurrent logs
> >>> -Cloud enabled
> >>> -Local and remote executions
> >>> -Plugins turned from FileSystem based to in memory based
> >>> -Async API to consume build result
> >>> -Incremental builds
> >>>
> >>> The current version is based on the Maven 3.3.9 but could be easily
> >>> updated
> >>> and is in our plans in the next months.
> >>> Currently we have four modules plus other testing module and the
> >>> offprocess
> >>> module is under development.
> >>>
> >>> core
> >>> service
> >>> maven-plugins (plugins turned from FS to in memory)
> >>> distribution
> >>>
> >>> Preso with details:
> >>> https://www.slideshare.net/desmax74/when-old-meets-new-turni
> >>> ng-maven-into-a-high-scalable-resource-efficient-cloud-ready-microservice
> >>>
> >>> I'd like to contribute this code, but since isn't a simple patch I'd
> >>> like
> >>> discuss about how to contribute,
> >>> code plus tests is around 18k loc.
> >>>
> >>> Best
> >>> Max
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]