Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

Massimiliano Dessì
Hi Robert

2018-08-07 18:35 GMT+02:00 Robert Scholte <[hidden email]>:

> Hi Max,
>
> regarding the contribution, I would like to propose the following:
> - At least create a JIRA issue for every feature.
> - Due to the large amount of code I can imagine there are some
> dependencies between the features, i.e some features should be implemented
> first.
> - Based on that we can decide what we want to adopt, prioritize/order and
> start working on the implementations.
> - I assume CLA[1] is not an issue ;)
>

+1


>
> Do you have statistics of the memory consumption?
>

Nope,

>
> It looks to me we need to introduce some sort of profiles, so people can
> choose the preferred setup.
> I see both the memory and concurrency as solutions that come with "costs"
> not everybody wants to pay.
> Did you solve this with a specific builder?
>

I've created the entire "compiler" to use with only one instance or
multiple, but now if multiple running in the same time they working
correctly,
the two major problems arised during the concurrent builds was the random
build failed to changed order of the parameters and the output interleaved
in the same output.


> Also, did you have to break APIs or are all current plugins and extensions
> still usable?
>

No API breaks, two aspects was mandatory, easy (or at least with few
changes) update of Maven version
and the same behaviour compared to the MavenCLI,
for this reason the changes on Maven API are limited to few classes of the
Maven embedder, some was suggested by the debugging experience,
everytime the major part of the time was spent on the creation of the
Plexus/Sisu Ioc Container and this suggest to reuse the Container and some
internal objects if the pom isn't changed between builds,
because in our workbench the editing is on the Java files and on drools
rules mostly,
in this way after the first build, when the maven infrastructure is created
and takari build all the buildable stuffs,
the others following builds are fast, the Maven infrastructure used by the
embedder is ready and only the changes classes needs to be recompiled,
in our workbench we cache the compiler created in this way with the project
associated, and this association is alive for the entire working session,
a step furter could be the changes of Maven internal but this mean breaks
the Maven API.
Then when we have writed some concurrent build test to see if the Maven
runtime was happy we have discovered some problems and fixed,
specific slf4j in memory appenders are used to write the specific build
output, since the embedder redirects the entire System.out to a single file.

Here the Mojo used to share the drools runtime objects with the client
caller, it's the culprit of all these stuffs :)
https://github.com/kiegroup/droolsjbpm-integration/blob/master/kie-maven-plugin/src/main/java/org/kie/maven/plugin/BuildMojo.java#L183



>
> thanks,
> Robert
>
> [1] https://www.apache.org/dev/new-committers-guide


Best
Max


>
>
>
> On Mon, 06 Aug 2018 11:16:36 +0200, Massimiliano Dessì <
> [hidden email]> wrote:
>
> Hi all,
>> as a part of my daily job in Red Hat
>> I've worked on a "customization" of Maven for our Kie Workbench used with
>> Drools, JBPM and Optaplanner.
>> The starting point was exports Objects created by Drools inside our Maven
>> plugin, but the features are growed a lot.
>>
>> A brief and not exhaustive list of the features:
>> -From static file producer to in memory producer and exporter
>> -Reusable internal components to optimize memory footprint and time of
>> execution
>> -Maven like a builder Daemon
>> -Configurable behaviour using a pipeline of decorators
>> -Stateless (when possible)
>> -Concurrent builds and concurrent logs
>> -Cloud enabled
>> -Local and remote executions
>> -Plugins turned from FileSystem based to in memory based
>> -Async API to consume build result
>> -Incremental builds
>>
>> The current version is based on the Maven 3.3.9 but could be easily
>> updated
>> and is in our plans in the next months.
>> Currently we have four modules plus other testing module and the
>> offprocess
>> module is under development.
>>
>> core
>> service
>> maven-plugins (plugins turned from FS to in memory)
>> distribution
>>
>> Preso with details:
>> https://www.slideshare.net/desmax74/when-old-meets-new-turni
>> ng-maven-into-a-high-scalable-resource-efficient-cloud-ready-microservice
>>
>> I'd like to contribute this code, but since isn't a simple patch I'd like
>> discuss about how to contribute,
>> code plus tests is around 18k loc.
>>
>> Best
>> Max
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
http://twitter.com/desmax74
http://www.linkedin.com/in/desmax74
http://www.slideshare.net/desmax74
Author of: Spring 2.5 Aspect Oriented Programming
"http://www.packtpub.com/aspect-oriented-programming-with-spring-2-5/book"
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

rfscholte
On Wed, 08 Aug 2018 09:49:17 +0200, Massimiliano Dessì  
<[hidden email]> wrote:

> Hi Robert
>
> 2018-08-07 18:35 GMT+02:00 Robert Scholte <[hidden email]>:
>
>> Hi Max,
>>
>> regarding the contribution, I would like to propose the following:
>> - At least create a JIRA issue for every feature.

Can we start creating these JIRA issues? I'd like to work on a list of  
features we should embed in Maven 4 and some of these are worth adding.
Would be nice to keep you as the reporter and that you can refer to the  
codechanges on your fork.
All to make it easier to implement it in Maven Core.

thanks,
Robert

>> - Due to the large amount of code I can imagine there are some
>> dependencies between the features, i.e some features should be  
>> implemented
>> first.
>> - Based on that we can decide what we want to adopt, prioritize/order  
>> and
>> start working on the implementations.
>> - I assume CLA[1] is not an issue ;)
>>
>
> +1
>
>
>>
>> Do you have statistics of the memory consumption?
>>
>
> Nope,
>
>>
>> It looks to me we need to introduce some sort of profiles, so people can
>> choose the preferred setup.
>> I see both the memory and concurrency as solutions that come with  
>> "costs"
>> not everybody wants to pay.
>> Did you solve this with a specific builder?
>>
>
> I've created the entire "compiler" to use with only one instance or
> multiple, but now if multiple running in the same time they working
> correctly,
> the two major problems arised during the concurrent builds was the random
> build failed to changed order of the parameters and the output  
> interleaved
> in the same output.
>
>
>> Also, did you have to break APIs or are all current plugins and  
>> extensions
>> still usable?
>>
>
> No API breaks, two aspects was mandatory, easy (or at least with few
> changes) update of Maven version
> and the same behaviour compared to the MavenCLI,
> for this reason the changes on Maven API are limited to few classes of  
> the
> Maven embedder, some was suggested by the debugging experience,
> everytime the major part of the time was spent on the creation of the
> Plexus/Sisu Ioc Container and this suggest to reuse the Container and  
> some
> internal objects if the pom isn't changed between builds,
> because in our workbench the editing is on the Java files and on drools
> rules mostly,
> in this way after the first build, when the maven infrastructure is  
> created
> and takari build all the buildable stuffs,
> the others following builds are fast, the Maven infrastructure used by  
> the
> embedder is ready and only the changes classes needs to be recompiled,
> in our workbench we cache the compiler created in this way with the  
> project
> associated, and this association is alive for the entire working session,
> a step furter could be the changes of Maven internal but this mean breaks
> the Maven API.
> Then when we have writed some concurrent build test to see if the Maven
> runtime was happy we have discovered some problems and fixed,
> specific slf4j in memory appenders are used to write the specific build
> output, since the embedder redirects the entire System.out to a single  
> file.
>
> Here the Mojo used to share the drools runtime objects with the client
> caller, it's the culprit of all these stuffs :)
> https://github.com/kiegroup/droolsjbpm-integration/blob/master/kie-maven-plugin/src/main/java/org/kie/maven/plugin/BuildMojo.java#L183
>
>
>
>>
>> thanks,
>> Robert
>>
>> [1] https://www.apache.org/dev/new-committers-guide
>
>
> Best
> Max
>
>
>>
>>
>>
>> On Mon, 06 Aug 2018 11:16:36 +0200, Massimiliano Dessì <
>> [hidden email]> wrote:
>>
>> Hi all,
>>> as a part of my daily job in Red Hat
>>> I've worked on a "customization" of Maven for our Kie Workbench used  
>>> with
>>> Drools, JBPM and Optaplanner.
>>> The starting point was exports Objects created by Drools inside our  
>>> Maven
>>> plugin, but the features are growed a lot.
>>>
>>> A brief and not exhaustive list of the features:
>>> -From static file producer to in memory producer and exporter
>>> -Reusable internal components to optimize memory footprint and time of
>>> execution
>>> -Maven like a builder Daemon
>>> -Configurable behaviour using a pipeline of decorators
>>> -Stateless (when possible)
>>> -Concurrent builds and concurrent logs
>>> -Cloud enabled
>>> -Local and remote executions
>>> -Plugins turned from FileSystem based to in memory based
>>> -Async API to consume build result
>>> -Incremental builds
>>>
>>> The current version is based on the Maven 3.3.9 but could be easily
>>> updated
>>> and is in our plans in the next months.
>>> Currently we have four modules plus other testing module and the
>>> offprocess
>>> module is under development.
>>>
>>> core
>>> service
>>> maven-plugins (plugins turned from FS to in memory)
>>> distribution
>>>
>>> Preso with details:
>>> https://www.slideshare.net/desmax74/when-old-meets-new-turni
>>> ng-maven-into-a-high-scalable-resource-efficient-cloud-ready-microservice
>>>
>>> I'd like to contribute this code, but since isn't a simple patch I'd  
>>> like
>>> discuss about how to contribute,
>>> code plus tests is around 18k loc.
>>>
>>> Best
>>> Max
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] When Old Meets New: Turning Maven into a High Scalable, Resource Efficient, Cloud Ready Microservice

rfscholte
Cool, just start creating these Jira issues.

The one you've mentioned is probably already covered by MNG-6444[1].
Depending on the set of changes, we can add related issues or just add  
them here.

thanks,
Robert

[1] https://issues.apache.org/jira/browse/MNG-6444


On Fri, 02 Nov 2018 14:45:17 +0100, Alexandre Porcelli Bakos  
<[hidden email]> wrote:

> Hi Robert,
>
>  I'm Alex Porcelli and I work with Max. I can also create some jiras
> about things that we couldn't do.
>
>  Here's an example (likely my first JIRA):
>  In our use case, all sources are stored in bare Git repositories and
> we manipulate it using NIO2 API [1], if Maven accepted NIO2 Path
> interface for example, instead of File, we didn't need to create a
> temp folder to build. This would also require split the input
> parameter to build, one for sources (that likely would just use
> directly Git) and another Path for the target folder.
>
>  This is one example out of others that we found during our journey.
>
> Regards,
> ---
> Alexandre Porcelli
> Principal Software Engineer
> Red Hat Business Systems and Intelligence Group
> On Fri, Nov 2, 2018 at 5:57 AM Robert Scholte <[hidden email]>  
> wrote:
>>
>> On Wed, 08 Aug 2018 09:49:17 +0200, Massimiliano Dessì
>> <[hidden email]> wrote:
>>
>> > Hi Robert
>> >
>> > 2018-08-07 18:35 GMT+02:00 Robert Scholte <[hidden email]>:
>> >
>> >> Hi Max,
>> >>
>> >> regarding the contribution, I would like to propose the following:
>> >> - At least create a JIRA issue for every feature.
>>
>> Can we start creating these JIRA issues? I'd like to work on a list of
>> features we should embed in Maven 4 and some of these are worth adding.
>> Would be nice to keep you as the reporter and that you can refer to the
>> codechanges on your fork.
>> All to make it easier to implement it in Maven Core.
>>
>> thanks,
>> Robert
>>
>> >> - Due to the large amount of code I can imagine there are some
>> >> dependencies between the features, i.e some features should be
>> >> implemented
>> >> first.
>> >> - Based on that we can decide what we want to adopt, prioritize/order
>> >> and
>> >> start working on the implementations.
>> >> - I assume CLA[1] is not an issue ;)
>> >>
>> >
>> > +1
>> >
>> >
>> >>
>> >> Do you have statistics of the memory consumption?
>> >>
>> >
>> > Nope,
>> >
>> >>
>> >> It looks to me we need to introduce some sort of profiles, so people  
>> can
>> >> choose the preferred setup.
>> >> I see both the memory and concurrency as solutions that come with
>> >> "costs"
>> >> not everybody wants to pay.
>> >> Did you solve this with a specific builder?
>> >>
>> >
>> > I've created the entire "compiler" to use with only one instance or
>> > multiple, but now if multiple running in the same time they working
>> > correctly,
>> > the two major problems arised during the concurrent builds was the  
>> random
>> > build failed to changed order of the parameters and the output
>> > interleaved
>> > in the same output.
>> >
>> >
>> >> Also, did you have to break APIs or are all current plugins and
>> >> extensions
>> >> still usable?
>> >>
>> >
>> > No API breaks, two aspects was mandatory, easy (or at least with few
>> > changes) update of Maven version
>> > and the same behaviour compared to the MavenCLI,
>> > for this reason the changes on Maven API are limited to few classes of
>> > the
>> > Maven embedder, some was suggested by the debugging experience,
>> > everytime the major part of the time was spent on the creation of the
>> > Plexus/Sisu Ioc Container and this suggest to reuse the Container and
>> > some
>> > internal objects if the pom isn't changed between builds,
>> > because in our workbench the editing is on the Java files and on  
>> drools
>> > rules mostly,
>> > in this way after the first build, when the maven infrastructure is
>> > created
>> > and takari build all the buildable stuffs,
>> > the others following builds are fast, the Maven infrastructure used by
>> > the
>> > embedder is ready and only the changes classes needs to be recompiled,
>> > in our workbench we cache the compiler created in this way with the
>> > project
>> > associated, and this association is alive for the entire working  
>> session,
>> > a step furter could be the changes of Maven internal but this mean  
>> breaks
>> > the Maven API.
>> > Then when we have writed some concurrent build test to see if the  
>> Maven
>> > runtime was happy we have discovered some problems and fixed,
>> > specific slf4j in memory appenders are used to write the specific  
>> build
>> > output, since the embedder redirects the entire System.out to a single
>> > file.
>> >
>> > Here the Mojo used to share the drools runtime objects with the client
>> > caller, it's the culprit of all these stuffs :)
>> >  
>> https://github.com/kiegroup/droolsjbpm-integration/blob/master/kie-maven-plugin/src/main/java/org/kie/maven/plugin/BuildMojo.java#L183
>> >
>> >
>> >
>> >>
>> >> thanks,
>> >> Robert
>> >>
>> >> [1] https://www.apache.org/dev/new-committers-guide
>> >
>> >
>> > Best
>> > Max
>> >
>> >
>> >>
>> >>
>> >>
>> >> On Mon, 06 Aug 2018 11:16:36 +0200, Massimiliano Dessì <
>> >> [hidden email]> wrote:
>> >>
>> >> Hi all,
>> >>> as a part of my daily job in Red Hat
>> >>> I've worked on a "customization" of Maven for our Kie Workbench used
>> >>> with
>> >>> Drools, JBPM and Optaplanner.
>> >>> The starting point was exports Objects created by Drools inside our
>> >>> Maven
>> >>> plugin, but the features are growed a lot.
>> >>>
>> >>> A brief and not exhaustive list of the features:
>> >>> -From static file producer to in memory producer and exporter
>> >>> -Reusable internal components to optimize memory footprint and time  
>> of
>> >>> execution
>> >>> -Maven like a builder Daemon
>> >>> -Configurable behaviour using a pipeline of decorators
>> >>> -Stateless (when possible)
>> >>> -Concurrent builds and concurrent logs
>> >>> -Cloud enabled
>> >>> -Local and remote executions
>> >>> -Plugins turned from FileSystem based to in memory based
>> >>> -Async API to consume build result
>> >>> -Incremental builds
>> >>>
>> >>> The current version is based on the Maven 3.3.9 but could be easily
>> >>> updated
>> >>> and is in our plans in the next months.
>> >>> Currently we have four modules plus other testing module and the
>> >>> offprocess
>> >>> module is under development.
>> >>>
>> >>> core
>> >>> service
>> >>> maven-plugins (plugins turned from FS to in memory)
>> >>> distribution
>> >>>
>> >>> Preso with details:
>> >>> https://www.slideshare.net/desmax74/when-old-meets-new-turni
>> >>>  
>> ng-maven-into-a-high-scalable-resource-efficient-cloud-ready-microservice
>> >>>
>> >>> I'd like to contribute this code, but since isn't a simple patch I'd
>> >>> like
>> >>> discuss about how to contribute,
>> >>> code plus tests is around 18k loc.
>> >>>
>> >>> Best
>> >>> Max
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: [hidden email]
>> >> For additional commands, e-mail: [hidden email]
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]