Filetrasformer design

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Filetrasformer design

Slawomir Jaranowski
Hi,


During a spike on maven-sign-plugin we discovered the chain feature in the
Transformation process will be useful.

Current transformation solution is dedicated to pom transformation, it will
be also useful to prepare more flexible solutions for another type of
transformation like preparing signatures for artefacts.

Present implementation of transformers cause of calling the whole
transformation process from beginning for all requests.

We process it twice - first for install and second for deploy.
So when we add more complicated and more resource demand (like computation
of signature) we can decrease performance in a visible way.

Now try to build a process with a chined transformer, we will have a
PomTransformer which can produce two kinds of pom.
Next transformer will be for signing SigTransformer, so we will have:

- pom.xml
    -> PomTransformer
        - pom.xml                     FileTransformer
            -> SigTransformer
                - pom.xml.asc         FileTransformer
        - pom5.xml                    FileTransformer
            -> SigTransformer
                - pom5.xml.asc        FileTransformer


In this example we can see that in output we need to produce 4 artifacts:
pom.xml, pom5.xml and pom.xml.asc, pom 5.xml.asc so we need 4 final
transformers and transformer from previous level was run twice.

The biggest problem which I see is that we need results from all
Transformers level.

I'm afraid that when we start to use transformers for all artifacts we can
decrease performance for projects with big artifacts.

And the end - summary:
1. problem of running transformers many times during project build
2.  multilevel / chained transformers - how to reuse output from previous
level

--
Sławomir Jaranowski
Reply | Threaded
Open this post in threaded view
|

Re: Filetrasformer design

rfscholte
It is true that for both install and deploy the transformers will be executed, however in general there is no need to do both, this is how Maven still works, but should be solved with MNG-5667[1].
This might be a good reason to give this a higher priority. (and to get "mvn clean install" out of everybody's system)

I don't think there's a lot of need for transformers, so I could image Maven will predefine the available transformation hooks, i.e. pom transformers and checksum/signature transformers.

I would like to see numbers of performance differences. (keep in mind it is probably an unfair comparison, because signing is now done by a third party program).
To reduce memory consumption the transfomer is using PipedStreams, which should imply that size doesn't matter anymore.

thanks,
Robert


[1] https://issues.apache.org/jira/browse/MNG-5667
On 11-10-2020 09:06:11, Slawomir Jaranowski <[hidden email]> wrote:
Hi,


During a spike on maven-sign-plugin we discovered the chain feature in the
Transformation process will be useful.

Current transformation solution is dedicated to pom transformation, it will
be also useful to prepare more flexible solutions for another type of
transformation like preparing signatures for artefacts.

Present implementation of transformers cause of calling the whole
transformation process from beginning for all requests.

We process it twice - first for install and second for deploy.
So when we add more complicated and more resource demand (like computation
of signature) we can decrease performance in a visible way.

Now try to build a process with a chined transformer, we will have a
PomTransformer which can produce two kinds of pom.
Next transformer will be for signing SigTransformer, so we will have:

- pom.xml
-> PomTransformer
- pom.xml FileTransformer
-> SigTransformer
- pom.xml.asc FileTransformer
- pom5.xml FileTransformer
-> SigTransformer
- pom5.xml.asc FileTransformer


In this example we can see that in output we need to produce 4 artifacts:
pom.xml, pom5.xml and pom.xml.asc, pom 5.xml.asc so we need 4 final
transformers and transformer from previous level was run twice.

The biggest problem which I see is that we need results from all
Transformers level.

I'm afraid that when we start to use transformers for all artifacts we can
decrease performance for projects with big artifacts.

And the end - summary:
1. problem of running transformers many times during project build
2. multilevel / chained transformers - how to reuse output from previous
level

--
Sławomir Jaranowski