Re: Why is old IO API used in maven resolver?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Why is old IO API used in maven resolver?

Michael Osipov-2
Am 2020-08-16 um 06:40 schrieb STEFAN REICH:
> Hi there!
>
> I am working on a very large code base, and build performance issues made me look at the maven-resolver source code. In terms of File usages, there are a lot of InputStreams being copied around using ByteBuffer, instead of using FileChannel.transferTo. Affected classes are DefaultFileProcessor, ChecksumCalculator, WagonTransporter, AbstractTransporter and potentially more. Was it a conscious decision to use this pattern over the more efficient transferTo? Would you accept a PR with more modern NIO API that still works with JDK 7?

Hi Stefan,

I have been working on Resolver for the last couple of releases. Also on
Wagon -- which goes hand in hand with Resolver. I don't think that there
was a decision to go a suboptimal route. No one just stood up to make it
better.

Please start with Wagon first, because Resolver sits on top of Wagon
which is low level.

> Here are the throughput results from a JMH benchmark, copying a 22MB file around using the pattern currently used in maven, and transferTo, as measured on macOS with JDK 11 on an SSD.
>
> Result "MyBenchmark.resolverCopy":
>    291.362 ±(99.9%) 5.443 ops/s [Average]
>    (min, avg, max) = (276.923, 291.362, 302.911), stdev = 7.266
>    CI (99.9%): [285.919, 296.804] (assumes normal distribution)
>
> Result "MyBenchmark.transferTo":
>    325.188 ±(99.9%) 8.838 ops/s [Average]
>    (min, avg, max) = (306.978, 325.188, 355.784), stdev = 11.799
>    CI (99.9%): [316.350, 334.026] (assumes normal distribution)

Please provide your clean room tests, I have four different operating
systems and JDKs to test.

I will happily accept every PR which makes Resolver and Wagon better as
long as the PRs are logically decoupled and wellreasoned.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Why is old IO API used in maven resolver?

STEFAN REICH
@Oliver: thanks, I’ll go ahead and test the proposed changes on our build this week. The performance benefit is expected to differ on various OS, but I don’t have benchmark results for them.

@Xeno: The two results I quoted are what you are asking for. resolverCopy is testing the current implementation and transferTo the new one. The existing code can copy the file 5.443 times a second, the proposed fix can perform 8.838 copies per second.

@Michael: ACK

Thanks for your support!
Stefan

On 2020/08/16 04:40:11, STEFAN REICH <[hidden email]> wrote:

> Hi there!>
>
> I am working on a very large code base, and build performance issues made me look at the maven-resolver source code. In terms of File usages, there are a lot of InputStreams being copied around using ByteBuffer, instead of using FileChannel.transferTo. Affected classes are DefaultFileProcessor, ChecksumCalculator, WagonTransporter, AbstractTransporter and potentially more. Was it a conscious decision to use this pattern over the more efficient transferTo? Would you accept a PR with more modern NIO API that still works with JDK 7?>
>
> Here are the throughput results from a JMH benchmark, copying a 22MB file around using the pattern currently used in maven, and transferTo, as measured on macOS with JDK 11 on an SSD.>
>
> Result "MyBenchmark.resolverCopy":>
>   291.362 ±(99.9%) 5.443 ops/s [Average]>
>   (min, avg, max) = (276.923, 291.362, 302.911), stdev = 7.266>
>   CI (99.9%): [285.919, 296.804] (assumes normal distribution)>
>
> Result "MyBenchmark.transferTo":>
>   325.188 ±(99.9%) 8.838 ops/s [Average]>
>   (min, avg, max) = (306.978, 325.188, 355.784), stdev = 11.799>
>   CI (99.9%): [316.350, 334.026] (assumes normal distribution)>
>
>
> Thanks!>
> Stefan>
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]