The quality of core-its

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

The quality of core-its

Tibor Digana
Hi all,

I have noticed several technical issues in core-its:

1. some tests are not listed in the IntegrationTestSuite.java.
    I have already added two (MavenITmng5175WagonHttpTest,
MavenIT0146InstallerSnapshotNaming).
    Some tests are broken, excluded from the execution and they are listed
in the comments.
    Not sure how many more are missing in the test suite.
2. The method "resetStreams()" is not called within the "finally {}" block.
Without it, we may expect that the begin of the logs of next test would
appear in log.txt of previous (forked) test.
3. we still use Maven 2 tests. My proposal is to delete them which improves
the performance of the build.

There are also non-technical issues where the team commits to broken build.
There is a practice to re-trigger the build manually in order to bypass
sporadic errors.
IMO there should be rules in the comunity where nothing would be pushed to
the Maven till the problem is fixed. I think, this is logicall and improves
the quality of the project.

We can discuss it.
Maybe you have noticed the same!

Cheers
Tibor
Reply | Threaded
Open this post in threaded view
|

Re: The quality of core-its

rfscholte
Hi,

it is good to have a closer look at this every now and than. In general  
Mavens tests and integration tests are stable. For the few time I see a  
non-related issue I don't mind re-kicking the builds, as it is happening  
rarely.

Regarding #1 I'm wondering why we have to this manually? Is it about  
order? In any way, I think it should be possible to autodetect Tests, but  
there are other issues that are way more important to solve.
#2 I don't know, haven't seen that as being an issue.
#3 I doubt performance is the issue here. I would suggest another project  
where can can improve buildtime, and that is actually the surefire  
project. To me it is insane to see 20 different builds that take between  
about 1h and 1.5h on a shared infrastructure, whereas ITs of Maven take  
less than 0.5h. So the biggest gain is not on Maven ITs.

thanks,
Robert


On Sun, 11 Aug 2019 01:09:40 +0200, Tibor Digana <[hidden email]>  
wrote:

> Hi all,
>
> I have noticed several technical issues in core-its:
>
> 1. some tests are not listed in the IntegrationTestSuite.java.
>     I have already added two (MavenITmng5175WagonHttpTest,
> MavenIT0146InstallerSnapshotNaming).
>     Some tests are broken, excluded from the execution and they are  
> listed
> in the comments.
>     Not sure how many more are missing in the test suite.
> 2. The method "resetStreams()" is not called within the "finally {}"  
> block.
> Without it, we may expect that the begin of the logs of next test would
> appear in log.txt of previous (forked) test.
> 3. we still use Maven 2 tests. My proposal is to delete them which  
> improves
> the performance of the build.
>
> There are also non-technical issues where the team commits to broken  
> build.
> There is a practice to re-trigger the build manually in order to bypass
> sporadic errors.
> IMO there should be rules in the comunity where nothing would be pushed  
> to
> the Maven till the problem is fixed. I think, this is logicall and  
> improves
> the quality of the project.
>
> We can discuss it.
> Maybe you have noticed the same!
>
> Cheers
> Tibor

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: The quality of core-its

Tibor Digana
Hello Robert,

Regarding #1 it is very important isue because when you write an IT you of
course you want to see it executed by `mvn verify ...`.
So yes there are also other important things to do, but tell me how you
want to guarantee the quality if the test is not employed by the test
suite. Then #1 becomes very important.
I had to repair MavenITmng5175WagonHttpTest because it was not run by the
suite. Now this test is plugged in to the suite and it does what it has to
do!
I am not sure how many such tests are "unplugged".

Regarding #2 I saw the Verifier code.
Everybody can do it too and see the control flow of the code when this
method is not called.
I think this method talks about reliability of std/out stream and logs
written in "log.txt". And because the tests rely on what has been written
in the logs, it is also important issue.
Unfortunately, there are a lot of tests to fix this way and use
"try-finally". Maybe another interceptor in JUnit3/4 can do that
automatically, just an idea.

#3 was answered in the previous email.

Cheers
Tibor

On Sun, Aug 11, 2019 at 2:04 PM Robert Scholte <[hidden email]> wrote:

> Hi,
>
> it is good to have a closer look at this every now and than. In general
> Mavens tests and integration tests are stable. For the few time I see a
> non-related issue I don't mind re-kicking the builds, as it is happening
> rarely.
>
> Regarding #1 I'm wondering why we have to this manually? Is it about
> order? In any way, I think it should be possible to autodetect Tests, but
> there are other issues that are way more important to solve.
> #2 I don't know, haven't seen that as being an issue.
> #3 I doubt performance is the issue here. I would suggest another project
> where can can improve buildtime, and that is actually the surefire
> project. To me it is insane to see 20 different builds that take between
> about 1h and 1.5h on a shared infrastructure, whereas ITs of Maven take
> less than 0.5h. So the biggest gain is not on Maven ITs.
>
> thanks,
> Robert
>
>
> On Sun, 11 Aug 2019 01:09:40 +0200, Tibor Digana <[hidden email]>
>
> wrote:
>
> > Hi all,
> >
> > I have noticed several technical issues in core-its:
> >
> > 1. some tests are not listed in the IntegrationTestSuite.java.
> >     I have already added two (MavenITmng5175WagonHttpTest,
> > MavenIT0146InstallerSnapshotNaming).
> >     Some tests are broken, excluded from the execution and they are
> > listed
> > in the comments.
> >     Not sure how many more are missing in the test suite.
> > 2. The method "resetStreams()" is not called within the "finally {}"
> > block.
> > Without it, we may expect that the begin of the logs of next test would
> > appear in log.txt of previous (forked) test.
> > 3. we still use Maven 2 tests. My proposal is to delete them which
> > improves
> > the performance of the build.
> >
> > There are also non-technical issues where the team commits to broken
> > build.
> > There is a practice to re-trigger the build manually in order to bypass
> > sporadic errors.
> > IMO there should be rules in the comunity where nothing would be pushed
> > to
> > the Maven till the problem is fixed. I think, this is logicall and
> > improves
> > the quality of the project.
> >
> > We can discuss it.
> > Maybe you have noticed the same!
> >
> > Cheers
> > Tibor
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>