[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
GitHub user lamyaa opened a pull request:

    https://github.com/apache/maven-surefire/pull/96

    Adding individual rerun times to XML reports

    When a test fails and is rerun (with rerunFailingTestsCount > 0), the XML report only shows the time elapsed for the first run of the test method.
   
    For example, I ran a test suite that has one failing test with `mvn test -Dsurefire.rerunFailingTestsCount=1`. The console output shows the time elapsed for each test method (re)run:
   
    ```
    Tests run: 19, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.807 sec <<< FAILURE! - in org.apache.http.pool.TestConnPool
    testStatefulConnectionRedistributionOnPerRouteMaxLimit(org.apache.http.pool.TestConnPool)  Time elapsed: 0.016 sec  <<< FAILURE!
    junit.framework.AssertionFailedError: null
            at junit.framework.Assert.fail(Assert.java:47)
            at junit.framework.Assert.fail(Assert.java:53)
            at org.apache.http.pool.TestConnPool.testStatefulConnectionRedistributionOnPerRouteMaxLimit(TestConnPool.java:484)
   
    testStatefulConnectionRedistributionOnPerRouteMaxLimit(org.apache.http.pool.TestConnPool)  Time elapsed: 0.007 sec  <<< FAILURE!
    junit.framework.AssertionFailedError: null
            at junit.framework.Assert.fail(Assert.java:47)
            at junit.framework.Assert.fail(Assert.java:53)
            at org.apache.http.pool.TestConnPool.testStatefulConnectionRedistributionOnPerRouteMaxLimit(TestConnPool.java:484)
    ```
   
    However, the XML report only shows the time elapsed for the first method run:
   
    ```xml
      <testcase name="testStatefulConnectionRedistributionOnPerRouteMaxLimit" classname="org.apache.http.pool.TestConnPool" time="0.016">
        <failure type="junit.framework.AssertionFailedError:">junit.framework.AssertionFailedError: null
            at junit.framework.Assert.fail(Assert.java:47)
            at junit.framework.Assert.fail(Assert.java:53)
            at org.apache.http.pool.TestConnPool.testStatefulConnectionRedistributionOnPerRouteMaxLimit(TestConnPool.java:484)
        </failure>
        <rerunFailure type="junit.framework.AssertionFailedError:">junit.framework.AssertionFailedError: null
            at junit.framework.Assert.fail(Assert.java:47)
            at junit.framework.Assert.fail(Assert.java:53)
            at org.apache.http.pool.TestConnPool.testStatefulConnectionRedistributionOnPerRouteMaxLimit(TestConnPool.java:484)
        </rerunFailure>
      </testcase>
    ```
   
    This pull request extends the XML reports to also show the individual times of each rerun as well as the total time for the original run + the reruns. With this change, the XML report looks as follows:
   
    ```xml
      <testcase name="testStatefulConnectionRedistributionOnPerRouteMaxLimit" classname="org.apache.http.pool.TestConnPool" time="0.025">
        <failure type="junit.framework.AssertionFailedError:" time="0.017">junit.framework.AssertionFailedError: null
            at junit.framework.Assert.fail(Assert.java:47)
            at junit.framework.Assert.fail(Assert.java:53)
            at org.apache.http.pool.TestConnPool.testStatefulConnectionRedistributionOnPerRouteMaxLimit(TestConnPool.java:484)
        </failure>
        <rerunFailure type="junit.framework.AssertionFailedError:" time="0.008">junit.framework.AssertionFailedError: null
            at junit.framework.Assert.fail(Assert.java:47)
            at junit.framework.Assert.fail(Assert.java:53)
            at org.apache.http.pool.TestConnPool.testStatefulConnectionRedistributionOnPerRouteMaxLimit(TestConnPool.java:484)
        </rerunFailure>
      </testcase>
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lamyaa/maven-surefire rerun_time_xml

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/maven-surefire/pull/96.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #96
   
----
commit a138d0aa6758a23760de12e4a23c1416c3303d02
Author: lamyaa <[hidden email]>
Date:   2015-05-08T17:19:42Z

    Adding individual rerun times and total times to xml reports

commit d74d57e73c842ac697eab739aa8bf66418631e6a
Author: Lamyaa Eloussi <[hidden email]>
Date:   2015-06-04T22:34:42Z

    Synching with original repository

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
Github user Tibor17 commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-142758232
 
    @lamyaa
    Did you have time to run this test using the fix we talked about in `SUREFIRE-1144`?
    Is the execution time calculation broken ?
    If it is not broken, we can continue with this. I would rather talk with Andreas before making any conclusion in this PR. First of all I will test rerun together with HTML repor. Having `time` in partial executions within testcase is maybe releval statistics. The only problem is with `failure` because it does not have `time` in testcase without rerun. There is also another problem with `time` in `failure` when JUnit tests use `ErrorCollector` which reports multiple failures in testcase which needs to be tested in current release 2.18.1. The problem is that the time of such failures is not aggregated in reality and thus individual times match the sum and the individual times are not necessary. I guess the only failures and rerunFailure elements may have `time` if and only if rerun is applied in such testcase.
   
    Currently this decision is not trivial for me to make because I need to make more investigations of current status.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
In reply to this post by jon-bell
Github user lamyaa commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-143900518
 
    @Tibor17 I just tested this PR with your recent SUREFIRE-1144 fix. It still works.
    Again, the point here was really just to provide the same level of information as the console (which already shows individual times for each rerun). As for the time shown in the test case, it could be the sum, the last time, or we could just leave it as it is currently (the first run time).
    It's also fine to speak with Andreas first and get his take on this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
In reply to this post by jon-bell
Github user Tibor17 commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-143955811
 
    @agudian I would like to include you into our discussion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
In reply to this post by jon-bell
Github user agudian commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-143957705
 
    I agree that it makes sense to add the same information to the xml report that we provide in the console output.
    Regarding what time to take for the test case, I'd opt for whatever we print in the console. If we get to choose, my first guess would have been the last execution as well. But heck - whether that's what a user needs really depends on the use case and the kind of error ;-). So let's not overthink this. Consistency is more important for this part ;-).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
In reply to this post by jon-bell
Github user Tibor17 commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-170148843
 
    @lamyaa
    Hi how are you?
    We should recover activity on this issue. As for your information, I created a separate branch 3.0-rc1 and the master branch is ready for bugfixing only. So we can still make a progress to improve stability on future Versions 2.19.2, 2.19.3, etc until the last 3.0-RCx is ready to become 3.0.
    I guess this is good strategy that everything has a chance to be in progress. Of course the only problem is human resources.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
In reply to this post by jon-bell
Github user lamyaa commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-170191124
 
    @Tibor17
    Hi Tibor. I'm doing great, thanks. Congrats on the release! :)
    I'll work on this in the next couple of days to resolve conflicts and test it again. Should I then make a new pull request for 3.0-rc1?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
In reply to this post by jon-bell
Github user Tibor17 commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-170199857
 
    >>Should I then make a new pull request for 3.0-rc1?
    no, no. Just keep going in the master. Thx.
   
    On Sat, Jan 9, 2016 at 5:15 AM, lamyaa <[hidden email]> wrote:
   
    > @Tibor17 <https://github.com/Tibor17>
    > Hi Tibor. I'm doing great, thanks. Congrats on the release! :)
    > I'll work on this in the next couple of days to resolve conflicts and test
    > it again. Should I then make a new pull request for 3.0-rc1?
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/maven-surefire/pull/96#issuecomment-170191124>.
    >
   
   
   
    --
    Cheers
    Tibor



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire pull request: Adding individual rerun times to XML ...

jon-bell
In reply to this post by jon-bell
Github user lamyaa commented on the pull request:

    https://github.com/apache/maven-surefire/pull/96#issuecomment-170758490
 
    @Tibor17  I merged the latest changes from master and resolved conflicts. I checked that it is still working as expected. Let me know if there is anything else I should do for this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[GitHub] maven-surefire issue #96: Adding individual rerun times to XML reports

jon-bell
In reply to this post by jon-bell
Github user lamyaa commented on the issue:

    https://github.com/apache/maven-surefire/pull/96
 
    @Tibor17  We kind of lost track of this PR. Is there anything else needed from me or can we get this merged?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Loading...