Re: Assumption fail treated as unexcepted exception?!

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Assumption fail treated as unexcepted exception?!

Tibor Digana
Yes Piotr, this is another example why Markus has these unpleasant
experiences with JUnit3.
The code with JUnit4 annotation should be fine.

On Mon, Aug 17, 2020 at 1:34 PM Piotr Żygieło <[hidden email]>
wrote:

> Please note, that TestCopyMojo is considered as JUnit3 TestCase
> (
> https://github.com/junit-team/junit4/blob/3a5c6b4d08f408c8ca6a8e0bae71a9bc5a8f97e8/src/main/java/org/junit/internal/builders/JUnit3Builder.java#L17
> ).
> So perhaps JUnit4+ features are not to be expected in such a case?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

AW: Assumption fail treated as unexcepted exception?!

Markus KARG-3
So the bug is that Maven Dependency Plugin's test suite uses the JUnit 4 library for test-compile, but actually executes the tests using JUnit 3, right?

-Markus


-----Ursprüngliche Nachricht-----
Von: Tibor Digana [mailto:[hidden email]]
Gesendet: Montag, 17. August 2020 14:09
An: Maven Developers List
Betreff: Re: Assumption fail treated as unexcepted exception?!

Yes Piotr, this is another example why Markus has these unpleasant
experiences with JUnit3.
The code with JUnit4 annotation should be fine.

On Mon, Aug 17, 2020 at 1:34 PM Piotr Żygieło <[hidden email]>
wrote:

> Please note, that TestCopyMojo is considered as JUnit3 TestCase
> (
> https://github.com/junit-team/junit4/blob/3a5c6b4d08f408c8ca6a8e0bae71a9bc5a8f97e8/src/main/java/org/junit/internal/builders/JUnit3Builder.java#L17
> ).
> So perhaps JUnit4+ features are not to be expected in such a case?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Assumption fail treated as unexcepted exception?!

Tibor Digana
In reply to this post by Tibor Digana
Markus, I was not talking about excuse. I only said that the JUnit library
has a bug regarding the old junit3 style. As far as I know the situation
with this library, it is very unlikely that it would be fixed.
The Surefire cannot do anything about it once the events are not fired from
JUnit3 TestCase.
As Piotr said you have to use the annotations and not to extend the class
TestCase. This would work for sure.
T

On Mon, Aug 17, 2020 at 7:58 PM Markus KARG <[hidden email]> wrote:

> That doesn't make it any better, and it is no excuse that we do not simply
> catch that particular exception inside the Surefire plugin.
> So did we simply forget to implement this is the Surefire plugin (so I can
> add it), did we not implement it deliberately (so I shall not add it), or
> is it broken (so I shall fix it)?
> -Markus
>
> -----Ursprüngliche Nachricht-----
> Von: Tibor Digana [mailto:[hidden email]]
> Gesendet: Montag, 17. August 2020 18:37
> An: Maven Developers List
> Betreff: Re: Assumption fail treated as unexcepted exception?!
>
> This is the outcome in IDEA:
>
> "Tests failed: 1, passed: 0"
>
> So the behavior is the same with Maven.
>
> On Mon, Aug 17, 2020 at 6:35 PM Tibor Digana <[hidden email]>
> wrote:
>
> > Hi Markus,
> >
> > It is a specific problem related to the JUnit library because the test
> > fails in IntelliJ IDEA and in Maven as well.
> >
> > The JUnit4 assumptions fail with yellow markers in IDEA but here the
> > JUnit3' TestCase fails in red as a typical error or failure.
> > And Maven fails this test as follows but i think this behavior starts in
> > the junit library itself.
> >
> > [INFO] Running test.TestAssumptions
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 0.028 s <<< FAILURE! - in test.TestAssumptions
> > [ERROR] test.TestAssumptions.testABC  Time elapsed: 0.01 s  <<< ERROR!
> > org.junit.AssumptionViolatedException: got: <false>, expected: is <true>
> > at test/test.TestAssumptions.testABC(TestAssumptions.java:9)
> >
> > [INFO]
> > [INFO] Results:
> > [INFO]
> > [ERROR] Errors:
> > [ERROR]   TestAssumptions.testABC:9 » AssumptionViolated got: <false>,
> > expected: is <tru...
> > [INFO]
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> > [INFO]
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD FAILURE
> >
> > On Mon, Aug 17, 2020 at 3:07 PM Markus KARG <[hidden email]>
> > wrote:
> >
> >> Martin,
> >>
> >> I don't understand your question.
> >>
> >> You posted the Javadocs, and they tell it clearly: "A test for which an
> >> assumption fails should NOT generate a test case failure.".
> >>
> >> Also Assume's Javadocs
> >> (https://junit.org/junit4/javadoc/4.12/org/junit/Assume.html) are
> pretty
> >> clear about it: "... A failed assumption does NOT mean the code is
> broken,
> >> but that the test provides no useful information. Assume basically means
> >> "don't RUN this test if these conditions don't apply". The default JUnit
> >> runner SKIPS tests with failing assumptions..."
> >>
> >> So do you really want me to ask the JUnit people whether they really
> MEAN
> >> what they clearly wrote...?
> >>
> >> -Markus
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Martin Gainty [mailto:[hidden email]]
> >> Gesendet: Montag, 17. August 2020 12:27
> >> An: Maven Developers List
> >> Betreff: Re: Assumption fail treated as unexcepted exception?!
> >>
> >>
> >>    /**
> >>      * Call to assume that <code>actual</code> satisfies the condition
> >> specified by <code>matcher</code>.
> >>      * If not, the test halts and is ignored.
> >>      * Example:
> >>      * <pre>:
> >>      *   assumeThat(1, is(1)); // passes
> >>      *   foo(); // will execute
> >>      *   assumeThat(0, is(1)); // assumption failure! test halts
> >>      *   int x = 1 / 0; // will never execute
> >>      * </pre>
> >>      *
> >>      * @param <T> the static type accepted by the matcher (this can flag
> >> obvious compile-time problems such as {@code assumeThat(1, is("a"))}
> >>      * @param actual the computed value being compared
> >>      * @param matcher an expression, built of {@link Matcher}s,
> specifying
> >> allowed values
> >>      * @see org.hamcrest.CoreMatchers
> >>      * @see org.junit.matchers.JUnitMatchers
> >>      */
> >>     public static <T> void assumeThat(T actual, Matcher<T> matcher) {
> >>         if (!matcher.matches(actual)) {
> >>             throw new AssumptionViolatedException(actual, matcher);
> >>         }
> >>     }
> >>
> >> //does AssumptionViolatedException ever produce test case failure?
> >>
> >> /**
> >>  * An exception class used to implement <i>assumptions</i> (state in
> >> which a
> >> given test
> >>  * is meaningful and should or should not be executed). A test for which
> >> an
> >> assumption
> >>  * fails should not generate a test case failure.
> >>  *
> >>  * @see org.junit.Assume
> >>  * @since 4.12
> >>  */
> >> @SuppressWarnings("deprecation")
> >> public class AssumptionViolatedException extends
> >> org.junit.internal.AssumptionViolatedException {
> >>     private static final long serialVersionUID = 1L;
> >>
> >>     /**
> >>      * An assumption exception with the given <i>actual</i> value and a
> >> <i>matcher</i> describing
> >>      * the expectation that failed.
> >>      */
> >>     public <T> AssumptionViolatedException(T actual, Matcher<T>
> matcher) {
> >>         super(actual, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with a message with the given
> <i>actual</i>
> >> value and a
> >>      * <i>matcher</i> describing the expectation that failed.
> >>      */
> >>     public <T> AssumptionViolatedException(String message, T expected,
> >> Matcher<T> matcher) {
> >>         super(message, expected, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message only.
> >>      */
> >>     public AssumptionViolatedException(String message) {
> >>         super(message);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message and a cause.
> >>      */
> >>     public AssumptionViolatedException(String assumption, Throwable t) {
> >>         super(assumption, t);
> >>     }
> >> }
> >>
> >> //will base class throw FAILURE for TestCase?
> >> /**
> >>  * An exception class used to implement <i>assumptions</i> (state in
> >> which a
> >> given test
> >>  * is meaningful and should or should not be executed). A test for which
> >> an
> >> assumption
> >>  * fails should not generate a test case failure.
> >>  *
> >>  * @see org.junit.Assume
> >>  */
> >> public class AssumptionViolatedException extends RuntimeException
> >> implements
> >> SelfDescribing {
> >>     private static final long serialVersionUID = 2L;
> >>
> >>     /*
> >>      * We have to use the f prefix until the next major release to
> ensure
> >>      * serialization compatibility.
> >>      * See https://github.com/junit-team/junit/issues/976
> >>      */
> >>     private final String fAssumption;
> >>     private final boolean fValueMatcher;
> >>     private final Object fValue;
> >>     private final Matcher<?> fMatcher;
> >>
> >>     /**
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption, boolean
> >> hasValue,
> >> Object value, Matcher<?> matcher) {
> >>         this.fAssumption = assumption;
> >>         this.fValue = value;
> >>         this.fMatcher = matcher;
> >>         this.fValueMatcher = hasValue;
> >>
> >>         if (value instanceof Throwable) {
> >>           initCause((Throwable) value);
> >>         }
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given <i>value</i> (String or
> >>      * Throwable) and an additional failing {@link Matcher}.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(Object value, Matcher<?>
> matcher) {
> >>         this(null, true, value, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given <i>value</i> (String or
> >>      * Throwable) and an additional failing {@link Matcher}.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption, Object value,
> >> Matcher<?> matcher) {
> >>         this(assumption, true, value, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message only.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption) {
> >>         this(assumption, false, null, null);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message and a cause.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption, Throwable e) {
> >>         this(assumption, false, null, null);
> >>         initCause(e);
> >>     }
> >>
> >>     @Override
> >>     public String getMessage() {
> >>         return StringDescription.asString(this);
> >>     }
> >>
> >>     public void describeTo(Description description) {
> >>         if (fAssumption != null) {
> >>             description.appendText(fAssumption);
> >>         }
> >>
> >>         if (fValueMatcher) {
> >>             // a value was passed in when this instance was constructed;
> >> print it
> >>             if (fAssumption != null) {
> >>                 description.appendText(": ");
> >>             }
> >>
> >>             description.appendText("got: ");
> >>             description.appendValue(fValue);
> >>
> >>             if (fMatcher != null) {
> >>                 description.appendText(", expected: ");
> >>                 description.appendDescriptionOf(fMatcher);
> >>             }
> >>         }
> >>     }
> >> }
> >>
> >> //at least with junit 4.12 if AssumptionViolatedException will not FAIL
> >> the
> >> test-case
> >> //AssumptionViolatedException falls between the exception-handlers to
> >> produce ERROR
> >>
> >> can you verify with junit author(s) AssumptionViolatedException  does
> not
> >> FAIL test-case is intended behaviour?
> >> martin
> >>
> >>
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: Markus KARG <[hidden email]>
> >> Sent: Sunday, August 16, 2020 1:39 PM
> >> To: 'Maven Developers List' <[hidden email]>
> >> Subject: AW: Assumption fail treated as unexcepted exception?!
> >>
> >> The debug output is quite huge, so I won't put it here. What in
> particular
> >> shall I lookup inside of that? The environment is:
> >>
> >> Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555;
> >> 2019-04-04T21:00:29+02:00)
> >> Maven home: C:\Program Files\apache-maven-3.6.1
> >> Java version: 1.8.0_212, vendor: Azul Systems, Inc., runtime: C:\Program
> >> Files\zulu8.38.0.13-ca-jdk8.0.212-win_x64\jre
> >> Default locale: de_DE, platform encoding: Cp1252
> >> OS name: "windows 8.1", version: "6.3", arch: "amd64", family: "windows"
> >> ...
> >> [DEBUG] Goal:
> >> org.apache.maven.plugins:maven-surefire-plugin:2.22.0:test
> (default-test)
> >> [DEBUG] Style:         Regular
> >> [DEBUG] Configuration: <?xml version="1.0" encoding="UTF-8"?>
> >> <configuration>
> >>
> >>
> >>
> <additionalClasspathElements>${maven.test.additionalClasspath}</additionalCl
> >> asspathElements>
> >>   <argLine>-Xmx384m</argLine>
> >>   <basedir default-value="${basedir}"/>
> >>   <childDelegation
> >> default-value="false">${childDelegation}</childDelegation>
> >>   <classesDirectory default-value="${project.build.outputDirectory}"/>
> >>
> >>
> >>
> <classpathDependencyExcludes>${maven.test.dependency.excludes}</classpathDep
> >> endencyExcludes>
> >>   <debugForkedProcess>${maven.surefire.debug}</debugForkedProcess>
> >>   <dependenciesToScan>${dependenciesToScan}</dependenciesToScan>
> >>   <disableXmlReport
> >> default-value="false">${disableXmlReport}</disableXmlReport>
> >>   <enableAssertions
> >> default-value="true">${enableAssertions}</enableAssertions>
> >>   <encoding
> >>
> >>
> default-value="${project.reporting.outputEncoding}">${surefire.encoding}</en
> >> coding>
> >>   <environmentVariables>
> >>     <JENKINS_MAVEN_AGENT_DISABLED>true</JENKINS_MAVEN_AGENT_DISABLED>
> >>   </environmentVariables>
> >>   <excludedGroups>${excludedGroups}</excludedGroups>
> >>   <excludesFile>${surefire.excludesFile}</excludesFile>
> >>
> >>
> >>
> <failIfNoSpecifiedTests>${surefire.failIfNoSpecifiedTests}</failIfNoSpecifie
> >> dTests>
> >>   <failIfNoTests>${failIfNoTests}</failIfNoTests>
> >>   <forkCount default-value="1">${forkCount}</forkCount>
> >>   <forkMode default-value="once">${forkMode}</forkMode>
> >>   <forkedProcessExitTimeoutInSeconds
> >>
> >>
> default-value="30">${surefire.exitTimeout}</forkedProcessExitTimeoutInSecond
> >> s>
> >>
> >>
> >>
> <forkedProcessTimeoutInSeconds>${surefire.timeout}</forkedProcessTimeoutInSe
> >> conds>
> >>   <groups>${groups}</groups>
> >>   <includesFile>${surefire.includesFile}</includesFile>
> >>   <junitArtifactName
> >> default-value="junit:junit">${junitArtifactName}</junitArtifactName>
> >>   <junitPlatformArtifactName
> >>
> >>
> default-value="org.junit.platform:junit-platform-engine">${junitPlatformArti
> >> factName}</junitPlatformArtifactName>
> >>   <jvm>${jvm}</jvm>
> >>   <localRepository default-value="${localRepository}"/>
> >>   <objectFactory>${objectFactory}</objectFactory>
> >>   <parallel>${parallel}</parallel>
> >>   <parallelMavenExecution default-value="${session.parallel}"/>
> >>   <parallelOptimized
> >> default-value="true">${parallelOptimized}</parallelOptimized>
> >>
> >>
> >>
> <parallelTestsTimeoutForcedInSeconds>${surefire.parallel.forcedTimeout}</par
> >> allelTestsTimeoutForcedInSeconds>
> >>
> >>
> >>
> <parallelTestsTimeoutInSeconds>${surefire.parallel.timeout}</parallelTestsTi
> >> meoutInSeconds>
> >>   <perCoreThreadCount
> >> default-value="true">${perCoreThreadCount}</perCoreThreadCount>
> >>   <pluginArtifactMap>${plugin.artifactMap}</pluginArtifactMap>
> >>   <pluginDescriptor default-value="${plugin}"/>
> >>   <printSummary
> >> default-value="true">${surefire.printSummary}</printSummary>
> >>   <projectArtifactMap>${project.artifactMap}</projectArtifactMap>
> >>   <projectBuildDirectory default-value="${project.build.directory}"/>
> >>   <redirectTestOutputToFile
> >>
> >>
> default-value="false">${maven.test.redirectTestOutputToFile}</redirectTestOu
> >> tputToFile>
> >>   <remoteRepositories
> >> default-value="${project.pluginArtifactRepositories}"/>
> >>   <reportFormat
> >> default-value="brief">${surefire.reportFormat}</reportFormat>
> >>   <reportNameSuffix
> >> default-value="">${surefire.reportNameSuffix}</reportNameSuffix>
> >>   <reportsDirectory
> >> default-value="${project.build.directory}/surefire-reports"/>
> >>   <rerunFailingTestsCount
> >>
> >>
> default-value="0">${surefire.rerunFailingTestsCount}</rerunFailingTestsCount
> >> >
> >>   <reuseForks default-value="true">${reuseForks}</reuseForks>
> >>   <runOrder default-value="filesystem">${surefire.runOrder}</runOrder>
> >>   <shutdown default-value="testset">${surefire.shutdown}</shutdown>
> >>   <skip default-value="false">${maven.test.skip}</skip>
> >>   <skipAfterFailureCount
> >>
> >>
> default-value="0">${surefire.skipAfterFailureCount}</skipAfterFailureCount>
> >>   <skipExec>${maven.test.skip.exec}</skipExec>
> >>   <skipTests default-value="false">${skipTests}</skipTests>
> >>   <suiteXmlFiles>${surefire.suiteXmlFiles}</suiteXmlFiles>
> >>   <systemPropertyVariables>
> >>     <maven.home>C:\Program Files\apache-maven-3.6.1</maven.home>
> >>   </systemPropertyVariables>
> >>   <tempDir default-value="surefire">${tempDir}</tempDir>
> >>   <test>${test}</test>
> >>   <testClassesDirectory
> >> default-value="${project.build.testOutputDirectory}"/>
> >>   <testFailureIgnore
> >> default-value="false">${maven.test.failure.ignore}</testFailureIgnore>
> >>   <testNGArtifactName
> >>
> >>
> default-value="org.testng:testng">${testNGArtifactName}</testNGArtifactName>
> >>   <testSourceDirectory
> >> default-value="${project.build.testSourceDirectory}"/>
> >>   <threadCount>${threadCount}</threadCount>
> >>   <threadCountClasses
> >> default-value="0">${threadCountClasses}</threadCountClasses>
> >>   <threadCountMethods
> >> default-value="0">${threadCountMethods}</threadCountMethods>
> >>   <threadCountSuites
> >> default-value="0">${threadCountSuites}</threadCountSuites>
> >>   <trimStackTrace
> default-value="true">${trimStackTrace}</trimStackTrace>
> >>   <useFile default-value="true">${surefire.useFile}</useFile>
> >>   <useManifestOnlyJar
> >> default-value="true">${surefire.useManifestOnlyJar}</useManifestOnlyJar>
> >>   <useSystemClassLoader
> >>
> >>
> default-value="true">${surefire.useSystemClassLoader}</useSystemClassLoader>
> >>   <useUnlimitedThreads
> >> default-value="false">${useUnlimitedThreads}</useUnlimitedThreads>
> >>   <workingDirectory>${basedir}</workingDirectory>
> >>   <project default-value="${project}"/>
> >>   <session default-value="${session}"/>
> >> </configuration>
> >> ...
> >> [ERROR]
> >>
> >>
> testProofClaim(org.apache.maven.plugins.dependency.fromConfiguration.TestCop
> >> yMojo)  Time elapsed: 0.146 s  <<< ERROR!
> >> org.junit.AssumptionViolatedException: always skip
> >>         at
> >>
> >>
> org.apache.maven.plugins.dependency.fromConfiguration.TestCopyMojo.testProof
> >> Claim(TestCopyMojo.java:76)
> >> ...
> >>
> >> IIUC, Maven debug doesn't contain any really specific information
> besides
> >> what I aready assumed: Surefire treats
> >> org.junit.AssumptionViolatedException
> >> as an error but not as a request to skip the test.
> >>
> >> This can be reproduced with this really simple test case:
> >>
> >>
> https://github.com/mkarg/maven-dependency-plugin/commit/af257d7987fc41ac4377
> >> 4d2dca60b201979d11a2
> >> <
> https://github.com/mkarg/maven-dependency-plugin/commit/af257d7987fc41ac43774d2dca60b201979d11a2
> >
> >> - as you can see, I just ask JUnit 4.13 to fail the
> >> assumptions, and JUnit CORRECTLY throws AssumptionViolatedException -
> but
> >> Maven seems to not specifically deal with it.
> >>
> >> -Markus
> >>
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Martin Gainty [mailto:[hidden email]]
> >> Gesendet: Sonntag, 16. August 2020 14:24
> >> An: Maven Developers List
> >> Betreff: Re: Assumption fail treated as unexcepted exception?!
> >>
> >> MG>below
> >>
> >> ________________________________
> >> From: Markus KARG <[hidden email]>
> >> Sent: Sunday, August 16, 2020 7:40 AM
> >> To: [hidden email] <[hidden email]>
> >> Subject: Assumption fail treated as unexcepted exception?!
> >>
> >> Guys,
> >>
> >>
> >>
> >> I'm stuck with working on a new feature due to this, so I hope you can
> >> help
> >> me quickly:
> >>
> >>
> >>
> >> JUnit knows assumptions, assertions and exceptions. Failing assertions
> >> will
> >> FAIL tests (hence will fail maven builds). Failing assumptions will SKIP
> >> tests (hence will pass maven builds). Exceptions will ERROR tests (hence
> >> will break maven builds). Unfortunately today I noticed that assumptions
> >> actually ERROR test (hence fail builds) in Maven!
> >>
> >>
> >>
> >> (I simply added another test to maven dependency plugin which contains
> an
> >> always-failing assumption to proof the claim)
> >>
> >>
> >>
> >> [INFO] Results:
> >>
> >> [INFO]
> >>
> >> [ERROR] Errors:
> >>
> >> [ERROR]   TestCopyMojo.proofClaim:274 ┐ AssumptionViolated always skip
> >>
> >> MG>get maven environment and debug information
> >> MG>mvn -e -X
> >> MG>also I know Junit 5.4.2 needs Hamcrest to be on classpath
> >> MG>can you tell us which version of Junit you are using?
> >>
> >> -Markus Karg
> >> ~gruss~
> >> ~martin~
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Assumption fail treated as unexcepted exception?!

Romain Manni-Bucau
In reply to this post by Tibor Digana
Think the answer had been written already but in case it is not clear - it
looks at least to me: it works as expected.
JUnit 3 does not support assumptions - even when executed with JUnit 4
launcher - exactly as JUnt Jupiter mimics JUnit 4 behavior when
executed with vintage engine.
Assumptions are JUnit >= 4.4 so if you want to use an assumption write a
JUnit 4 test and not a JUnit 3 one.

Hope it helps.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le mar. 18 août 2020 à 13:59, Martin Gainty <[hidden email]> a écrit :

> have you tried with junit4?
>
> if you need to stick with older junit then you can implement your
> AssumptionViolatedException to extend BuildFailureException
>
> any feedback from junit author on this behaviour?
>
> gruss
> ~martin~
>
>
> ________________________________
> From: Markus KARG <[hidden email]>
> Sent: Monday, August 17, 2020 1:53 PM
> To: 'Maven Developers List' <[hidden email]>
> Subject: AW: Assumption fail treated as unexcepted exception?!
>
> That doesn't make it any better, and it is no excuse that we do not simply
> catch that particular exception inside the Surefire plugin.
> So did we simply forget to implement this is the Surefire plugin (so I can
> add it), did we not implement it deliberately (so I shall not add it), or
> is it broken (so I shall fix it)?
> -Markus
>
> -----Ursprüngliche Nachricht-----
> Von: Tibor Digana [mailto:[hidden email]]
> Gesendet: Montag, 17. August 2020 18:37
> An: Maven Developers List
> Betreff: Re: Assumption fail treated as unexcepted exception?!
>
> This is the outcome in IDEA:
>
> "Tests failed: 1, passed: 0"
>
> So the behavior is the same with Maven.
>
> On Mon, Aug 17, 2020 at 6:35 PM Tibor Digana <[hidden email]>
> wrote:
>
> > Hi Markus,
> >
> > It is a specific problem related to the JUnit library because the test
> > fails in IntelliJ IDEA and in Maven as well.
> >
> > The JUnit4 assumptions fail with yellow markers in IDEA but here the
> > JUnit3' TestCase fails in red as a typical error or failure.
> > And Maven fails this test as follows but i think this behavior starts in
> > the junit library itself.
> >
> > [INFO] Running test.TestAssumptions
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 0.028 s <<< FAILURE! - in test.TestAssumptions
> > [ERROR] test.TestAssumptions.testABC  Time elapsed: 0.01 s  <<< ERROR!
> > org.junit.AssumptionViolatedException: got: <false>, expected: is <true>
> > at test/test.TestAssumptions.testABC(TestAssumptions.java:9)
> >
> > [INFO]
> > [INFO] Results:
> > [INFO]
> > [ERROR] Errors:
> > [ERROR]   TestAssumptions.testABC:9 » AssumptionViolated got: <false>,
> > expected: is <tru...
> > [INFO]
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> > [INFO]
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD FAILURE
> >
> > On Mon, Aug 17, 2020 at 3:07 PM Markus KARG <[hidden email]>
> > wrote:
> >
> >> Martin,
> >>
> >> I don't understand your question.
> >>
> >> You posted the Javadocs, and they tell it clearly: "A test for which an
> >> assumption fails should NOT generate a test case failure.".
> >>
> >> Also Assume's Javadocs
> >> (https://junit.org/junit4/javadoc/4.12/org/junit/Assume.html) are
> pretty
> >> clear about it: "... A failed assumption does NOT mean the code is
> broken,
> >> but that the test provides no useful information. Assume basically means
> >> "don't RUN this test if these conditions don't apply". The default JUnit
> >> runner SKIPS tests with failing assumptions..."
> >>
> >> So do you really want me to ask the JUnit people whether they really
> MEAN
> >> what they clearly wrote...?
> >>
> >> -Markus
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Martin Gainty [mailto:[hidden email]]
> >> Gesendet: Montag, 17. August 2020 12:27
> >> An: Maven Developers List
> >> Betreff: Re: Assumption fail treated as unexcepted exception?!
> >>
> >>
> >>    /**
> >>      * Call to assume that <code>actual</code> satisfies the condition
> >> specified by <code>matcher</code>.
> >>      * If not, the test halts and is ignored.
> >>      * Example:
> >>      * <pre>:
> >>      *   assumeThat(1, is(1)); // passes
> >>      *   foo(); // will execute
> >>      *   assumeThat(0, is(1)); // assumption failure! test halts
> >>      *   int x = 1 / 0; // will never execute
> >>      * </pre>
> >>      *
> >>      * @param <T> the static type accepted by the matcher (this can flag
> >> obvious compile-time problems such as {@code assumeThat(1, is("a"))}
> >>      * @param actual the computed value being compared
> >>      * @param matcher an expression, built of {@link Matcher}s,
> specifying
> >> allowed values
> >>      * @see org.hamcrest.CoreMatchers
> >>      * @see org.junit.matchers.JUnitMatchers
> >>      */
> >>     public static <T> void assumeThat(T actual, Matcher<T> matcher) {
> >>         if (!matcher.matches(actual)) {
> >>             throw new AssumptionViolatedException(actual, matcher);
> >>         }
> >>     }
> >>
> >> //does AssumptionViolatedException ever produce test case failure?
> >>
> >> /**
> >>  * An exception class used to implement <i>assumptions</i> (state in
> >> which a
> >> given test
> >>  * is meaningful and should or should not be executed). A test for which
> >> an
> >> assumption
> >>  * fails should not generate a test case failure.
> >>  *
> >>  * @see org.junit.Assume
> >>  * @since 4.12
> >>  */
> >> @SuppressWarnings("deprecation")
> >> public class AssumptionViolatedException extends
> >> org.junit.internal.AssumptionViolatedException {
> >>     private static final long serialVersionUID = 1L;
> >>
> >>     /**
> >>      * An assumption exception with the given <i>actual</i> value and a
> >> <i>matcher</i> describing
> >>      * the expectation that failed.
> >>      */
> >>     public <T> AssumptionViolatedException(T actual, Matcher<T>
> matcher) {
> >>         super(actual, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with a message with the given
> <i>actual</i>
> >> value and a
> >>      * <i>matcher</i> describing the expectation that failed.
> >>      */
> >>     public <T> AssumptionViolatedException(String message, T expected,
> >> Matcher<T> matcher) {
> >>         super(message, expected, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message only.
> >>      */
> >>     public AssumptionViolatedException(String message) {
> >>         super(message);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message and a cause.
> >>      */
> >>     public AssumptionViolatedException(String assumption, Throwable t) {
> >>         super(assumption, t);
> >>     }
> >> }
> >>
> >> //will base class throw FAILURE for TestCase?
> >> /**
> >>  * An exception class used to implement <i>assumptions</i> (state in
> >> which a
> >> given test
> >>  * is meaningful and should or should not be executed). A test for which
> >> an
> >> assumption
> >>  * fails should not generate a test case failure.
> >>  *
> >>  * @see org.junit.Assume
> >>  */
> >> public class AssumptionViolatedException extends RuntimeException
> >> implements
> >> SelfDescribing {
> >>     private static final long serialVersionUID = 2L;
> >>
> >>     /*
> >>      * We have to use the f prefix until the next major release to
> ensure
> >>      * serialization compatibility.
> >>      * See https://github.com/junit-team/junit/issues/976
> >>      */
> >>     private final String fAssumption;
> >>     private final boolean fValueMatcher;
> >>     private final Object fValue;
> >>     private final Matcher<?> fMatcher;
> >>
> >>     /**
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption, boolean
> >> hasValue,
> >> Object value, Matcher<?> matcher) {
> >>         this.fAssumption = assumption;
> >>         this.fValue = value;
> >>         this.fMatcher = matcher;
> >>         this.fValueMatcher = hasValue;
> >>
> >>         if (value instanceof Throwable) {
> >>           initCause((Throwable) value);
> >>         }
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given <i>value</i> (String or
> >>      * Throwable) and an additional failing {@link Matcher}.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(Object value, Matcher<?>
> matcher) {
> >>         this(null, true, value, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given <i>value</i> (String or
> >>      * Throwable) and an additional failing {@link Matcher}.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption, Object value,
> >> Matcher<?> matcher) {
> >>         this(assumption, true, value, matcher);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message only.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption) {
> >>         this(assumption, false, null, null);
> >>     }
> >>
> >>     /**
> >>      * An assumption exception with the given message and a cause.
> >>      *
> >>      * @deprecated Please use {@link
> >> org.junit.AssumptionViolatedException}
> >> instead.
> >>      */
> >>     @Deprecated
> >>     public AssumptionViolatedException(String assumption, Throwable e) {
> >>         this(assumption, false, null, null);
> >>         initCause(e);
> >>     }
> >>
> >>     @Override
> >>     public String getMessage() {
> >>         return StringDescription.asString(this);
> >>     }
> >>
> >>     public void describeTo(Description description) {
> >>         if (fAssumption != null) {
> >>             description.appendText(fAssumption);
> >>         }
> >>
> >>         if (fValueMatcher) {
> >>             // a value was passed in when this instance was constructed;
> >> print it
> >>             if (fAssumption != null) {
> >>                 description.appendText(": ");
> >>             }
> >>
> >>             description.appendText("got: ");
> >>             description.appendValue(fValue);
> >>
> >>             if (fMatcher != null) {
> >>                 description.appendText(", expected: ");
> >>                 description.appendDescriptionOf(fMatcher);
> >>             }
> >>         }
> >>     }
> >> }
> >>
> >> //at least with junit 4.12 if AssumptionViolatedException will not FAIL
> >> the
> >> test-case
> >> //AssumptionViolatedException falls between the exception-handlers to
> >> produce ERROR
> >>
> >> can you verify with junit author(s) AssumptionViolatedException  does
> not
> >> FAIL test-case is intended behaviour?
> >> martin
> >>
> >>
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: Markus KARG <[hidden email]>
> >> Sent: Sunday, August 16, 2020 1:39 PM
> >> To: 'Maven Developers List' <[hidden email]>
> >> Subject: AW: Assumption fail treated as unexcepted exception?!
> >>
> >> The debug output is quite huge, so I won't put it here. What in
> particular
> >> shall I lookup inside of that? The environment is:
> >>
> >> Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555;
> >> 2019-04-04T21:00:29+02:00)
> >> Maven home: C:\Program Files\apache-maven-3.6.1
> >> Java version: 1.8.0_212, vendor: Azul Systems, Inc., runtime: C:\Program
> >> Files\zulu8.38.0.13-ca-jdk8.0.212-win_x64\jre
> >> Default locale: de_DE, platform encoding: Cp1252
> >> OS name: "windows 8.1", version: "6.3", arch: "amd64", family: "windows"
> >> ...
> >> [DEBUG] Goal:
> >> org.apache.maven.plugins:maven-surefire-plugin:2.22.0:test
> (default-test)
> >> [DEBUG] Style:         Regular
> >> [DEBUG] Configuration: <?xml version="1.0" encoding="UTF-8"?>
> >> <configuration>
> >>
> >>
> >>
> <additionalClasspathElements>${maven.test.additionalClasspath}</additionalCl
> >> asspathElements>
> >>   <argLine>-Xmx384m</argLine>
> >>   <basedir default-value="${basedir}"/>
> >>   <childDelegation
> >> default-value="false">${childDelegation}</childDelegation>
> >>   <classesDirectory default-value="${project.build.outputDirectory}"/>
> >>
> >>
> >>
> <classpathDependencyExcludes>${maven.test.dependency.excludes}</classpathDep
> >> endencyExcludes>
> >>   <debugForkedProcess>${maven.surefire.debug}</debugForkedProcess>
> >>   <dependenciesToScan>${dependenciesToScan}</dependenciesToScan>
> >>   <disableXmlReport
> >> default-value="false">${disableXmlReport}</disableXmlReport>
> >>   <enableAssertions
> >> default-value="true">${enableAssertions}</enableAssertions>
> >>   <encoding
> >>
> >>
> default-value="${project.reporting.outputEncoding}">${surefire.encoding}</en
> >> coding>
> >>   <environmentVariables>
> >>     <JENKINS_MAVEN_AGENT_DISABLED>true</JENKINS_MAVEN_AGENT_DISABLED>
> >>   </environmentVariables>
> >>   <excludedGroups>${excludedGroups}</excludedGroups>
> >>   <excludesFile>${surefire.excludesFile}</excludesFile>
> >>
> >>
> >>
> <failIfNoSpecifiedTests>${surefire.failIfNoSpecifiedTests}</failIfNoSpecifie
> >> dTests>
> >>   <failIfNoTests>${failIfNoTests}</failIfNoTests>
> >>   <forkCount default-value="1">${forkCount}</forkCount>
> >>   <forkMode default-value="once">${forkMode}</forkMode>
> >>   <forkedProcessExitTimeoutInSeconds
> >>
> >>
> default-value="30">${surefire.exitTimeout}</forkedProcessExitTimeoutInSecond
> >> s>
> >>
> >>
> >>
> <forkedProcessTimeoutInSeconds>${surefire.timeout}</forkedProcessTimeoutInSe
> >> conds>
> >>   <groups>${groups}</groups>
> >>   <includesFile>${surefire.includesFile}</includesFile>
> >>   <junitArtifactName
> >> default-value="junit:junit">${junitArtifactName}</junitArtifactName>
> >>   <junitPlatformArtifactName
> >>
> >>
> default-value="org.junit.platform:junit-platform-engine">${junitPlatformArti
> >> factName}</junitPlatformArtifactName>
> >>   <jvm>${jvm}</jvm>
> >>   <localRepository default-value="${localRepository}"/>
> >>   <objectFactory>${objectFactory}</objectFactory>
> >>   <parallel>${parallel}</parallel>
> >>   <parallelMavenExecution default-value="${session.parallel}"/>
> >>   <parallelOptimized
> >> default-value="true">${parallelOptimized}</parallelOptimized>
> >>
> >>
> >>
> <parallelTestsTimeoutForcedInSeconds>${surefire.parallel.forcedTimeout}</par
> >> allelTestsTimeoutForcedInSeconds>
> >>
> >>
> >>
> <parallelTestsTimeoutInSeconds>${surefire.parallel.timeout}</parallelTestsTi
> >> meoutInSeconds>
> >>   <perCoreThreadCount
> >> default-value="true">${perCoreThreadCount}</perCoreThreadCount>
> >>   <pluginArtifactMap>${plugin.artifactMap}</pluginArtifactMap>
> >>   <pluginDescriptor default-value="${plugin}"/>
> >>   <printSummary
> >> default-value="true">${surefire.printSummary}</printSummary>
> >>   <projectArtifactMap>${project.artifactMap}</projectArtifactMap>
> >>   <projectBuildDirectory default-value="${project.build.directory}"/>
> >>   <redirectTestOutputToFile
> >>
> >>
> default-value="false">${maven.test.redirectTestOutputToFile}</redirectTestOu
> >> tputToFile>
> >>   <remoteRepositories
> >> default-value="${project.pluginArtifactRepositories}"/>
> >>   <reportFormat
> >> default-value="brief">${surefire.reportFormat}</reportFormat>
> >>   <reportNameSuffix
> >> default-value="">${surefire.reportNameSuffix}</reportNameSuffix>
> >>   <reportsDirectory
> >> default-value="${project.build.directory}/surefire-reports"/>
> >>   <rerunFailingTestsCount
> >>
> >>
> default-value="0">${surefire.rerunFailingTestsCount}</rerunFailingTestsCount
> >> >
> >>   <reuseForks default-value="true">${reuseForks}</reuseForks>
> >>   <runOrder default-value="filesystem">${surefire.runOrder}</runOrder>
> >>   <shutdown default-value="testset">${surefire.shutdown}</shutdown>
> >>   <skip default-value="false">${maven.test.skip}</skip>
> >>   <skipAfterFailureCount
> >>
> >>
> default-value="0">${surefire.skipAfterFailureCount}</skipAfterFailureCount>
> >>   <skipExec>${maven.test.skip.exec}</skipExec>
> >>   <skipTests default-value="false">${skipTests}</skipTests>
> >>   <suiteXmlFiles>${surefire.suiteXmlFiles}</suiteXmlFiles>
> >>   <systemPropertyVariables>
> >>     <maven.home>C:\Program Files\apache-maven-3.6.1</maven.home>
> >>   </systemPropertyVariables>
> >>   <tempDir default-value="surefire">${tempDir}</tempDir>
> >>   <test>${test}</test>
> >>   <testClassesDirectory
> >> default-value="${project.build.testOutputDirectory}"/>
> >>   <testFailureIgnore
> >> default-value="false">${maven.test.failure.ignore}</testFailureIgnore>
> >>   <testNGArtifactName
> >>
> >>
> default-value="org.testng:testng">${testNGArtifactName}</testNGArtifactName>
> >>   <testSourceDirectory
> >> default-value="${project.build.testSourceDirectory}"/>
> >>   <threadCount>${threadCount}</threadCount>
> >>   <threadCountClasses
> >> default-value="0">${threadCountClasses}</threadCountClasses>
> >>   <threadCountMethods
> >> default-value="0">${threadCountMethods}</threadCountMethods>
> >>   <threadCountSuites
> >> default-value="0">${threadCountSuites}</threadCountSuites>
> >>   <trimStackTrace
> default-value="true">${trimStackTrace}</trimStackTrace>
> >>   <useFile default-value="true">${surefire.useFile}</useFile>
> >>   <useManifestOnlyJar
> >> default-value="true">${surefire.useManifestOnlyJar}</useManifestOnlyJar>
> >>   <useSystemClassLoader
> >>
> >>
> default-value="true">${surefire.useSystemClassLoader}</useSystemClassLoader>
> >>   <useUnlimitedThreads
> >> default-value="false">${useUnlimitedThreads}</useUnlimitedThreads>
> >>   <workingDirectory>${basedir}</workingDirectory>
> >>   <project default-value="${project}"/>
> >>   <session default-value="${session}"/>
> >> </configuration>
> >> ...
> >> [ERROR]
> >>
> >>
> testProofClaim(org.apache.maven.plugins.dependency.fromConfiguration.TestCop
> >> yMojo)  Time elapsed: 0.146 s  <<< ERROR!
> >> org.junit.AssumptionViolatedException: always skip
> >>         at
> >>
> >>
> org.apache.maven.plugins.dependency.fromConfiguration.TestCopyMojo.testProof
> >> Claim(TestCopyMojo.java:76)
> >> ...
> >>
> >> IIUC, Maven debug doesn't contain any really specific information
> besides
> >> what I aready assumed: Surefire treats
> >> org.junit.AssumptionViolatedException
> >> as an error but not as a request to skip the test.
> >>
> >> This can be reproduced with this really simple test case:
> >>
> >>
> https://github.com/mkarg/maven-dependency-plugin/commit/af257d7987fc41ac4377
> >> 4d2dca60b201979d11a2
> >> <
> https://github.com/mkarg/maven-dependency-plugin/commit/af257d7987fc41ac43774d2dca60b201979d11a2
> >
> >> - as you can see, I just ask JUnit 4.13 to fail the
> >> assumptions, and JUnit CORRECTLY throws AssumptionViolatedException -
> but
> >> Maven seems to not specifically deal with it.
> >>
> >> -Markus
> >>
> >>
> >> -----Ursprüngliche Nachricht-----
> >> Von: Martin Gainty [mailto:[hidden email]]
> >> Gesendet: Sonntag, 16. August 2020 14:24
> >> An: Maven Developers List
> >> Betreff: Re: Assumption fail treated as unexcepted exception?!
> >>
> >> MG>below
> >>
> >> ________________________________
> >> From: Markus KARG <[hidden email]>
> >> Sent: Sunday, August 16, 2020 7:40 AM
> >> To: [hidden email] <[hidden email]>
> >> Subject: Assumption fail treated as unexcepted exception?!
> >>
> >> Guys,
> >>
> >>
> >>
> >> I'm stuck with working on a new feature due to this, so I hope you can
> >> help
> >> me quickly:
> >>
> >>
> >>
> >> JUnit knows assumptions, assertions and exceptions. Failing assertions
> >> will
> >> FAIL tests (hence will fail maven builds). Failing assumptions will SKIP
> >> tests (hence will pass maven builds). Exceptions will ERROR tests (hence
> >> will break maven builds). Unfortunately today I noticed that assumptions
> >> actually ERROR test (hence fail builds) in Maven!
> >>
> >>
> >>
> >> (I simply added another test to maven dependency plugin which contains
> an
> >> always-failing assumption to proof the claim)
> >>
> >>
> >>
> >> [INFO] Results:
> >>
> >> [INFO]
> >>
> >> [ERROR] Errors:
> >>
> >> [ERROR]   TestCopyMojo.proofClaim:274 ┐ AssumptionViolated always skip
> >>
> >> MG>get maven environment and debug information
> >> MG>mvn -e -X
> >> MG>also I know Junit 5.4.2 needs Hamcrest to be on classpath
> >> MG>can you tell us which version of Junit you are using?
> >>
> >> -Markus Karg
> >> ~gruss~
> >> ~martin~
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>