title: Maven returns exit 0 even though junit5 tests fail
body: <p>I have setup Maven to trigger some junit5 tests.</p>
  
  <p>When I run:</p>
  
  <pre><code>mvn clean integration-test
  </code></pre>
  
  <p>I see the project build and then it also runs the tests. I see a failure in one of the unit tests.</p>
  
  <p>Further down in the console output, I see:</p>
  
  <pre><code>[INFO] BUILD SUCCESS
  </code></pre>
  
  <p>What am I doing wrong? My expectation is that a failed test run would stop a tool like maven and return exit code 1.</p>
  
  <p>This is particularly frustrating in CI because it appears as the tests are healthy since exit code returns zero.</p>
  
  <p>EDIT: Here's the POM XML:</p>
  
  <pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"&gt;
      &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
      &lt;groupId&gt;com.github.(removedforstackoverflowpost)&lt;/groupId&gt;
      &lt;artifactId&gt;(removedforstackoverflowpost)&lt;/artifactId&gt;
      &lt;packaging&gt;jar&lt;/packaging&gt;
      &lt;version&gt;0.0.1&lt;/version&gt;
      &lt;name&gt;(removedforstackoverflowpost)&lt;/name&gt;
      &lt;url&gt;(removedforstackoverflowpost)&lt;/url&gt;
      &lt;description&gt;(removedforstackoverflowpost)&lt;/description&gt;
      &lt;licenses&gt;
          &lt;license&gt;
              &lt;name&gt;MIT License&lt;/name&gt;
              &lt;url&gt;http://www.opensource.org/licenses/mit-license.php&lt;/url&gt;
          &lt;/license&gt;
      &lt;/licenses&gt;
      &lt;developers&gt;
          &lt;developer&gt;
              &lt;name&gt;(removedforstackoverflowpost)&lt;/name&gt;
              &lt;email&gt;(removedforstackoverflowpost)&lt;/email&gt;
              &lt;organization&gt;(removedforstackoverflowpost)&lt;/organization&gt;
              &lt;organizationUrl&gt;(removedforstackoverflowpost)&lt;/organizationUrl&gt;
          &lt;/developer&gt;
      &lt;/developers&gt;
      &lt;scm&gt;
          &lt;connection&gt;(removedforstackoverflowpost)&lt;/connection&gt;
          &lt;developerConnection&gt;(removedforstackoverflowpost)&lt;/developerConnection&gt;
          &lt;url&gt;(removedforstackoverflowpost)&lt;/url&gt;
      &lt;/scm&gt;
      &lt;properties&gt;
          &lt;maven.compiler.source&gt;1.6&lt;/maven.compiler.source&gt;
          &lt;maven.compiler.target&gt;1.6&lt;/maven.compiler.target&gt;
          &lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourceEncoding&gt;
      &lt;/properties&gt;
      &lt;dependencies&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.junit.jupiter&lt;/groupId&gt;
              &lt;artifactId&gt;junit-jupiter-api&lt;/artifactId&gt;
              &lt;version&gt;5.3.1&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.google.code.gson&lt;/groupId&gt;
              &lt;artifactId&gt;gson&lt;/artifactId&gt;
              &lt;version&gt;2.8.5&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.googlecode.json-simple&lt;/groupId&gt;
              &lt;artifactId&gt;json-simple&lt;/artifactId&gt;
              &lt;version&gt;1.1&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;javassist&lt;/groupId&gt;
              &lt;artifactId&gt;javassist&lt;/artifactId&gt;
              &lt;version&gt;3.12.1.GA&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.junit.platform&lt;/groupId&gt;
              &lt;artifactId&gt;junit-platform-console-standalone&lt;/artifactId&gt;
              &lt;version&gt;1.3.2&lt;/version&gt;
          &lt;/dependency&gt;
      &lt;/dependencies&gt;
      &lt;build&gt;
          &lt;plugins&gt;
              &lt;plugin&gt;
                  &lt;artifactId&gt;maven-surefire-plugin&lt;/artifactId&gt;
                  &lt;!-- latest version (2.20.1) does not work well with JUnit5 --&gt;
                  &lt;version&gt;2.19.1&lt;/version&gt;
                  &lt;dependencies&gt;
                      &lt;dependency&gt;
                          &lt;groupId&gt;org.junit.platform&lt;/groupId&gt;
                          &lt;artifactId&gt;junit-platform-surefire-provider&lt;/artifactId&gt;
                          &lt;version&gt;1.0.3&lt;/version&gt;
                      &lt;/dependency&gt;
                      &lt;dependency&gt;
                          &lt;groupId&gt;org.junit.jupiter&lt;/groupId&gt;
                          &lt;artifactId&gt;junit-jupiter-engine&lt;/artifactId&gt;
                          &lt;version&gt;5.0.3&lt;/version&gt;
                      &lt;/dependency&gt;
                  &lt;/dependencies&gt;
                  &lt;configuration&gt;
                      &lt;useSystemClassLoader&gt;false&lt;/useSystemClassLoader&gt;
                      &lt;properties&gt;
                          &lt;configurationParameters&gt;
                              junit.jupiter.extensions.autodetection.enabled = true
                          &lt;/configurationParameters&gt;
                      &lt;/properties&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
                  &lt;version&gt;3.8.0&lt;/version&gt;
                  &lt;configuration&gt;
                      &lt;release&gt;11&lt;/release&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-surefire-plugin&lt;/artifactId&gt;
                  &lt;version&gt;2.19&lt;/version&gt;
                  &lt;configuration&gt;
                      &lt;testFailureIgnore&gt;true&lt;/testFailureIgnore&gt;
                      &lt;argLine&gt;
                          --illegal-access=permit
                      &lt;/argLine&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
          &lt;/plugins&gt;
      &lt;/build&gt;
  &lt;/project&gt;
  </code></pre>
  
  <p>I'm running java 11</p>
  
  <p>When I enter this test command:</p>
  
  <pre><code>mvn -ff clean integration-test
  </code></pre>
  
  <p>I get this output that ends like this:</p>
  
  <pre><code>Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in com.github.(removedforstackoverflowpost)
  
  Results :
  
  Failed tests:
    SomeTest.testSomething1:13 expected: &lt;1&gt; but was: &lt;2&gt;
  
  Tests run: 4, Failures: 1, Errors: 0, Skipped: 0
  
  [ERROR] There are test failures.
  
  Please refer to (removedforstackoverflowpost)/target/surefire-reports for the individual test results.
  [INFO]
  [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ (removedforstackoverflowpost) ---
  [INFO] Building jar: (removedforstackoverflowpost)....-0.0.1.jar
  [INFO] ------------------------------------------------------------------------
  [INFO] BUILD SUCCESS
  [INFO] ------------------------------------------------------------------------
  [INFO] Total time:  3.360 s
  [INFO] Finished at: 2018-12-12T00:45:03-08:00
  [INFO] ------------------------------------------------------------------------
  </code></pre>
  
  <p>I'm a total beginner so I apologize for asking an overly simplistic question. I am new to Java/Maven/JUnit5, but I'm just looking for what I should do to have the test cause a non zero exit code. This way, my CI run (I happen to use CircleCI but that doesnt matter) would show as failed when the tests fail.</p>
  
  <p>I work on a Mac and the undesired behavior that I'm describing on this ticket happens the same way on both my local Mac workstation as well as in CircleCI.</p>