title: Jenkins, Junit and Flaky Test Handler plugin
body: <p>I'm trying to get some visibility of flaky tests running in Jenkins.</p>
  
  <p>In order to do so I've installed <a href="https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin" rel="nofollow noreferrer">https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin</a> which seems to have this purpose.</p>
  
  <p>All the requirements specified in the plugin page are met by my installation (Jenkins version, Junit plugin, etc...)</p>
  
  <p>After having installed the plugin, the suggest to </p>
  
  <blockquote>
    <p>check on “Publish JUnit flaky test reports” under “Additional test
    report features.</p>
  </blockquote>
  
  <p>and their screenshot shows</p>
  
  <p><a href="https://i.stack.imgur.com/MCZ2y.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MCZ2y.png" alt="enter image description here"></a></p>
  
  <p>My <em>Maven Project</em> configuration does not give me the option of specifying the <em>Test report XMLs</em>, and looks like this instead:</p>
  
  <p><a href="https://i.stack.imgur.com/vKYvN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vKYvN.png" alt="enter image description here"></a></p>
  
  <p>Although, when I run the Jenkins build for the configured project, all I get is an empty table:</p>
  
  <p><a href="https://i.stack.imgur.com/shnl5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/shnl5.png" alt="enter image description here"></a></p>
  
  <p>The <em>Build</em> section of the project configuration is using the <code>-Dsurefire.rerunFailingTestsCount=3</code> option.</p>
  
  <p>I'm a bit confused whether the plugin is not behaving with my Jenkins installation or I'm missing something.</p>
  
  <p><strong><em>EDIT</em></strong>: I've been trying version 1.0.3 which contains the fix mentioned in one of the answers, and at the end of the job execution I get the following error:</p>
  
  <pre><code>ERROR: Publisher 'Publish JUnit test result report' aborted due to exception: 
  java.io.IOException: Unable to serialize com.google.jenkins.flakyTestHandler.junit.FlakyTestResult@55c1d6e9
      at hudson.remoting.UserRequest.serialize(UserRequest.java:169)
      at hudson.remoting.UserRequest.perform(UserRequest.java:129)
      at hudson.remoting.UserRequest.perform(UserRequest.java:49)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      at ......remote call to some.slave.com(Native Method)
      at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
      at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
      at hudson.remoting.Channel.call(Channel.java:778)
      at com.google.jenkins.flakyTestHandler.plugin.JUnitFlakyTestDataPublisher.getTestData(JUnitFlakyTestDataPublisher.java:49)
      at hudson.tasks.junit.TestDataPublisher.contributeTestData(TestDataPublisher.java:62)
      at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:166)
      at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
      at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
      at hudson.model.Build$BuildExecution.post2(Build.java:185)
      at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
      at hudson.model.Run.execute(Run.java:1766)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:98)
      at hudson.model.Executor.run(Executor.java:408)
  Caused by: java.io.NotSerializableException: com.google.jenkins.flakyTestHandler.junit.FlakyCaseResult$FlakyRunInformation
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
      at java.util.ArrayList.writeObject(ArrayList.java:762)
      at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:483)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
      at java.util.ArrayList.writeObject(ArrayList.java:762)
      at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:483)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
      at java.util.ArrayList.writeObject(ArrayList.java:762)
      at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:483)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
      at hudson.remoting.UserRequest._serialize(UserRequest.java:158)
      at hudson.remoting.UserRequest.serialize(UserRequest.java:167)
      at hudson.remoting.UserRequest.perform(UserRequest.java:129)
      at hudson.remoting.UserRequest.perform(UserRequest.java:49)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>This seem to happen only when the job is executed on a slave, it seem to work fine on master.</p>
  
  <p><strong><em>EDIT 2</em></strong>: 1.0.4 fixed the issue above.</p>
  
  <p><strong><em>EDIT 3</em></strong>: Even though 1.0.4 fixed the serialisation issue, the plugin reporting still seem to be broken. In the following situation for example:</p>
  
  <p><a href="https://i.stack.imgur.com/bM3Pq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bM3Pq.png" alt="wrong reporting"></a></p>
  
  <ul>
  <li>job #21 ran successfully</li>
  <li>job #22 ran and had one flaky tests, which passed on the second run</li>
  <li>job #23 ran successfully</li>
  </ul>
  
  <p>The report of the project is saying that there is one flaky test with one Flakes, 0 Fails and 0 Passes. I would expect the 2 Passes, 0 Fails and 1 Flakes instead.</p>
  
  <p>Am I missing something?</p>