title: Multi threaded application on server slower than single threaded (unlike in JUnit tests)
body: <p>I've switched in my application from a single to a multi threaded routine.</p>
  
  <p>This works pretty fine in the JUnit tests. When running it with <code>10</code> threads, the test needs <code>195</code> ms to complete and when running it with only one thread the application takes <code>406</code> ms to finish. So there clearly is a performance advantage.</p>
  
  <p>But when running it on the server, the application now needs much longer than when it was only single threaded.</p>
  
  <p>Basically, my application reads a line in a csv file, puts one of its value in a set and prints the line to another file.
  The size of the input file in the JUnit tests is about <code>35</code> lines long, the one on the server about <code>6 000 000</code> lines long.</p>
  
  <p>The set in which those values are put is a synchronized <code>HashSet</code> which can contain <code>Long</code> objects.</p>
  
  <p>I'm monitoring my application with the Java VisualVM but unfortunately I don't know what to look for.</p>
  
  <p>Do you have any hints for me on how to solve this performance crisis?</p>
  
  <hr>
  
  <p>P. S.: Most of the time my threads are marked as waiting, but I don't know if they are really waiting or if they are just too fast for the Java VisualVM to display it.</p>
  
  <hr>
  
  <p>To further clarify my routine: I read the file single threaded, but as soon as the line is read I pass the resulting object to a <code>Runnable</code> that puts it into a set and prints it into another file. Meanwhile the next lines are read and passed to other threads.</p>
  
  <hr>
  
  <p>As I can see it in my log file, the threads are doing something and aren't just waiting. But there are certain <em>jumps</em>, periods longer than <code>100</code> ms where nothing is happening.</p>
  
  <hr>
  
  <p>One of those <em>jumps:</em></p>
  
  <pre><code>2011-04-08 12:27:16,580 DEBUG [Thread-10]  runnables.Runner - 7070927
  2011-04-08 12:27:16,580 DEBUG [Thread-10]  runnables.Runner - 9058759
  2011-04-08 12:27:16,580 DEBUG [Thread-10]  runnables.Runner - 7030928
  2011-04-08 12:27:16,580 DEBUG [Thread-10]  runnables.Runner - 15301035
  2011-04-08 12:27:16,684 DEBUG [Thread-10]  runnables.Runner - 7700929
  2011-04-08 12:27:16,684 DEBUG [Thread-10]  runnables.Runner - 17116545
  2011-04-08 12:27:16,685 DEBUG [Thread-10]  runnables.Runner - 4933581
  2011-04-08 12:27:16,685 DEBUG [Thread-10]  runnables.Runner - 2861116
  </code></pre>
  
  <p>Note: No GC happened at that time.</p>
  
  <hr>
  
  <p>As written in a comment below: I am using a threadpool. My threads are fighting* over the same output file. They all write to a <code>synchronized</code> method.</p>
  
  <hr>
  
  <p>Even if I reduce the size of my tread pool to one, the performance is still horrible. Nothing compared to the previous implementation. Wouldn't that rule out things like IO dependency or thread switching?</p>
  
  <hr>
  
  <p>I've modified my code now so that inside the <code>Runnable</code> nearly nothing is done. No <code>Set</code>, no writing. Just one log statement. And still I get those <code>jumps</code>.
   So I rule out the writing or <code>Set</code> problem proposed by some. And when running only one thread, I also got these idle times. So thread switiching also doesn't seem to be the problem.</p>