title: Java & MySQL Performance Issue
body: <p>So I have a bit of a performance problem. I have made a java program that constructs a database. The problem is when loading in the data. I am loading in 5,000 files into a sql Database. When the program starts off, it can process about 10% of the files in 10 minutes however it gets much slower as it progresses. Currently at 28% it is going to finish in 16 hours at its current rate. However that rate is slowing down considerably.</p>
  
  <p>My question is why does the program get progressively slower as it runs and how to fix that. </p>
  
  <p>EDIT: I have two versions. One is threaded (capped at 5 threads) and one is not. The difference between the two is negligible. I can post the code again if any one likes, but I took it out because I am now fairly certain that the bottle neck is the MySQL (Also appropriately re tagged). I went ahead and used batch inserts. This did cause an initial increase in speed but once again after processing about 30% of the data it does drop of quickly. </p>
  
  <p>So SQL Points</p>
  
  <ol>
  <li>My Engine for all 64 tables is InnoDB version 10.</li>
  <li>The table have about 300k rows at this point (~30% of the data)</li>
  <li>All tables have one "joint" primary key. A id and a date.</li>
  <li>Looking at MySQL WorkBench I see that there is a query per thread (5 queries)</li>
  <li>I am not sure the unit of time (Just reading from MySQL Administrator), but the queries to check if a file is already inserted are taking `300. (This query should be fast as it is a SELECT MyIndex from MyTable Limit 1 to 1 where Date = date.) As I have been starting and stopping the program I built in this check to see if the file was already inserted. That way I am able to start it after each change and see what if any improvement there is without starting the process again.</li>
  <li>I am fairly certain that the degradation of preformance is related to the tables' sizes. (I can stop and start the program now and the process remains slow. It is only when the tables are small that the process is going at an acceptable speed.)</li>
  <li>Please, please ask and I will post what ever information you need.</li>
  </ol>
  
  <p>DONE! Well I just let it run for the 4 Days it needed to. Thank you all for the help. </p>
  
  <p>Cheers,</p>
  
  <p>--Orlan</p>