title: Compare Huge XML Rows with Database Table Records - Custom Requirement
body: <h2>Problem</h2>
  
  <p>We have an XML like (its having some non unicode which needs to be filtered of) data,</p>
  
  <pre><code>&lt;row&gt;&lt;div&gt;1234&lt;/div&gt;&lt;dept&gt;ABCD&lt;/dept&gt;&lt;/row&gt;
  &lt;row&gt;&lt;div&gt;5678&lt;/div&gt;&lt;dept&gt;EFGH&lt;/dept&gt;&lt;/row&gt;
  </code></pre>
  
  <p><em>Just mentioning only 2 column tags for ease of understanding. Actually it has more than 20 column tags in each </em></p>
  
  <p>XML data is directly inserted as records into an Oracle schema table as,</p>
  
  <pre><code>div_c  qdept
  1234   ABCD
  5678   EFGH
  </code></pre>
  
  <h2>More information</h2>
  
  <ol>
  <li>XML file is more than 9 Gigs and available in FTP.</li>
  <li>Database table column names might be different from XML column tag names.</li>
  <li>Might have to add/define some Rules to filter out the rows.</li>
  </ol>
  
  <h2>Question</h2>
  
  <p>What would be the appropriate way to parse this huge XML and find out whether that record exists in the database table? Any open source tools available to utilize?</p>
  
  <h2>What Am Trying</h2>
  
  <ul>
  <li>Wrote StAX parser using default implementation(XMLInputFactory) with Invalid characters fiter (FilterReader)</li>
  <li>Planning to split the XML as chunks</li>
  <li>Have concurrent threads processing each of the chunks </li>
  <li>Each thread will generate a query to check whether that exists in database or not (i know its absurd)</li>
  <li>Have a connection pool created and execute those queries by each of the thread</li>
  </ul>
  
  <p>I know this is really worst what I am doing and it will take years to complete, I really need some advice on this like whether to go with any ORM to make the checking easy and make the XML parsing fast.</p>
  
  <p>Some suggestions like that would really help me.</p>