title: Insert words directly to table in db rather than storing in arraylist using Java
body: <p>I am building an indexing program where I extract a file (PDF) and extract all the words inside and stored it in an <code>arrayList</code>. At the same time, I have to define my word token to see what sort of words to be indexed and its rule , hence I stored them in an arrayList so I could replace the regular expression to suit my needs.</p>
  
  <p>Code:</p>
  
  <pre><code>public void index(String path) throws Exception {
      ArrayList&lt;String&gt; list = new ArrayList&lt;String&gt;();
      PDDocument document = PDDocument.load(new File(path));
  
      if (!document.isEncrypted()) {
          PDFTextStripper tStripper = new PDFTextStripper();
          String pdfFileInText = tStripper.getText(document);
          String lines[] = pdfFileInText.split("\\r?\\n");
          for (String line : lines) {
              String[] words = line.split(" ");
  
              for (String word : words) {
                  //check if one/more special characters at end of string then remove OR
                  //check special characters in beginning of the string then remove
  
                  list.add(word.replaceAll("([\\W]+$)|(^[\\W]+)", ""));
  
              }
          }
      }
  
      String[] words1 = list.toArray(new String[list.size()]);
      String sql = "insert IGNORE into  test.indextable123 values (?,?)";
      preparedStatement = con.connect().prepareStatement(sql);
  
      for (int i = 1; i &lt; words1.length; i++) {
          preparedStatement.setString(1, words1[i]);
          preparedStatement.setString(2, path);
          preparedStatement.addBatch();
  
          if (i % 1000 == 0) {
              preparedStatement.executeBatch();
              System.out.print("Add Thousand");
          }
      }
  
      if (words1.length % 1000 &gt; 0) {
          preparedStatement.executeBatch();
          System.out.print("Add Remaining");
      }
  
      preparedStatement.close();
      System.out.println("Completed");
  }
  </code></pre>
  
  <p>The problem here is if I want to index a file with over 10 million words, it's not resource efficient to store in the arrayList and it's going to throw an <code>outofmemory exception</code> too.</p>
  
  <p>At the same time, I need to store it in an array to "replace" with the regular expression as shown in the code. Is there a way to insert the words directly to the db right after extracting while filtering the words to suit the regular expression I need?</p>