title: How can I create a parquet file bigger than node's assigned memory?
body: <p>I'm trying to create a parquet file from a table stored in mysql. The source contains millions of rows and I get a GC Overhead limit exception after a couple of minutes.</p>
  
  <p>Can apache drill be configured in a way that allows operations to use disk temporarily in case there is no more RAM available?</p>
  
  <p>This were my steps before getting the error:</p>
  
  <ul>
  <li>Put the mysql jdbc connector inside jars/3rdparty</li>
  <li>Execute sqlline.bat -u "jdbc:drill:zk=local"</li>
  <li>Navigate to <a href="http://localhost:8047/storage" rel="nofollow noreferrer">http://localhost:8047/storage</a></li>
  <li>Configure a new storage pluggin to connect to mysql</li>
  <li>Navigate to <a href="http://localhost:8047/query" rel="nofollow noreferrer">http://localhost:8047/query</a> and execute the following queries</li>
  <li>ALTER SESSION SET `store.format` = 'parquet';</li>
  <li>ALTER SESSION SET `store.parquet.compression` = 'snappy';</li>
  <li>create table dfs.tmp.`bigtable.parquet` as (select * from mysql.schema.bigtable)</li>
  </ul>
  
  <p>Then I get the error and the aplication ends:</p>
  
  <p>Node ran out of Heap memory, exiting.
  java.lang.OutOfMemoryError: GC overhead limit exceeded
          at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2149)
          at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956)
          at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3308)
          at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463)
          at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032)
          at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280)
          at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
          at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546)
          at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504)
          at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370)
          at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
          at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
          at org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(JdbcRecordReader.java:177)
          at org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:101)
          at org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:128)
          at org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:40)
          at org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:33)
          at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151)
          at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
          at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
          at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
          at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
          at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
          at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
          at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
          at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:105)
          at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79)
          at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230)
          at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)</p>