title: Hive Query failing with Heap Issue
body: <p>Below is the hive query am running,</p>
  
  <p>INSERT INTO TABLE temp.table_output</p>
  
  <pre><code>SELECT /*+ STREAMTABLE(tableB) */ c.column1 as client, a.column2 as testData, 
      CASE WHEN ca.updated_date IS NULL OR ca.updated_date = 'null' THEN null ELSE CONCAT(ca.updated_date, '+0000') END as update
      FROM temp.tableA as a 
      INNER JOIN default.tableB as ca ON a.column5=ca.column2
      INNER JOIN default.tableC as c ON ca.column3=c.column1 WHERE a.name='test';
  </code></pre>
  
  <p>TableB is having 2.4 billion rows (140 GB), TableA and TableC is having 200 million records around.</p>
  
  <p>Cluster consists of 3 Cassandra data nodes and 3 Analytics node (Hive on top of cassandra), with 130GB ram on each node.</p>
  
  <p>TableA, TableB, TableC are hive internal tables.</p>
  
  <p>Hive cluster heap size is 12GB.</p>
  
  <p>Can someone tell me when I run hive query, am having heap issue and it fails to complete the task?? Its the only job running on hive server.</p>
  
  <p>Task fails with below error,</p>
  
  <pre><code>Caused by: java.io.IOException: Read failed from file: cfs://172.31.x.x/tmp/hive-root/hive_2015-03-17_00-27-25_132_17376615815827139-1/-mr-10002/000049_0
      at com.datastax.bdp.hadoop.cfs.CassandraInputStream.read(CassandraInputStream.java:178)
      at java.io.DataInputStream.readFully(DataInputStream.java:195)
      at java.io.DataInputStream.readFully(DataInputStream.java:169)
      at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
      at org.apache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1486)
      at org.apache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1475)
      at org.apache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1470)
      at org.apache.hadoop.mapred.SequenceFileRecordReader.&lt;init&gt;(SequenceFileRecordReader.java:43)
      at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
      at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.&lt;init&gt;(CombineHiveRecordReader.java:65)
      ... 16 more
  
  Caused by: java.io.IOException: org.apache.thrift.TApplicationException: Internal error processing get_remote_cfs_sblock
      at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveSubBlock(CassandraFileSystemThriftStore.java:537)
      at com.datastax.bdp.hadoop.cfs.CassandraSubBlockInputStream.subBlockSeekTo(CassandraSubBlockInputStream.java:145)
      at com.datastax.bdp.hadoop.cfs.CassandraSubBlockInputStream.read(CassandraSubBlockInputStream.java:95)
      at com.datastax.bdp.hadoop.cfs.CassandraInputStream.read(CassandraInputStream.java:159)
      ... 25 more
  
  Caused by: org.apache.thrift.TApplicationException: Internal error processing get_remote_cfs_sblock
      at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
      at org.apache.cassandra.thrift.Dse$Client.recv_get_remote_cfs_sblock(Dse.java:271)
      at org.apache.cassandra.thrift.Dse$Client.get_remote_cfs_sblock(Dse.java:254)
      at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at com.datastax.bdp.util.CassandraProxyClient.invokeDseClient(CassandraProxyClient.java:655)
      at com.datastax.bdp.util.CassandraProxyClient.invoke(CassandraProxyClient.java:631)
      at com.sun.proxy.$Proxy5.get_remote_cfs_sblock(Unknown Source)
      at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveSubBlock(CassandraFileSystemThriftStore.java:515)
      ... 28 more
  </code></pre>
  
  <p>Hive.log</p>
  
  <pre><code>2015-03-17 23:10:39,576 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_r_000023 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,579 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_r_000052 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,582 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_m_000207 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,585 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_r_000087 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,588 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_m_000223 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,591 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_m_000045 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,594 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_m_000235 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,597 ERROR exec.Task (SessionState.java:printError(419)) - Examining task ID: task_201503171816_0036_m_002140 (and more) from job job_201503171816_0036
  2015-03-17 23:10:39,761 ERROR exec.Task (SessionState.java:printError(419)) - 
  Task with the most failures(4): 
  -----
  Task ID:
    task_201503171816_0036_m_000036
  
  URL:
    http://sjvtncasl064.mcafee.int:50030/taskdetails.jsp?jobid=job_201503171816_0036&amp;tipid=task_201503171816_0036_m_000036
  -----
  Diagnostic Messages for this Task:
  Error: Java heap space
  
  2015-03-17 23:10:39,777 ERROR ql.Driver (SessionState.java:printError(419)) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
  </code></pre>