title: Spark refresh cache table asynchronously
body: <p>We have cached tables in Spark Thrift Server. Data is cached on weekly basis:</p>
  
  <pre><code>cache table event_2018_12_1 as select * from ... where ...;
  cache table event_2018_12_2 as select * from ... where ...;
  </code></pre>
  
  <p>Since we're constantly updating our source data (Cassandra), we need to refresh the cache:</p>
  
  <pre><code>refresh table event_2018_12_1;
  select count(*) from event_2018_12_1;
  
  refresh table event_2018_12_2;
  select count(*) from event_2018_12_2
  </code></pre>
  
  <p>Refresh operation is lazy, so I need to trigger the refresh with <code>count(*)</code> operation.</p>
  
  <p><strong>The problem is when other clients selects data from cached table in the same time its being refreshed</strong> - The select hangs until the cache is fully refreshed (which takes minutes to complete).</p>
  
  <p>I'd like to refresh the cache asynchronously and expose the fresh data only when loading is finished (similar to guava's <code>LoadingCache</code> behaviour).</p>
  
  <p>How can I achieve this in Spark ?</p>
  
  <p><strong>Possible workaround:</strong></p>
  
  <p>Create a view as layer above the cached tables:</p>
  
  <pre><code>create or replace TEMPORARY VIEW event_2018_10 as 
  select * from event_2018_12_1
  union all select * from event_2018_12_2
  </code></pre>
  
  <p>Instead of refresh just create a new cache and replace the view:</p>
  
  <pre><code>cache table event_2018_12_1_c1 as select * from ... where ...;
  
  create or replace TEMPORARY VIEW event_2018_10 as 
  select * from event_2018_12_1_c1
  union all select * from event_2018_12_2;
  
  --after some delay drop the first cache
  drop table event_2018_12_1;
  </code></pre>
  
  <p>I'm concerned about changing the view's DDL and having selects in the same time. Can something go wrong ?</p>