title: Understanding spark physical plan
body: <p>I'm trying to understand physical plans on spark but I'm not understanding some parts because they seem different from traditional rdbms. For example, in this plan below, it's a plan about a query over a hive table. The query is this:</p>
  
  <pre><code>select
          l_returnflag,
          l_linestatus,
          sum(l_quantity) as sum_qty,
          sum(l_extendedprice) as sum_base_price,
          sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,
          sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,
          avg(l_quantity) as avg_qty,
          avg(l_extendedprice) as avg_price,
          avg(l_discount) as avg_disc,
          count(*) as count_order
      from
          lineitem
      where
          l_shipdate &lt;= '1998-09-16'
      group by
          l_returnflag,
          l_linestatus
      order by
          l_returnflag,
          l_linestatus;
  
  
  == Physical Plan ==
  Sort [l_returnflag#35 ASC,l_linestatus#36 ASC], true, 0
  +- ConvertToUnsafe
     +- Exchange rangepartitioning(l_returnflag#35 ASC,l_linestatus#36 ASC,200), None
        +- ConvertToSafe
           +- TungstenAggregate(key=[l_returnflag#35,l_linestatus#36], functions=[(sum(l_quantity#31),mode=Final,isDistinct=false),(sum(l_extendedpr#32),mode=Final,isDistinct=false),(sum((l_extendedprice#32 * (1.0 - l_discount#33))),mode=Final,isDistinct=false),(sum(((l_extendedprice#32 * (1.0l_discount#33)) * (1.0 + l_tax#34))),mode=Final,isDistinct=false),(avg(l_quantity#31),mode=Final,isDistinct=false),(avg(l_extendedprice#32),mode=Fl,isDistinct=false),(avg(l_discount#33),mode=Final,isDistinct=false),(count(1),mode=Final,isDistinct=false)], output=[l_returnflag#35,l_linestatus,sum_qty#0,sum_base_price#1,sum_disc_price#2,sum_charge#3,avg_qty#4,avg_price#5,avg_disc#6,count_order#7L])
              +- TungstenExchange hashpartitioning(l_returnflag#35,l_linestatus#36,200), None
                 +- TungstenAggregate(key=[l_returnflag#35,l_linestatus#36], functions=[(sum(l_quantity#31),mode=Partial,isDistinct=false),(sum(l_exdedprice#32),mode=Partial,isDistinct=false),(sum((l_extendedprice#32 * (1.0 - l_discount#33))),mode=Partial,isDistinct=false),(sum(((l_extendedpri32 * (1.0 - l_discount#33)) * (1.0 + l_tax#34))),mode=Partial,isDistinct=false),(avg(l_quantity#31),mode=Partial,isDistinct=false),(avg(l_extendedce#32),mode=Partial,isDistinct=false),(avg(l_discount#33),mode=Partial,isDistinct=false),(count(1),mode=Partial,isDistinct=false)], output=[l_retulag#35,l_linestatus#36,sum#64,sum#65,sum#66,sum#67,sum#68,count#69L,sum#70,count#71L,sum#72,count#73L,count#74L])
                    +- Project [l_discount#33,l_linestatus#36,l_tax#34,l_quantity#31,l_extendedprice#32,l_returnflag#35]
                       +- Filter (l_shipdate#37 &lt;= 1998-09-16)
                          +- HiveTableScan [l_discount#33,l_linestatus#36,l_tax#34,l_quantity#31,l_extendedprice#32,l_shipdate#37,l_returnflag#35], astoreRelation default, lineitem, None
  </code></pre>
  
  <p>For what I'm understanding in the plan is:</p>
  
  <ol>
  <li><p>First starts with a Hive table scan</p></li>
  <li><p>Then it filter using where the condition</p></li>
  <li><p>Then project to get the columns we want</p></li>
  <li><p>Then TungstenAggregate?</p></li>
  <li><p>Then TungstenExchange?</p></li>
  <li><p>Then TungstenAggregate again?</p></li>
  <li><p>Then ConvertToSafe?</p></li>
  <li><p>Then sorts the final result</p></li>
  </ol>
  
  <p>But I'm not understanding the 4, 5, 6 and 7 steps. Do you know what they are? I'm looking for information about this so I can understand the plan but I'm not finding anything concrete.</p>