title: Different floating point precision from RDD and DataFrame
body: <p>I changed an RDD to DataFrame and compared the results with another DataFrame which I imported using read.csv but the floating point precision  is not the same from the two approaches. I appreciate your help.</p>
  
  <p>The data I am using is from <a href="https://github.com/dgadiraju/data/tree/master/retail_db" rel="noreferrer">here</a>.</p>
  
  <pre><code>from pyspark.sql import Row
  from pyspark.sql.types import *
  </code></pre>
  
  <h2>RDD way</h2>
  
  <pre><code>orders = sc.textFile("retail_db/orders")
  order_items = sc.textFile('retail_db/order_items')
  orders_comp = orders.filter(lambda line: ((line.split(',')[-1] == 'CLOSED') or  (line.split(',')[-1] == 'COMPLETE')))
  orders_compMap = orders_comp.map(lambda line: (int(line.split(',')[0]), line.split(',')[1]))
  
  order_itemsMap = order_items.map(lambda line: (int(line.split(',')[1]), 
                                             (int(line.split(',')[2]), float(line.split(',')[4])) ))
  
   joined = orders_compMap.join(order_itemsMap)
   joined2 = joined.map(lambda line: ((line[1][0], line[1][1][0]), line[1][1][1]))
  
  joined3 = joined2.reduceByKey(lambda a, b : a +b).sortByKey()
  
  df1 = joined3.map(lambda x:Row(date = x[0][0], product_id = x[0][1], total  = x[1])).toDF().select(['date','product_id', 'total'])
  </code></pre>
  
  <h2>DataFrame</h2>
  
  <pre><code> schema = StructType([StructField('order_id', IntegerType(), True),
                  StructField('date', StringType(), True),
                  StructField('customer_id', StringType(), True),
                   StructField('status', StringType(), True)])
  
  
   orders2 = spark.read.csv("retail_db/orders",schema = schema)
  
  
   schema = StructType([StructField('item_id', IntegerType(), True),
                  StructField('order_id', IntegerType(), True),
                  StructField('product_id', IntegerType(), True),
                   StructField('quantity', StringType(), True),
                   StructField('sub_total', FloatType(), True),
                   StructField('product_price', FloatType(), True)])
  
  
  
  orders_items2 = spark.read.csv("retail_db/order_items", schema = schema)
  
  orders2.registerTempTable("orders2t")
  orders_items2.registerTempTable("orders_items2t")
  
   df2 = spark.sql('select o.date, oi.product_id, sum(oi.sub_total)  \
        as total from  orders2t as o inner join orders_items2t as oi on 
        o.order_id = oi.order_id \
        where o.status in ("CLOSED", "COMPLETE") group by o.date, 
       oi.product_id order by  o.date, oi.product_id')
  </code></pre>
  
  <h2>Are they the same?</h2>
  
  <pre><code>df1.registerTempTable("df1t")
  df2.registerTempTable("df2t")
  
   spark.sql("select d1.total - d2.total as difference from df1t as d1 inner 
   join df2t as d2 on d1.date = d2.date \
   and d1.product_id =d2.product_id ").show(truncate = False)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/pXUIm.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pXUIm.png" alt="enter image description here"></a></p>