title: Comparing Previous data with Current data in spark scala
body: <p>I want to compare Prev.data with Current data by monthly . I am having data like below.</p>
  
  <pre><code>Data-set 1 : (Prev)                             Data-set 2 : (Latest)
  
     Year-month  Sum-count                 Year-Month    Sum-count
        --          --                       201808          48     
       201807       30                       201807          22   
       201806       20                       201806          20
       201805       35                       201805          20
       201804       12                       201804           9
       201803       15                       --              --
  </code></pre>
  
  <p>I have data sets like as shown above. I want to compare the both data sets based on year-month column and sum-count and need to find out difference in percentage. </p>
  
  <p>I am using spark 2.3.0 and Scala 2.11.</p>
  
  <p>Here is mode :</p>
  
  <pre><code>import org.apache.spark.sql.functions.lag
  
  val mdf = spark.read.format("csv").
            option("InferSchema","true").
            option("header","true").
            option("delimiter",",").
            option("charset","utf-8").
            load("c:\\test.csv")
  mdf.createOrReplaceTempView("test")
  val res= spark.sql("select year-month,SUM(Sum-count) as SUM_AMT from test d group by year-month")
  val win = org.apache.spark.sql.expressions.Window.orderBy("data_ym")
  
  val res1 = res.withColumn("Prev_month", lag("SUM_AMT", 1,0).over(win)).withColumn("percentage",col("Prev_month") / sum("SUM_AMT").over()).show()
  </code></pre>
  
  <p>I need output like this : </p>
  
  <p>if percentage is more than 10% then i need to set flag as F.</p>
  
  <pre><code>set1     cnt             set2    cnt     output(Percentage)  Flag
  201807   30             201807   22         7%                T
  201806   20             201806   20         0%                T
  201805   35             201805   20         57%               F
  </code></pre>
  
  <p>Please help me on this.</p>