title: How to calculate a value for each row?
body: <p>I have an input dataframe (created from a hive table) containing more than 100 rows. For each row of the dataframe, I need to extract the column values (most strings) and pass those values to a user-defined function. For each row, the function uses these input values and other intermediate dataframes (created from hive tables) to calculate a set of rows and stores in a result dataframe. 
  How do I achieve this - please help. </p>
  
  <p>I tried this:</p>
  
  <pre><code>var df1= hiveContext.sql("Select event_date,channelcode,st,tc,startsec,endsec from program_master")
  var count1=df1.count()
  df1 = df1.withColumn("INDEX", monotonically_increasing_id())
  var i=1
  while (i &lt;= count1){
    var ed = df1.filter(df1("INDEX") === s"""$i""").select(to_date(unix_timestamp(df1("ed"), "dd-MM-yy").cast(TimestampType)).cast(DateType)).first().getDate(0)
    var cc = df1.filter(df1("INDEX") === s"""$i""").select(df1("cc")).first().getInt(0)
    var ST = df1.filter(df1("INDEX") === s"""$i""").select(df1("ST")).first().getString(0)
    var TC = df1.filter(df1("INDEX") === s"""$i""").select(df1("TC")).first().getString(0)
    var ss = df1.filter(df1("INDEX") === s"""$i""").select(df1("ss")).first().getInt(0)
    var es = df1.filter(df1("INDEX") === s"""$i""").select(df1("es")).first().getInt(0)
    calculate_values(ed, cc, st, tc, ss, ss, sparkSession)
    i=i+1
  }
  </code></pre>
  
  <p>calculate_values def</p>
  
  <pre><code>def calculate_values(ed: Date,cc:Integer,ST:String,TC:String,ss:Integer,ss:Integer,sparkSession: SparkSession):Unit=
  </code></pre>
  
  <p>2 problems in what I tried: hence do not have an output
  line 3: I expected it to give numbers like 1,2,3,......100.... to iterate using i - but its generating random very large numbers. 
  line 5: It throws java.util.NoSuchElementException: next on empty iterator</p>