title: How to do a Spark dataframe(1 million rows) cartesian product with a list(1000 entries) efficiently to generate a new dataframe with 1 billion rows
body: <p>I want to take each row of a dataframe which has 1 million rows and generate 1000 rows from each row of it by taking a cross product with a list having 1000 entries thereby generating a dataframe with 1 billion rows. What is the best approach to do it efficiently.
  I have tried with broadcasting the list and then using it while mapping each row of the dataframe. But this seems to be taking too much time.</p>
  
  <pre><code> val mappedrdd = validationDataFrames.map(x =&gt; {
       val cutoffList : List[String] = cutoffListBroadcast.value
       val arrayTruthTableVal = arrayTruthTableBroadcast.value
  
       var listBufferRow: ListBuffer[Row] = new ListBuffer()
  
         for(cutOff &lt;- cutoffList){
            val conversion = x.get(0).asInstanceOf[Int]
            val probability = x.get(1).asInstanceOf[Double]
  
            var columnName : StringBuffer = new StringBuffer
            columnName = columnName.append(conversion)
  
            if(probability &gt; cutOff.toDouble){
                columnName = columnName.append("_").append("1")
            }else{
                columnName = columnName.append("_").append("0")
            }
            val index:Int  = arrayTruthTableVal.indexOf(columnName.toString)
            var listBuffer : ListBuffer[String] = new ListBuffer()
            listBuffer :+= cutOff
  
            for(i &lt;- 1 to 4){
               if((index + 1) == i) listBuffer :+= "1" else listBuffer :+= "0"
            }
            val row = Row.fromSeq(listBuffer)
            listBufferRow = listBufferRow :+ row
          }
  
        listBufferRow
  
       })
  </code></pre>