title: How to convert an RDD (that read in a directory of text files) into dataFrame in Apache Spark in Scala?
body: <p>I'm developing a Scala feature extracting app using Apache Spark TF-IDF. I need to read in from a directory of text files. I'm trying to convert an RDD to a dataframe but I'm getting the error "value toDF() is not a member of org.apache.spark.rdd.RDD[streamedRDD]". This is what I have right now ...</p>
  
  <p>I have spark-2.2.1 &amp; Scala 2.1.11. Thanks in advance.</p>
  
  <p>Code:</p>
  
  <pre><code>// Creating the Spark context that will interface with Spark
  val conf = new SparkConf()
             .setMaster("local")
             .setAppName("TextClassification")
  val sc = new SparkContext(conf)
  
  // Load documents (one per line)
  val data = sc.wholeTextFiles("C:/Users/*")    
  val text = data.map{case(filepath,text) =&gt; text}    
  val id = data.map{case(filepath, text) =&gt; text.split("@").takeRight(1)(0)}
  
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  import sqlContext.implicits._
  
  case class dataStreamed(id: String, input: String)
  
  val tweetsDF = data
                .map{case (filepath, text) =&gt; 
                     val id = text.split("@").takeRight(1)(0)
                     val input = text.split(":").takeRight(2)(0)
                     dataStreamed(id, input)}
                .as[dataStreamed]
                .toDF()
                .cache()
  
  // -------------------- TF-IDF --------------------
  // From spark.apache.org
  // URL http://spark.apache.org/docs/latest/ml-features.html#tf-idf
  
  val tokenizer = new Tokenizer().setInputCol("input").setOutputCol("words")
  val wordsData = tokenizer.transform(tweetsDF)
  val hashingTF = new HashingTF()
                  .setInputCol("words")
                  .setOutputCol("rawFeatures")
  
  val tf = hashingTF.transform(wordsData).cache()  // Hashed words
  
  // Compute for the TFxIDF    
  val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features")
  val tfidf = idf.fit(tf)
  </code></pre>
  
  <p>Data: (Text files like these in a folder is what I need read in) 
        <a href="https://www.dropbox.com/s/cw3okhaosu7i1md/cars.txt?dl=0" rel="nofollow noreferrer">https://www.dropbox.com/s/cw3okhaosu7i1md/cars.txt?dl=0</a>
        <a href="https://www.dropbox.com/s/29tgqg7ifpxzwwz/Italy.txt?dl=0" rel="nofollow noreferrer">https://www.dropbox.com/s/29tgqg7ifpxzwwz/Italy.txt?dl=0</a></p>