title: Spark sql Dataframe - import sqlContext.implicits._
body: <p>I have main that creates spark context:</p>
  
  <pre><code>    val sc = new SparkContext(sparkConf)
      val sqlContext = new org.apache.spark.sql.SQLContext(sc)
      import sqlContext.implicits._
  </code></pre>
  
  <p>Then creates dataframe and does filters and validations on the dataframe.</p>
  
  <pre><code>    val convertToHourly = udf((time: String) =&gt; time.substring(0, time.indexOf(':')) + ":00:00")
  
      val df = sqlContext.read.schema(struct).format("com.databricks.spark.csv").load(args(0))
      // record length cannot be &lt; 2 
      .na.drop(3)
      // round to hours
      .withColumn("time",convertToHourly($"time"))
  </code></pre>
  
  <p>This works great.</p>
  
  <p>BUT When I try moving my validations to another file by sending the dataframe to          </p>
  
  <pre><code>function ValidateAndTransform(df: DataFrame) : DataFrame = {...}
  </code></pre>
  
  <p>that gets the Dataframe &amp; does the validations and transformations: It seems like I need the </p>
  
  <pre><code> import sqlContext.implicits._
  </code></pre>
  
  <blockquote>
    <p>To avoid the error:    “value $ is not a member of StringContext”
    that happens on line:
             .withColumn("time",convertToHourly(<strong>$</strong>"time"))</p>
  </blockquote>
  
  <p>But to use the  <code>import sqlContext.implicits._</code>
  I also need the <code>sqlContext</code> either defined in the new file like so:</p>
  
  <pre><code>val sc = new SparkContext(sparkConf)
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  </code></pre>
  
  <p>or send it to the </p>
  
  <pre><code>function ValidateAndTransform(df: DataFrame) : DataFrame = {...}
  function
  </code></pre>
  
  <p>I feel like the separation I'm trying to do to 2 files (main &amp; validation) is not done correctly...</p>
  
  <p>Any idea on how to design this? Or simply send the sqlContext to the function?</p>
  
  <p>Thanks!</p>