title: Spark load data and add filename as dataframe column
body: <p>I am loading some data into Spark with a wrapper function:</p>
  
  
  
  <pre class="lang-py prettyprint-override"><code>def load_data( filename ):
      df = sqlContext.read.format("com.databricks.spark.csv")\
          .option("delimiter", "\t")\
          .option("header", "false")\
          .option("mode", "DROPMALFORMED")\
          .load(filename)
      # add the filename base as hostname
      ( hostname, _ ) = os.path.splitext( os.path.basename(filename) )
      ( hostname, _ ) = os.path.splitext( hostname )
      df = df.withColumn('hostname', lit(hostname))
      return df
  </code></pre>
  
  <p>specifically, I am using a glob to load a bunch of files at once:</p>
  
  <pre class="lang-py prettyprint-override"><code>df = load_data( '/scratch/*.txt.gz' )
  </code></pre>
  
  <p>the files are:</p>
  
  <pre class="lang-py prettyprint-override"><code>/scratch/host1.txt.gz
  /scratch/host2.txt.gz
  ...
  </code></pre>
  
  <p>I would like the column 'hostname' to actually contain the real name of the file being loaded rather than the glob (ie <code>host1</code>, <code>host2</code> etc, rather than <code>*</code>).</p>
  
  <p>How can I do this?</p>