title: Printing Unique Values in a Column as a Percentage of Total Rows
body: <p>For one of the data cleaning steps, I would like to gather insight into how the unique values are existing as a percentage of the total row count so that I can apply a threshold and decide if I should completely remove this column / feature. For this I came up with this function as below:</p>
  <pre><code>def uniqueValuesAsPercentage(data: DataFrame) = {
    val (rowCount, columnCount) = shape(data)
    data.selectExpr(data.head().getValuesMap[Long](data.columns).map(elem =&gt; {
        val (columnName, uniqueCount) = elem
        val percentage = uniqueCount / rowCount * 100
        (columnName, uniqueCount, percentage) 
    }))
  }
  </code></pre>
  <p>But it fails with the following error:</p>
  <pre><code>&lt;console&gt;:90: error: type mismatch;
   found   : scala.collection.immutable.Iterable[(String, Long, Long)]
   required: String
           data.selectExpr(data.head().getValuesMap[Long](data.columns).map(elem =&gt; {
  </code></pre>
  <p>Unfortunately since this is in the Apache Zeppelin notebook, I'm also missing the capabilities of an IDE. I have the IntelliJ untilate, but the Big Data Tools support seem not to be available for my version of the IDE. Very annoying!</p>
  <p>Any ideas as to what the problem is here? I guess I'm messing with the DataFrame in the selectExpr(....). As it can be seen, that I'm returning a tuple with the information I calculate.</p>