title: Get value from Pyspark Column and compare it to a Python dictionary
body: <p>So I have a pyspark dataframe that I want to add another column to using the value from the Section_1 column and find its corresponding value in a python dictionary. So basically use the value from the Section_1 cell as the key and then fill in the value from the python dictionary in the new column like below.</p>
  <p><strong>Original dataframe</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">DataId</th>
  <th style="text-align: left;">ObjId</th>
  <th style="text-align: left;">Name</th>
  <th style="text-align: left;">Object</th>
  <th style="text-align: left;">Section_1</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">My data</td>
  <td style="text-align: left;">Data name</td>
  <td style="text-align: left;">Object name</td>
  <td style="text-align: left;">rd.111</td>
  <td style="text-align: left;">rd.123</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Python Dictionary</p>
  <pre><code>object_map= {'rd.123' : 'rd.567'}
  </code></pre>
  <p>Where section 1 has a value of rd.123 and I will search in the dictionary for the key 'rd.123' and want to return that value of rd.567 and place that in the new column</p>
  <p><strong>Desired DataFrame</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">DataId</th>
  <th style="text-align: left;">ObjId</th>
  <th style="text-align: left;">Name</th>
  <th style="text-align: left;">Object</th>
  <th style="text-align: left;">Section_1</th>
  <th style="text-align: left;">Section_2</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">My data</td>
  <td style="text-align: left;">Data name</td>
  <td style="text-align: left;">Object name</td>
  <td style="text-align: left;">rd.111</td>
  <td style="text-align: left;">rd.123</td>
  <td style="text-align: left;">rd.567</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Right now I got this error with my current code and I dont really know what I did wrong as I am not to familiar with pyspark</p>
  <blockquote>
  <p>There is an incorrect call to a Column object in your code. Please
  review your code.</p>
  </blockquote>
  <p>Here is my code that I am currently using where object_map is the python dictionary.</p>
  <pre><code>test_df = output.withColumn('Section_2', object_map.get(output.Section_1.collect()))
  </code></pre>