title: Handle surrogates with pandas
body: <p>When saving the data</p>
  
  <pre><code>data.to_csv(outp_file, encoding='utf-8')
  </code></pre>
  
  <p>I sometimes get errors like this</p>
  
  <blockquote>
    <p>UnicodeEncodeError: 'utf-8' codec can't encode characters in position
    233-234: surrogates not allowed</p>
  </blockquote>
  
  <p>In python3 you can simply replace such characters </p>
  
  <pre><code>&gt;&gt; "abc\udc34xyz".encode('utf-8', 'replace').decode('utf-8')
  'abc?xyz'
  </code></pre>
  
  <p>But here I have a dataframe with N rows and M columns. It's fine for me to remove rows with surrogates, but it's not fine to skip the whole dataframe. </p>
  
  <p>The problem is I don't know in which rows and in which columns they are.</p>
  
  <p>I am looking for a solution that could be applied in following way</p>
  
  <pre><code>try:
     data.to_csv(outp_file, encoding='utf-8')
  except UnicodeEncodeError:
     # process data and save it without surrogates...
  </code></pre>
  
  <p>Any help?</p>