title: How to pre-process data before pandas.read_csv()
body: <p>I have a slightly broken CSV file that I want to pre-process before reading it with pandas.read_csv(), i.e. do some search/replace on it.</p>
  
  <p>I tried to open the file and and do the pre-processing in a generator, that I then hand over to read_csv():</p>
  
  <pre><code>    def in_stream():
      with open("some.csv") as csvfile:
          for line in csvfile:
              l = re.sub(r'","',r',',line)
              yield l
  
      df = pd.read_csv(in_stream())
  </code></pre>
  
  <p>Sadly, this just throws a </p>
  
  <pre><code>ValueError: Invalid file path or buffer object type: &lt;class 'generator'&gt;
  </code></pre>
  
  <p>Although, when looking at Panda's source, I'd expect it to be able to work on iterators, thus generators.</p>
  
  <p>I only found this [article] (<a href="https://stackoverflow.com/questions/46310030/using-a-custom-object-in-pandas-read-csv">Using a custom object in pandas.read_csv()</a>), outlining how to wrap a generator into a file-like object, but it seems to only work on files in byte-mode.</p>
  
  <p>So in the end I'm looking for a pattern to build a pipeline that opens a file, reads it line-by-line, allows pre-processing and then feeds it into e.g. pandas.read_csv(). </p>