Merge 2 csv files based on a common field using python
<p>I generated 2 csv files from 2 mysql tables.
now I want to merge the 2 files together.</p>

<p>I manually added this header for the first csv:</p>

<pre><code>ID,name,sector,sub_sector
</code></pre>

<p>and this is the second csv header:</p>

<pre><code>ID,url
</code></pre>

<p>my goal is to have 1 file:</p>

<pre><code>ID,name,sector,sub_sector,url
</code></pre>

<p>note: not whole records in the first file have a match in the second file.</p>

<p>this is the snippet I was using:</p>

<pre><code>#!/usr/bin/env python
import glob, csv
if __name__ == '__main__':

    infiles = glob.glob('./*.csv')
    out = 'temp.csv'
    data = {}
    fields = []

    for fname in infiles:
        df = open(fname, 'rb')
        reader = csv.DictReader(df)
        for line in reader:
            # assuming the field is called ID
            if line['ID'] not in data:
                data[line['ID']] = line
            else:
                for k,v in line.iteritems():
                    if k not in data[line['ID']]:
                        data[line['ID']][k] = v
            for k in line.iterkeys():
                if k not in fields:
                    fields.append(k)
        del reader
        df.close()

    writer = csv.DictWriter(open(out, "wb"), fields, extrasaction='ignore', dialect='excel')
    # write the header at the top of the file
    writer.writeheader()
    writer.writerows(data)
    del writer
</code></pre>

<p>taken from another sof thread.
and this is the error I'm getting:</p>

<pre><code>  File "db_work.py", line 30, in &lt;module&gt;
    writer.writerows(data)
  File "/usr/lib/python2.7/csv.py", line 153, in writerows
    rows.append(self._dict_to_list(rowdict))
  File "/usr/lib/python2.7/csv.py", line 144, in _dict_to_list
    ", ".join(wrong_fields))
ValueError: dict contains fields not in fieldnames: 4, 4, 4, 6
~/Development/python/DB$ python db_work.py
Traceback (most recent call last):
  File "db_work.py", line 30, in &lt;module&gt;
    writer.writerows(data)
  File "/usr/lib/python2.7/csv.py", line 153, in writerows
    rows.append(self._dict_to_list(rowdict))
  File "/usr/lib/python2.7/csv.py", line 145, in _dict_to_list
    return [rowdict.get(key, self.restval) for key in self.fieldnames]
AttributeError: 'str' object has no attribute 'get'
</code></pre>

<p>any ideas how to fix this?  </p>
