More efficient way to skip columns in csv to namedtuple function?
Question: I'm downloading a master data table from Bloomberg's Open Symbology. The csv
has columns that I'm not interested in.

**Question**

Is there an efficient/Pythonic way to produce namedtuple instances from a
subset of the columns found within a csv file?

**What I've tried**

My current process (Python 3.3 code below) is as follows:

  1. Create a TempRecord namedtuple with all columns in the csv.
  2. Create a TempRecord instance for each record in the csv file.
  3. Create a BSYMRecord (with fewer and renamed attributes) from a given TempRecord.
  4. Yield BSYMRecord.

This smells really inefficient.

    
    
    from csv import reader
    from collections import namedtuple
    from datetime import date
    from io import BytesIO
    from urllib.request import urlopen
    from urllib.error import HTTPError
    from zipfile import ZipFile
    
    
    def bsym_records(sector, security_type, file_date):
        """Yield BSYMRecord for given sector and security type."""
        template = 'http://bdn-ak.bloomberg.com/precanned/{s}_{t}_{d}.txt.zip'
        url = template.format(s=sector, t=security_type, d=file_date)
        response = urlopen(url)
        zipfile = ZipFile(BytesIO(response.read()))
        for filename in zipfile.namelist():
            with zipfile.open(filename) as f:
                line = f.readline().decode('utf-8')
                headers = line.strip().replace(' ', '_').split('|')
                TempRecord = namedtuple('BSYMRecord', headers)
                while True:
                    line = f.readline().decode('utf-8')
                    if line[0] == '#':
                        break
                    t = TempRecord._make(line.strip().split('|'))
                    yield reduce_bsym_record(t)
    
    
    BSYMRecord = namedtuple('BSYMRecord', ['name',
                                           'ticker',
                                           'pricing_source',
                                           'security_type',
                                           'market_sector',
                                           'BBGID',
                                           'BBGID_composite',
                                           'BSID',
                                           'unique_id'])
    
    
    def reduce_bsym_record(record):
        """Eliminate non-essential fields."""
        return BSYMRecord._make((record.NAME,
                                 record.ID_BB_SEC_NUM_DES,
                                 record.FEED_SOURCE,
                                 record.SECURITY_TYP,
                                 record.MARKET_SECTOR_DES,
                                 record.ID_BB_GLOBAL,
                                 record.COMPOSITE_ID_BB_GLOBAL,
                                 record.ID_BB_SEC_NUM_SRC,
                                 record.ID_BB_UNIQUE))
    


Answer: You're currently importing the `csv` module but not using it. If you _were_
using it, you could use the
[`csv.DictReader`](http://docs.python.org/3.3/library/csv.html#csv.DictReader)
class to create a dictionary instead of a list for each line in the file. You
can construct a `namedtuple` using keyword arguments, but it doesn't ignore
spurious ones. So you'll still need to filter them manually - but you can now
do this with a dict comprehension, rather than a different namedtuple:

    
    
    for line in csvfile:
        yield BSYMRecord(**{k:v for k,v in line if k in BSYMRecord._fieldnames})
    

The trick is getting the DictReader set up in the first place. It needs a
file-like object that yields strings; `ZipFile.open` gives a file-like object
that yields _bytes_ , and can't take an encoding. The
[codecs](http://docs.python.org/3.3/library/codecs.html) module comes to the
rescue here - you can get a StreamReader that transparently decodes utf8 bytes
to strings for you like this:

    
    
    import codecs
    utf8 = codecs.lookup('utf8').streamreader
    

And use it like so:

    
    
    for filename in zipfile.namelist():
        with zipfile.open(filename) as f:
            csvfile = csv.DictReader(utf8(f))
            for line in csvfile:
                 yield BSYMRecord(**{k:v for k,v in line if k in BSYMRecord._fieldnames})
    

