Python script to read CSVs and regroup it into the data types I need This code does the following:


Read CSV file and determine sub_type based on series_desc.
Turn row into dictionaries new_person and new_subscriptions.
Check list people for the new_person and add the person if it is actually new.
Check each person in the list people for the new_subscription and add it if it is not there.


My desired output is a list of dictionaries called people. A dictionary in people might look like this:

{'id': group_id, 'subscriptions': [sub_dict1, sub_dict2]}


where a sub_dict looks like this:

{'id': sub_id, 'season': season_desc, 'sub_type': sub_type}


The code:

import csv

people = []

#open csv file
csv_file = csv.reader(open(filename, 'rb'))

#read CSV file into a list of people
for record in csv_file:
    sub_id = record[0]
    group_id = record[6]
    season_desc = record[11]
    series_desc = record[12]
    price = record[13]

    if "MW" in series_desc:
        sub_type = "Masterworks"
    elif "Pops" in series_desc:
        sub_type = "Pops"
    elif "Chamber" in series_desc:
        sub_type = "Chamber"

    new_person = {'id': group_id}
    new_subscription = {'id': sub_id, 'season': season_desc, 'sub_type': sub_type}

    #if this is the first time through the loop
    if len(people) == 0:
        new_person['subscriptions'] = []
        new_person['subscriptions'].append(new_subscription)
        people.append(new_person)

    #if this is not the first time through the loop
    else:
        #check if this person has already been recorded
        if any(person['id'] == new_person['id'] for person in people) == True:
            #check if this subscription has already been added to this person
            #use enumerate to get a key to search people
            for i in enumerate(people):
                if i[1]['id'] == new_person['id']:
                    key = i[0]

                    if any(sub['id'] == new_subscription['id'] for sub in people[key]['subscriptions']) == True:
                        pass
                else:
                    people[key]['subscriptions'].append(new_subscription)

        #if not, add the subscription to them and save the person
        else:
            new_person['subscriptions'] = []
            new_person['subscriptions'].append(new_subscription)
            people.append(new_person)


The challenge: It is REALLY slow.

I have for/if/else loops nested inside for/if/else loops, so I am sure that is the cause. What can I do differently?
The opened file is never closed, a better habit is to use
with open(filename, 'rb') as foo:.
foo == True is rarely needed; unless you really need to
differentiate between True/False and the various other truthy
values it's easier just to test for the variable instead.  Same goes
for the empty list for that matter.
The pass doesn't do anything; price is unused.
Instead of linear search you could keep a set or dict of already
recorded people around.
With enumerate it's easier to destructure the result directly in the
loop; that said, the enumerate is also not necessary here.
The use of key looks buggy:  In the else branch it has never been
set (except maybe because of Pythons moronic scoping rules).  In any
case it'd better to make clear how that works.


That would be something the lines of the following then:

import csv

people = []
people_ids = set()

def add_person(group_id, new_subscription):
    people.append({
        'id': group_id,
        'subscriptions': [new_subscription]
    })
    people_ids.add(group_id)

#open csv file
with open(filename, 'rb') as file:
    csv_file = csv.reader(file)

    #read CSV file into a list of people
    for record in csv_file:
        sub_id = record[0]
        group_id = record[6]
        season_desc = record[11]
        series_desc = record[12]

        if "MW" in series_desc:
            sub_type = "Masterworks"
        elif "Pops" in series_desc:
            sub_type = "Pops"
        elif "Chamber" in series_desc:
            sub_type = "Chamber"

        new_subscription = {
            'id': sub_id,
            'season': season_desc,
            'sub_type': sub_type
        }

        #if this is the first time through the loop
        if len(people) == 0:
            add_person(group_id, new_subscription)
            continue

        #check if this person has already been recorded
        #if not, add the subscription to them and save the person
        if group_id not in people_ids:
            add_person(new_subscription)
            continue

        #check if this subscription has already been added to this person
        #use enumerate to get a key to search people
        for person in people:
            if person['id'] != group_id:
                person['subscriptions'].append(new_subscription)


I also shuffled the branches around, which together with continue
makes for less indented blocks.

If you're going to extend this, I'd recommend more factoring into functions, a
main function, etc.