title: Can Lists cope with a million items?
body: <p>I am slowly developing a data processing application in Python 2.6 (*). My test data are very small, like 5000 cases, but it is expected that there will be a million cases in the near future and I wondering if my current approach is workable under those conditions.</p>
  
  <p>Structure of the problem:
  I have two csv files, one contains call (5000 rows, 20 columns) and another one details for a call (500 rows, 10 columns). I have to build a third csv file which will contain all cases from the "call" file where additional details where found. Behind the scenes there is some heavy lifting going on (merging and restructuring data in the details list, data comparison between lists). 
  But I am very nervous about building the output list: at the moment the code looks like this:</p>
  
  <pre><code>def reduceOutputListToPossibleMatches(outputList, detailsList):
      reducedList = list()
  
      for outputItem in outputList:
          isFound = False
          for detailsItem in detailsList:
              if detailsItem[14] == outputItem[4]:
                  if isfound:
                      detailsItem[30] = "1" #ambigous case
                                            # - more than one match was found
  # 1 is an indicator for true - I am not using python here because spss has no support for booleans.
                  isFound = True
          if isFound:
              reducedList.append(detailsItem )
  
      return reducedList
  </code></pre>
  
  <p>I think that this algorithm will take a very long time, simply because I have to loop about two large lists.
  So my questions boils down to: How fast are lists in Python and are there better alternatives? Additionally: The double-List are somewhat inconvenient to handle, because I have to remember the index position of each column - is there a better alternative?</p>
  
  <p>*=I am calling SPSS Version 19 later on, which refuses to work with newer versions of python.</p>