Is there a limit on tuple size in python?
Question: I would like to assemble a training set for a neural network. I have a large
csv file with 1017209 rows and 7 columns. I need to extract certain features
from this file, and put them into a tuple. I do it the following way:

    
    
    import pandas as pd
    
    data = pd.read_csv('data.csv', index_col=False, header=0)
    matrix = data.as_matrix()
    
    training_data = ()
    
    for x in range(1017209):
        a = np.ndarray((3,1), buffer=np.array([matrix[x][0],matrix[x][1],matrix[x][5]]), dtype=float)
        b = np.ndarray((1,1), buffer=np.array(matrix[x][3]), dtype=float)
        training_data = training_data + ((a,b),)
    

It works when I loop for about 100000:

    
    
    for x in range(100000)
    

But not with the full set:

    
    
    for x in range(1017209)
    

I run it in spyder. It does not break, but does not seem to finish either.
When I set to 100000 it finishes in about a minute. When I set to 1017209 it
does not finish even in 45 minutes.

What might be happening? Why do I not receive an error message? Is there a
better way to create the same tuple?


Answer: I think C32 hit on the problem (although he answered it in a comment for some
reason). Since tuples are immutable, you have to create a new tuple each time
rather than appending to an existing tuple. This gives you quadratic
behaviour, and if you make your tuple 10 times longer it'll take 100 times as
long. Your code probably would have finished in about 100 minutes or so. For
example:

    
    
    def build_tup(n):
        td = ()
        for i in range(n):
            td = td + ((1,2),)
        return td    
    
    def add_to_list(n):
        td = []
        for i in range(n):
            td += (1,2),
        return td
    

gives quadratic behaviour for `build_tup`:

    
    
    >>> %timeit build_tup(100)
    10000 loops, best of 3: 21.7 µs per loop
    >>> %timeit build_tup(1000)
    1000 loops, best of 3: 1.7 ms per loop
    >>> %timeit build_tup(10000)
    10 loops, best of 3: 165 ms per loop
    

but effectively linear behaviour for `add_to_list`:

    
    
    >>> %timeit add_to_list(100)
    100000 loops, best of 3: 3.64 µs per loop
    >>> %timeit add_to_list(1000)
    10000 loops, best of 3: 35 µs per loop
    >>> %timeit add_to_list(10000)
    The slowest run took 4.96 times longer than the fastest. This could mean that an intermediate result is being cached 
    1000 loops, best of 3: 348 µs per loop
    

