python - Overhead on looping over an iterable class
Question: I was fiddling around with Python's generators and iterable class, just for
fun. Basically I wanted test out something that I've never been too sure
about: that classes in Pythons have some significant overhead and it's better
to rely on methods that implement `yield` instead of classes that implement an
iterator protocol, if you can.

I couldn't find a satisfying explanation on this topic in Google, so I decided
to test them out on my own using these two simple scripts: `func_iter.py` and
`class_iter.py`

Here's `func_iter.py`:

    
    
    #!/usr/bin/env python
    
    import time  
    
    x = 0
    def create_generator(num):
        mylist = range(num)
        for i in mylist:
            yield i
    
    t = time.time()
    gen = create_generator(100000)
    
    for i in gen:
        x = x + i
    
    print "%.3f" % (time.time() - t)
    

And here's `class_iter.py`:

    
    
    #!/usr/bin/env python
    
    import time
    
    x = 0
    
    class Generator(object):
    
        def __init__(self, num):
            self.start = 0
            self.end = num
    
        def __iter__(self):
            return self
    
        def next(self):
            if self.start == self.end:
                raise StopIteration
            else:
                self.start = self.start + 1
                return self.start
    
    t = time.time()
    gen = Generator(100000)
    
    for i in gen:
        x = x + i
    
    print "%.3f" % (time.time() - t)
    

I then ran each of them 10 times using this in bash (for `class_iter.py`, for
example):

    
    
    for i in {1..10}; do ./class_iter.py; done
    

And here are the average running times for each of them:

    
    
    class_iter.py: 0.0864
    func_iter.py: 0.0307
    

Now, my questions are:

  1. Are my methods correct? Is my comparison fair?
  2. If so, why the big difference? Why did `class_iter.py` take almost three times as long as `func_iter.py` to run?
  3. If not, how can I improve my methods or come up with a better comparison? 

EDIT: As Dacav suggested, I also tried running `func_iter.py` using `xrange`
instead of `range`. This decreases its average running time to 0.0263 seconds.


Answer: The class version spends lots of time accessing its own variables. Each
`self.whatever` costs cycles. If you define your `__iter__` as a generator and
minimize the use of instance variables, the difference between class and
function versions will be negligible:

    
    
    setup = """
    def create_generator(num):
        mylist = range(num)
        for i in mylist:
            yield i
    
    class Generator(object):
    
        def __init__(self, num):
            self.start = 0
            self.end = num
    
        def __iter__(self):
            return self
    
        def next(self):
            if self.start == self.end:
                raise StopIteration
            else:
                self.start = self.start + 1
                return self.start
    
    class Generator2(object):
    
        def __init__(self, num):
            self.mylist = range(num)
    
        def __iter__(self):
            for i in self.mylist:
                yield i
    """
    
    import timeit
    
    print timeit.timeit('for p in create_generator(1000):p', setup, number=1000)
    print timeit.timeit('for p in Generator(1000):p', setup, number=1000)
    print timeit.timeit('for p in Generator2(1000):p', setup, number=1000)
    

Results:

    
    
    0.158941984177
    0.696810007095
    0.160784959793
    

so the second generator class is almost as fast as the function version.

Please do note that `Generator` and `Generator2` in the example are not fully
equivalent, there are cases when you cannot simply replace a "plain" iterator
with a generator (e.g. marshaling).

