Why does python thread consume so much memory?
Question: Why does python thread consumes so much memory?

I measured that spawning one thread consumes 8 megs of memory, almost as big
as a whole new python process!

OS: Ubuntu 10.10

Edit: due to popular demand I'll give some extraneous examples, here it is:

    
    
    from os import getpid
    from time import sleep
    from threading import Thread
    
    def nap():
        print 'sleeping child'
        sleep(999999999)
    
    print getpid()
    child_thread = Thread(target=nap)
    sleep(999999999)
    

On my box, pmap pid will give 9424K

Now, let's run the child thread:

    
    
    from os import getpid
    from time import sleep
    from threading import Thread
    
    def nap():
        print 'sleeping child'
        sleep(999999999)
    
    print getpid()
    child_thread = Thread(target=nap)
    child_thread.start()             # <--- ADDED THIS LINE
    sleep(999999999)
    

Now pmap pid will give 17620K

So, the cost for the extra thread is 17620K - 9424K = 8196K

ie. 87% of running a whole new separate process!

Now isn't that just, wrong?


Answer: This is not Python-specific, and has to do with the separate stack that gets
allocated by the OS for every thread. The default maximum stack size on your
OS happens to be 8MB.

Note that the 8MB is simply a chunk of address space that gets set aside, with
very little memory committed to it initially. Additional memory gets committed
to the stack when required, up to the 8MB limit.

The limit can be tweaked using `ulimit -s`, but in this instance I see no
reason to do this.

As an aside, `pmap` shows _address space_ usage. It isn't a good way to gauge
_memory_ usage. The two concepts are quite distinct, if related.

