How can python subprocess.Popen see select.poll and then later not? (select 'module' object has no attribute 'poll')
Question: I'm using the (awesome) [mrjob](http://packages.python.org/mrjob/) library
from Yelp to run my python programs in Amazon's Elastic Map Reduce. It depends
on subprocess in the standard python library. From my mac running python2.7.2,
everything works as expected

However, when I switched to using the exact same code on Ubuntu LTS 11.04 also
with python2.7.2, I encountered something strange:

mrjob loads the job, and then attempts to communicate with its child processes
using subprocess and generates this error:

    
    
          File "/usr/local/lib/python2.7/dist-packages/mrjob-0.3.1-py2.7.egg/mrjob/emr.py", line 1212, in _build_steps
            steps = self._get_steps()
          File "/usr/local/lib/python2.7/dist-packages/mrjob-0.3.1-py2.7.egg/mrjob/runner.py", line 1003, in _get_steps
            stdout, stderr = steps_proc.communicate()
          File "/usr/lib/python2.7/subprocess.py", line 754, in communicate
            return self._communicate(input)
          File "/usr/lib/python2.7/subprocess.py", line 1302, in _communicate
            stdout, stderr = self._communicate_with_poll(input)
          File "/usr/lib/python2.7/subprocess.py", line 1332, in _communicate_with_poll
            poller = select.poll()
        AttributeError: 'module' object has no attribute 'poll'
    

This appears to be a problem with subprocess and not mrjob.

I dug into /usr/lib/python2.7/subprocess.py and found that during import it
runs:

    
    
        if mswindows:
            ... snip ...
        else:
            import select
            _has_poll = hasattr(select, 'poll')
    

By editing that, I verified that it really does set _has_poll==True. And this
is correct; easily verified on the command line.

However, when execution progresses to using Popen._communicate_with_poll
somehow the select module has changed! This is generated by printing
dir(select) right before it attempts to use select.poll().

    
    
        ['EPOLLERR', 'EPOLLET', 'EPOLLHUP', 'EPOLLIN', 'EPOLLMSG', 
        'EPOLLONESHOT', 'EPOLLOUT', 'EPOLLPRI', 'EPOLLRDBAND', 
        'EPOLLRDNORM', 'EPOLLWRBAND', 'EPOLLWRNORM', 'PIPE_BUF', 
        'POLLERR', 'POLLHUP', 'POLLIN', 'POLLMSG', 'POLLNVAL', 
        'POLLOUT', 'POLLPRI', 'POLLRDBAND', 'POLLRDNORM',
        'POLLWRBAND', 'POLLWRNORM', '__doc__', '__name__', 
        '__package__', 'error', 'select']
    

no attribute called 'poll'!?!? How did it go away?

So, I hardcoded _has_poll=False and then mrjob happily continues with its
work, runs my job in AWS EMR, with subprocess using communicate_with_select...
and I'm stuck with a hand-modified standard library...

Any advice? :-)


Answer: I had a similar problem and it turns out that gevent replaces the built-in
`select` module with `gevent.select.select` which doesn't have a `poll` method
(as it is a blocking method). However for some reason by default gevent
doesn't patch `subprocess` which uses `select.poll`.

An easy fix is to replace `subprocess` with `gevent.subprocess`:

    
    
    import gevent.monkey
    gevent.monkey.patch_all(subprocess=True)
    
    import sys
    import gevent.subprocess
    sys.modules['subprocess'] = gevent.subprocess
    

If you do this before importing the mrjob library, it should work fine.

