Registration timeout on PBS cluster with Ipython parallel
Question: I'm trying to set up ipython parallel on a linux cluster using PBS scheduling.

I was following instructions at <http://www.andreazonca.com/2013/04/ipython-
parallell-setup-on-carver-at.html> (the official instructions are much harder
to follow). I'm running the command on the head node, which sends the jobs to
the slave nodes with PBS (ie, a standard cluster configuration).

My problem is that I get a timeout. I tried increasing the waiting time from
2s up to 20s but without success. Any help would be appreciated. Full output
is below.

Actually, in the end I want to be able to run the ipython commands from my
ssh-connected laptop rather than from the cluster-head-node, but I thought
this was a reasonable first step.

    
    
    2013-08-11 13:56:07,380.380 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,381.381 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,381.381 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,382.382 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,381.381 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,381.381 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,383.383 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,382.382 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,382.382 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,383.383 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,382.382 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,383.383 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,383.383 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,383.383 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,382.382 [IPEngineApp] Config changed:
    2013-08-11 13:56:07,383.383 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:07,387.387 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,387.387 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,388.388 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,388.388 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,388.388 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,389.389 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,389.389 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,389.389 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,389.389 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,389.389 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,389.389 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,389.389 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,389.389 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,389.389 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,390.390 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,390.390 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,390.390 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07,390.390 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:07,390.390 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:07,391.391 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:07,391.391 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07,391.391 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07,391.391 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07,391.391 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07.391 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07,391.391 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:07.391 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07,391.391 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07.391 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07,391.391 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:07,392.392 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07.392 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07.392 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07.392 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07,392.392 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:07.392 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07.393 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:07.402 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:07.402 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:07.402 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:07.402 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:07.403 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:07.403 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:07.403 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:07.403 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01,273.273 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,273.273 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,274.274 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,274.274 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,272.272 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,274.274 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,275.275 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,275.275 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,273.273 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,276.276 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,276.276 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,276.276 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,278.278 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,278.278 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,279.279 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01,279.279 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01,279.279 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01,279.279 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01,279.279 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01,279.279 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01,279.279 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01,280.280 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01,280.280 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01,280.280 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01,280.280 [IPEngineApp] Config changed:
    2013-08-11 13:56:01,280.280 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01,280.280 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] {'EngineFactory': {'timeout': 10}, 'IPEngineApp': {'log_level': 10}}
    2013-08-11 13:56:01,280.280 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,280.280 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01,281.281 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01,281.281 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01,281.281 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01,282.282 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01,282.282 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,282.282 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,282.282 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01.282 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01,282.282 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01.282 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01,282.282 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01.282 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01,283.283 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01,283.283 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,283.283 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,284.284 [IPEngineApp] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-08-11 13:56:01,284.284 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01,284.284 [IPEngineApp] Searching path [u'/home/username', u'/home/username/.ipython/profile_default'] for config files
    2013-08-11 13:56:01,284.284 [IPEngineApp] Attempting to load config file: ipython_config.py
    2013-08-11 13:56:01.284 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01.284 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01.284 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01.284 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01,284.284 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipython_config.py
    2013-08-11 13:56:01,285.285 [IPEngineApp] Attempting to load config file: ipengine_config.py
    2013-08-11 13:56:01,286.286 [IPEngineApp] Loaded config file: /home/username/.ipython/profile_default/ipengine_config.py
    2013-08-11 13:56:01.286 [IPEngineApp] Loading url_file u'/home/username/.ipython/profile_default/security/ipcontroller-engine.json'
    2013-08-11 13:56:01.295 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01.295 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01.296 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01.296 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01.296 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01.296 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01.298 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:01.299 [IPEngineApp] Registering with controller at tcp://127.0.0.1:53956
    2013-08-11 13:56:17.411 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:17.412 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:17.413 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:17.412 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:17.412 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:17.413 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:17.413 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:17.413 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.304 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.304 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.305 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.306 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.306 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.306 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.307 [IPEngineApp] Registration timed out after 10.0 seconds
    2013-08-11 13:56:11.309 [IPEngineApp] Registration timed out after 10.0 seconds
    --------------------------------------------------------------------------
    mpirun noticed that the job aborted, but has no info as to the process
    that caused that situation.
    --------------------------------------------------------------------------
    

**Update following advice from Andrea Zonca**

Adding

    
    
    c.HubFactory.ip = '*'
    

appears to help, although not immediately for some unknown reason.

But it still doesn't work:

I run with

    
    
    ./.ipython/profile_default/ipc 2 1
    
    2013-09-09 12:19:51,884.884 [IPClusterStart] Using existing profile dir: u'/home/username/.ipython/profile_default'
    2013-09-09 12:19:51.889 [IPClusterStart] Starting ipcluster with [daemon=False]
    2013-09-09 12:19:51.890 [IPClusterStart] Creating pid file: /home/username/.ipython/profile_default/pid/ipcluster.pid
    2013-09-09 12:19:51.890 [IPClusterStart] Starting Controller with LocalControllerLauncher
    2013-09-09 12:19:52.890 [IPClusterStart] Starting 2 Engines with PBS
    2013-09-09 12:19:52.904 [IPClusterStart] Job submitted with job id: '4783'
    2013-09-09 12:20:22.904 [IPClusterStart] Engines appear to have started successfully
    

and then run on the head node:

    
    
    >IPython.parallel import Client
    >rc = Client()
    >lview = rc.load_balanced_view()
    

but the output from

    
    
    >rc.ids
    

is

    
    
    []
    

So I tried running

    
    
    ipcontroller --port=8888
    

and then run nmap

    
    
    $nmap -sT -O localhost
    ...
    8888/tcp  open  sun-answerbook
    ...
    

which shows the port is open, and sure-enough telnet gives me a response from
the slave node.

But when I run the original command above ie,

    
    
    ./.ipython/profile_default/ipc 2 1
    

nmap shows no port as being open. So the problem appears to be that ipengine
run in the qsub file is not opening the port like ipcontroller run from the
command line is.

Here is the qsub file:

    
    
    $cat /home/username/.ipython/profile_default/pbs.engine.template.ppn2 
    #!/bin/sh
    #PBS -q longqueue
    #PBS -l nodes={n/2}:ppn=2
    cd $PBS_O_WORKDIR
    which ipengine
    mpirun -np {n} ipengine --timeout=20
    

and here is my /home/username/.ipython/profile_default/ipcluster_config.py:

    
    
    c = get_config()
    
    c.IPClusterStart.controller_launcher_class = 'LocalControllerLauncher'
    c.IPClusterStart.engine_launcher_class = 'PBS'
    c.PBSLauncher.batch_template_file = u'/home/username/.ipython/profile_default/pbs.engine.template' 
    


Answer: You need to allow connections to the controller from other hosts, setting in
`~/.config/ipython/profile_default/ipcontroller_config.py`:

    
    
    c.HubFactory.ip = '*'
    

Btw, I also updated the blog post.

