parallelization issue using python view library
Question: so i Worte some code with basic structure like this:

    
    
    from numpy import *
    from dataloader import loadfile
    from IPython.parallel import Client
    from clustering import * 
    
    data = loadfile(0)
    
    N_CLASSES = 10
    rowmax = nanmax(data.values, 0)
    rowmin = nanmin(data.values, 0)
    
    # Defines the size of block processed at a time
    BLOCK_SIZE = 50000
    classmean, classcov, classcovinv, classlogdet, classlogprob = init_stats(data, N_CLASSES, rowmax, rowmin)
    
    client = Client()
    ids    = client.ids
    nodes  = len(ids)
    view   = client.load_balanced_view()
    dview  = client[:]
    
    def get_ml_class(data, args): do sth
    dview.scatter('datablock', data)
    dview.execute('res1, res2 = get_ml_class(datablock, args)', block=False)
    

the output of dview.execute part is

    
    
    <AsyncResult: execute>
    

which means it is executed, however, when i was trying to pull the result by

    
    
    dview.pull(['res1','res2'], block=True)
    

it shows:

    
    
    NameError: name 'res1' is not defined
    

Can someone please tell me what is wrong with my code?? Thank you so much!


Answer: Let us make a simpler example:

    
    
    from IPython.parallel import Client
    
    rc = Client()
    dview = rc[:]
    dview.scatter('a', Range(16))
    dview.execute('res1,res2 = a[0], a[1]', block=False)
    dview.pull(['res1'], block=True)
    

This works as expected and gives a the result:

    
    
    [[0], [4], [8], [12]]
    

So, we do at least this one right. But let me change the code a bit:

    
    
    from IPython.parallel import Client
    
    rc = Client()
    dview = rc[:]
    dview.scatter('a', Range(16))
    dview.execute('res1,res2 = a[0], b[1]', block=False)
    dview.pull(['res1'], block=True)
    

Now we have the `NameError`. Why?

Because there is an error on the `execute` line (it references to variable `b`
which does not exist). The non-blocking `execute` does not complain much. In
the first (working) case the status is:

    
    
    <AsyncResult: finished>
    

and in the second (non-working) case:

    
    
    <AsyncResult: execute>
    

Other than that it is very quiet, and the second message does not necessarily
mean an error has occured. In order to see the real error message, change
`blocking` to `True`. Then you'll see what is going wrong.

If you want to know if your non-blocking execute works, you have to capture
the `AsyncResult` object returned by `execute`. It has several interesting
methods, but you would be most interested in `ready` and `successful` methods:

    
    
    ar = dview.execute(...)
    ar.ready()                # True if the process has finished
    ar.successful()           # True if there were no exceptions raised
    

Also, the possible exceptions raised during the execution can be fetched by
using the `get` method of the `AsyncResult` object. For example my bad example
gives in the interactive shell:

    
    
    >>> ar.get()
    
    [0:execute]: 
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)<ipython-input-1-608f57d70b2f> in <module>()
    ----> 1 res1,res2=a[0]**2,b[1]**2
    NameError: name 'b' is not defined
    
    [1:execute]: 
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)<ipython-input-1-608f57d70b2f> in <module>()
    ----> 1 res1,res2=a[0]**2,b[1]**2
    NameError: name 'b' is not defined
    
    ...
    

So, as a summary: Try to find out what goes wrong with the function you try to
run remotely. Now it seems to raise some execption. The error might have
something to do with `args` which does not seem to be available for the remote
scripts. Maybe a `scatter` is missing?

