deprecated_code/workflows/mpi/imaging/imaging_mpi.py
def predict_list_mpi_workflow(vis_list, model_imagelist, context, vis_slices=1, facets=1,
                                     gcfcf=None, comm=MPI.COMM_WORLD, **kwargs):
    """Predict, iterating over both the scattered vis_list and image
    
    The visibility and image are scattered, the visibility is predicted on each part, and then the
    parts are assembled. About data distribution: vis_list and model_imagelist
    live in rank 0; vis_slices, facets, context are replicated in all nodes.
    gcfcf if exists lives in rank 0; if not every mpi proc will create its own
    for the  corresponding subset of the image.

    :param vis_list:
    :param model_imagelist: Model used to determine image parameters
    :param vis_slices: Number of vis slices (w stack or timeslice)
    :param facets: Number of facets (per axis)
    :param context: Type of processing e.g. 2d, wstack, timeslice or facets
    :param gcfcg: tuple containing grid correction and convolution function
    :param comm: MPI communicator
    :param kwargs: Parameters for functions in components
    :return: List of vis_lists
   """
    rank = comm.Get_rank()
    size = comm.Get_size()
    log.info('%d: In predict_list_mpi_workflow: %d elements in vis_list' % (rank,len(vis_list)))
    # the assert only makes sense in proc 0 as for the others both lists are
    # empty
    assert len(vis_list) == len(model_imagelist), "Model must be the same length as the vis_list"

    # The use_serial_predict version paralelizes by freq (my opt version) 
    assert get_parameter(kwargs, "use_serial_predict", True),"Only freq paralellization implemented"
    #if get_parameter(kwargs, "use_serial_predict", False):
    if get_parameter(kwargs, "use_serial_predict", True):
        from workflows.serial.imaging.imaging_serial import predict_list_serial_workflow
        
        image_results_list = list()
        # Distribute visibilities and model by freq
        sub_vis_list= numpy.array_split(vis_list, size)
        sub_vis_list=comm.scatter(sub_vis_list,root=0)
        sub_model_imagelist= numpy.array_split(model_imagelist, size)
        sub_model_imagelist=comm.scatter(sub_model_imagelist,root=0)
        if gcfcf is not None:
            sub_gcfcf = numpy.array_split(gcfcf,size)
            sub_gcfcf=comm.scatter(sub_gcfcf,root=0)
        isinstance(sub_vis_list[0], Visibility)
        image_results_list= [predict_list_serial_workflow(vis_list=[sub_vis_list[i]],
                     model_imagelist=[sub_model_imagelist[i]], vis_slices=vis_slices,
                     facets=facets, context=context, gcfcf=gcfcf, **kwargs)[0]
                for i, _ in enumerate(sub_vis_list)]
        #print(image_results_list)
        
        image_results_list=comm.gather(image_results_list,root=0)
        if rank == 0:
            #image_results_list_list=[x for x in image_results_list_list if x]
            image_results_list=numpy.concatenate(image_results_list)
        else:
            image_results_list=list()

    return image_results_list
Predict, iterating over both the scattered vis_list and image
    
    The visibility and image are scattered, the visibility is predicted on each part, and then the
    parts are assembled. About data distribution: vis_list and model_imagelist
    live in rank 0; vis_slices, facets, context are replicated in all nodes.
    gcfcf if exists lives in rank 0; if not every mpi proc will create its own
    for the  corresponding subset of the image.

    :param vis_list:
    :param model_imagelist: Model used to determine image parameters
    :param vis_slices: Number of vis slices (w stack or timeslice)
    :param facets: Number of facets (per axis)
    :param context: Type of processing e.g. 2d, wstack, timeslice or facets
    :param gcfcg: tuple containing grid correction and convolution function
    :param comm: MPI communicator
    :param kwargs: Parameters for functions in components
    :return: List of vis_lists
   
Predict, iterating over both the scattered vis_list and image
The visibility and image are scattered, the visibility is predicted on each part, and then the
parts are assembled. About data distribution: vis_list and model_imagelist
live in rank 0; vis_slices, facets, context are replicated in all nodes.
gcfcf if exists lives in rank 0; if not every mpi proc will create its own
for the  corresponding subset of the image.
Predict iterating over both the scattered vis_list and image The visibility and image are scattered the visibility is predicted on each part and then the parts are assembled . About data distribution : vis_list and model_imagelist live in rank 0 ; vis_slices facets context are replicated in all nodes . gcfcf if exists lives in rank 0 ; if not every mpi proc will create its own for the corresponding subset of the image .
def predict_list_mpi_workflow(vis_list, model_imagelist, context, vis_slices=1, facets=1,
                                     gcfcf=None, comm=MPI.COMM_WORLD, **kwargs):
    rank = comm.Get_rank()
    size = comm.Get_size()
    log.info('%d: In predict_list_mpi_workflow: %d elements in vis_list' % (rank,len(vis_list)))
    assert len(vis_list) == len(model_imagelist), "Model must be the same length as the vis_list"
    assert get_parameter(kwargs, "use_serial_predict", True),"Only freq paralellization implemented"
    if get_parameter(kwargs, "use_serial_predict", True):
        from workflows.serial.imaging.imaging_serial import predict_list_serial_workflow
        image_results_list = list()
        sub_vis_list= numpy.array_split(vis_list, size)
        sub_vis_list=comm.scatter(sub_vis_list,root=0)
        sub_model_imagelist= numpy.array_split(model_imagelist, size)
        sub_model_imagelist=comm.scatter(sub_model_imagelist,root=0)
        if gcfcf is not None:
            sub_gcfcf = numpy.array_split(gcfcf,size)
            sub_gcfcf=comm.scatter(sub_gcfcf,root=0)
        isinstance(sub_vis_list[0], Visibility)
        image_results_list= [predict_list_serial_workflow(vis_list=[sub_vis_list[i]],
                     model_imagelist=[sub_model_imagelist[i]], vis_slices=vis_slices,
                     facets=facets, context=context, gcfcf=gcfcf, **kwargs)[0]
                for i, _ in enumerate(sub_vis_list)]
        image_results_list=comm.gather(image_results_list,root=0)
        if rank == 0:
            image_results_list=numpy.concatenate(image_results_list)
        else:
            image_results_list=list()
    return image_results_list
def predict_list_mpi_workflow ( vis_list , model_imagelist , context , vis_slices = 1 , facets = 1 , gcfcf = None , comm = MPI . COMM_WORLD , ** kwargs ) : rank = comm . Get_rank ( ) size = comm . Get_size ( ) log . info ( '%d: In predict_list_mpi_workflow: %d elements in vis_list' % ( rank , len ( vis_list ) ) ) assert len ( vis_list ) == len ( model_imagelist ) , "Model must be the same length as the vis_list" assert get_parameter ( kwargs , "use_serial_predict" , True ) , "Only freq paralellization implemented" if get_parameter ( kwargs , "use_serial_predict" , True ) : from workflows . serial . imaging . imaging_serial import predict_list_serial_workflow image_results_list = list ( ) sub_vis_list = numpy . array_split ( vis_list , size ) sub_vis_list = comm . scatter ( sub_vis_list , root = 0 ) sub_model_imagelist = numpy . array_split ( model_imagelist , size ) sub_model_imagelist = comm . scatter ( sub_model_imagelist , root = 0 ) if gcfcf is not None : sub_gcfcf = numpy . array_split ( gcfcf , size ) sub_gcfcf = comm . scatter ( sub_gcfcf , root = 0 ) isinstance ( sub_vis_list [ 0 ] , Visibility ) image_results_list = [ predict_list_serial_workflow ( vis_list = [ sub_vis_list [ i ] ] , model_imagelist = [ sub_model_imagelist [ i ] ] , vis_slices = vis_slices , facets = facets , context = context , gcfcf = gcfcf , ** kwargs ) [ 0 ] for i , _ in enumerate ( sub_vis_list ) ] image_results_list = comm . gather ( image_results_list , root = 0 ) if rank == 0 : image_results_list = numpy . concatenate ( image_results_list ) else : image_results_list = list ( ) return image_results_list
Predict, iterating over both the scattered vis_list and image
The visibility and image are scattered, the visibility is predicted on each part, and then the
parts are assembled.
Predict iterating over both the scattered vis_list and image The visibility and image are scattered the visibility is predicted on each part and then the parts are assembled .
"""Predict, iterating over both the scattered vis_list and image
    
    The visibility and image are scattered, the visibility is predicted on each part, and then the
    parts are assembled. About data distribution: vis_list and model_imagelist
    live in rank 0; vis_slices, facets, context are replicated in all nodes.
    gcfcf if exists lives in rank 0; if not every mpi proc will create its own
    for the  corresponding subset of the image.

    :param vis_list:
    :param model_imagelist: Model used to determine image parameters
    :param vis_slices: Number of vis slices (w stack or timeslice)
    :param facets: Number of facets (per axis)
    :param context: Type of processing e.g. 2d, wstack, timeslice or facets
    :param gcfcg: tuple containing grid correction and convolution function
    :param comm: MPI communicator
    :param kwargs: Parameters for functions in components
    :return: List of vis_lists
   """ # the assert only makes sense in proc 0 as for the others both lists are # empty # The use_serial_predict version paralelizes by freq (my opt version)  #if get_parameter(kwargs, "use_serial_predict", False): # Distribute visibilities and model by freq #print(image_results_list) #image_results_list_list=[x for x in image_results_list_list if x]
List of vis_lists
List of vis_lists
vis_list
None
model_imagelist
Model used to determine image parameters
Model used to determine image parameters
context
Type of processing e.g. 2d, wstack, timeslice or facets
Type of processing e . g . 2d wstack timeslice or facets
vis_slices
Number of vis slices (w stack or timeslice)
Number of vis slices ( w stack or timeslice )
facets
Number of facets (per axis)
Number of facets ( per axis )
gcfcf
comm
None
gcfcg
tuple containing grid correction and convolution function
tuple containing grid correction and convolution function
kwargs
Parameters for functions in components
Parameters for functions in components