cytometer/data.py
def load_datasets(file_list, prefix_from='im', prefix_to=[], nblocks=1, shuffle_seed=None):
    """
    Loads image files and prepare them for training or testing, returning numpy.ndarrays.
    Image files can be of any type loadable by the PIL module, but they must have the same size.

    Multiple sets of corresponding images can be loaded using prefix_to. For instance,

    im_file_1.tif          seg_file_1.tif          mask_file_1.tif
    im_file_2.tif          seg_file_2.tif          mask_file_2.tif
    im_file_3.tif          seg_file_3.tif          mask_file_3.tif

    will return

    outs['im']   --> numpy.ndarray (3, rows, cols, channels)
    outs['seg']  --> numpy.ndarray (3, rows, cols, channels)
    outs['mask'] --> numpy.ndarray (3, rows, cols, channels)

    This function also provides the following optional functionality:
        * images can be split into equally-sized blocks (this is useful when full images are too
        large for training).
        * images can be shuffled randomly.

    :param file_list: list of paths and filenames (for one of the datasets).
    :param prefix_from: (def 'im') string with the prefix (e.g. 'im') in file_list that when changed gives the other
    datasets. Note that the prefix refers to the name of the file, not its path.
    :param prefix_to: (def []) list of strings with the prefixes of the datasets that will be loaded. E.g.
    ['im', 'mask'] will load the datasets from the filenames that start with 'im' and 'mask'.
    :param nblocks: (def 1) number of equi-sized blocks to split the images into. Note that the edges of the images may
    need to be trimmed so that splitting creates blocks with the same size.
    :param shuffle_seed: (def None) If provided, images are shuffled after splitting.
    :return: out, out_file_list, shuffle_idx:
       * out: dictionary where out[prefix] contains a numpy.ndarray with the data corresponding to the "prefix" dataset.
       * out_file_list: list of the filenames for the out[prefix] dataset.
       * shuffle_idx: list of indices used to shuffle the images after splitting.
    """

    if not isinstance(prefix_to, list):
        raise TypeError('data_prefixes must be a list of strings')

    # using prefixes, get list of filenames for the different datasets
    out_file_list = {}
    for prefix in prefix_to:
        out_file_list[prefix] = []
        for x in file_list:
            x_path, x_basename = os.path.split(x)
            if re.match('^' + prefix_from, x_basename) is None:
                raise ValueError('Prefix not found in filename: ' + x)
            prefix_file = os.path.join(x_path,
                                       re.sub('^' + prefix_from, prefix, x_basename, count=1))
            if not os.path.isfile(prefix_file):
                raise FileExistsError(prefix_file)
            out_file_list[prefix].append(prefix_file)

    # load all datasets
    out = {}
    shuffle_idx = {}
    for prefix in prefix_to:
        # load dataset
        out[prefix] = load_file_list_to_array(out_file_list[prefix])

        # data type conversions
        if prefix == 'im' and out[prefix].dtype == 'uint8':
            out[prefix] = out[prefix].astype(np.float32)
            out[prefix] /= 255
        elif prefix in {'mask', 'dmap'}:
            out[prefix] = out[prefix].astype(np.float32)
        elif prefix == 'seg':
            out[prefix] = out[prefix].astype(np.uint8)

        # split image into smaller blocks, if requested
        if nblocks > 1:
            out[prefix] = split_images(out[prefix], nblocks=nblocks)

        # create shuffling indices if required
        if prefix == prefix_to[0] and shuffle_seed is not None:
            n = out[prefix].shape[0]  # number of images
            shuffle_idx = np.arange(n)
            np.random.seed(shuffle_seed)
            np.random.shuffle(shuffle_idx)

        # shuffle data
        if shuffle_seed is not None:
            out[prefix] = out[prefix][shuffle_idx, ...]

    if DEBUG:
        i = 5
        plt.clf()
        for pi, prefix in enumerate(prefix_to):
            plt.subplot(1, len(prefix_to), pi+1)
            if out[prefix].shape[-1] == 1:
                plt.imshow(out[prefix][i, :, :, 0])
            else:
                plt.imshow(out[prefix][i, :, :, :])
            plt.title('out[' + prefix + ']')

    return out, out_file_list, shuffle_idx

    Loads image files and prepare them for training or testing, returning numpy.ndarrays.
    Image files can be of any type loadable by the PIL module, but they must have the same size.

    Multiple sets of corresponding images can be loaded using prefix_to. For instance,

    im_file_1.tif          seg_file_1.tif          mask_file_1.tif
    im_file_2.tif          seg_file_2.tif          mask_file_2.tif
    im_file_3.tif          seg_file_3.tif          mask_file_3.tif

    will return

    outs['im']   --> numpy.ndarray (3, rows, cols, channels)
    outs['seg']  --> numpy.ndarray (3, rows, cols, channels)
    outs['mask'] --> numpy.ndarray (3, rows, cols, channels)

    This function also provides the following optional functionality:
        * images can be split into equally-sized blocks (this is useful when full images are too
        large for training).
        * images can be shuffled randomly.

    :param file_list: list of paths and filenames (for one of the datasets).
    :param prefix_from: (def 'im') string with the prefix (e.g. 'im') in file_list that when changed gives the other
    datasets. Note that the prefix refers to the name of the file, not its path.
    :param prefix_to: (def []) list of strings with the prefixes of the datasets that will be loaded. E.g.
    ['im', 'mask'] will load the datasets from the filenames that start with 'im' and 'mask'.
    :param nblocks: (def 1) number of equi-sized blocks to split the images into. Note that the edges of the images may
    need to be trimmed so that splitting creates blocks with the same size.
    :param shuffle_seed: (def None) If provided, images are shuffled after splitting.
    :return: out, out_file_list, shuffle_idx:
       * out: dictionary where out[prefix] contains a numpy.ndarray with the data corresponding to the "prefix" dataset.
       * out_file_list: list of the filenames for the out[prefix] dataset.
       * shuffle_idx: list of indices used to shuffle the images after splitting.
    
Loads image files and prepare them for training or testing, returning numpy.ndarrays.
Image files can be of any type loadable by the PIL module, but they must have the same size.

Multiple sets of corresponding images can be loaded using prefix_to. For instance.



will return



This function also provides the following optional functionality:
images can be split into equally-sized blocks (this is useful when full images are too
large for training).
images can be shuffled randomly.
Loads image files and prepare them for training or testing returning numpy . ndarrays . Image files can be of any type loadable by the PIL module but they must have the same size . Multiple sets of corresponding images can be loaded using prefix_to . For instance . will return This function also provides the following optional functionality : images can be split into equally - sized blocks ( this is useful when full images are too large for training ) . images can be shuffled randomly .
def load_datasets(file_list, prefix_from='im', prefix_to=[], nblocks=1, shuffle_seed=None):
    if not isinstance(prefix_to, list):
        raise TypeError('data_prefixes must be a list of strings')
    out_file_list = {}
    for prefix in prefix_to:
        out_file_list[prefix] = []
        for x in file_list:
            x_path, x_basename = os.path.split(x)
            if re.match('^' + prefix_from, x_basename) is None:
                raise ValueError('Prefix not found in filename: ' + x)
            prefix_file = os.path.join(x_path,
                                       re.sub('^' + prefix_from, prefix, x_basename, count=1))
            if not os.path.isfile(prefix_file):
                raise FileExistsError(prefix_file)
            out_file_list[prefix].append(prefix_file)
    out = {}
    shuffle_idx = {}
    for prefix in prefix_to:
        out[prefix] = load_file_list_to_array(out_file_list[prefix])
        if prefix == 'im' and out[prefix].dtype == 'uint8':
            out[prefix] = out[prefix].astype(np.float32)
            out[prefix] /= 255
        elif prefix in {'mask', 'dmap'}:
            out[prefix] = out[prefix].astype(np.float32)
        elif prefix == 'seg':
            out[prefix] = out[prefix].astype(np.uint8)
        if nblocks > 1:
            out[prefix] = split_images(out[prefix], nblocks=nblocks)
        if prefix == prefix_to[0] and shuffle_seed is not None:
            n = out[prefix].shape[0]  
            shuffle_idx = np.arange(n)
            np.random.seed(shuffle_seed)
            np.random.shuffle(shuffle_idx)
        if shuffle_seed is not None:
            out[prefix] = out[prefix][shuffle_idx, ...]
    if DEBUG:
        i = 5
        plt.clf()
        for pi, prefix in enumerate(prefix_to):
            plt.subplot(1, len(prefix_to), pi+1)
            if out[prefix].shape[-1] == 1:
                plt.imshow(out[prefix][i, :, :, 0])
            else:
                plt.imshow(out[prefix][i, :, :, :])
            plt.title('out[' + prefix + ']')
    return out, out_file_list, shuffle_idx
def load_datasets ( file_list , prefix_from = 'im' , prefix_to = [ ] , nblocks = 1 , shuffle_seed = None ) : if not isinstance ( prefix_to , list ) : raise TypeError ( 'data_prefixes must be a list of strings' ) out_file_list = { } for prefix in prefix_to : out_file_list [ prefix ] = [ ] for x in file_list : x_path , x_basename = os . path . split ( x ) if re . match ( '^' + prefix_from , x_basename ) is None : raise ValueError ( 'Prefix not found in filename: ' + x ) prefix_file = os . path . join ( x_path , re . sub ( '^' + prefix_from , prefix , x_basename , count = 1 ) ) if not os . path . isfile ( prefix_file ) : raise FileExistsError ( prefix_file ) out_file_list [ prefix ] . append ( prefix_file ) out = { } shuffle_idx = { } for prefix in prefix_to : out [ prefix ] = load_file_list_to_array ( out_file_list [ prefix ] ) if prefix == 'im' and out [ prefix ] . dtype == 'uint8' : out [ prefix ] = out [ prefix ] . astype ( np . float32 ) out [ prefix ] /= 255 elif prefix in { 'mask' , 'dmap' } : out [ prefix ] = out [ prefix ] . astype ( np . float32 ) elif prefix == 'seg' : out [ prefix ] = out [ prefix ] . astype ( np . uint8 ) if nblocks > 1 : out [ prefix ] = split_images ( out [ prefix ] , nblocks = nblocks ) if prefix == prefix_to [ 0 ] and shuffle_seed is not None : n = out [ prefix ] . shape [ 0 ] shuffle_idx = np . arange ( n ) np . random . seed ( shuffle_seed ) np . random . shuffle ( shuffle_idx ) if shuffle_seed is not None : out [ prefix ] = out [ prefix ] [ shuffle_idx , ... ] if DEBUG : i = 5 plt . clf ( ) for pi , prefix in enumerate ( prefix_to ) : plt . subplot ( 1 , len ( prefix_to ) , pi + 1 ) if out [ prefix ] . shape [ - 1 ] == 1 : plt . imshow ( out [ prefix ] [ i , : , : , 0 ] ) else : plt . imshow ( out [ prefix ] [ i , : , : , : ] ) plt . title ( 'out[' + prefix + ']' ) return out , out_file_list , shuffle_idx
Loads image files and prepare them for training or testing, returning numpy.ndarrays.
Loads image files and prepare them for training or testing returning numpy . ndarrays .
"""
    Loads image files and prepare them for training or testing, returning numpy.ndarrays.
    Image files can be of any type loadable by the PIL module, but they must have the same size.

    Multiple sets of corresponding images can be loaded using prefix_to. For instance,

    im_file_1.tif          seg_file_1.tif          mask_file_1.tif
    im_file_2.tif          seg_file_2.tif          mask_file_2.tif
    im_file_3.tif          seg_file_3.tif          mask_file_3.tif

    will return

    outs['im']   --> numpy.ndarray (3, rows, cols, channels)
    outs['seg']  --> numpy.ndarray (3, rows, cols, channels)
    outs['mask'] --> numpy.ndarray (3, rows, cols, channels)

    This function also provides the following optional functionality:
        * images can be split into equally-sized blocks (this is useful when full images are too
        large for training).
        * images can be shuffled randomly.

    :param file_list: list of paths and filenames (for one of the datasets).
    :param prefix_from: (def 'im') string with the prefix (e.g. 'im') in file_list that when changed gives the other
    datasets. Note that the prefix refers to the name of the file, not its path.
    :param prefix_to: (def []) list of strings with the prefixes of the datasets that will be loaded. E.g.
    ['im', 'mask'] will load the datasets from the filenames that start with 'im' and 'mask'.
    :param nblocks: (def 1) number of equi-sized blocks to split the images into. Note that the edges of the images may
    need to be trimmed so that splitting creates blocks with the same size.
    :param shuffle_seed: (def None) If provided, images are shuffled after splitting.
    :return: out, out_file_list, shuffle_idx:
       * out: dictionary where out[prefix] contains a numpy.ndarray with the data corresponding to the "prefix" dataset.
       * out_file_list: list of the filenames for the out[prefix] dataset.
       * shuffle_idx: list of indices used to shuffle the images after splitting.
    """ # using prefixes, get list of filenames for the different datasets # load all datasets # load dataset # data type conversions # split image into smaller blocks, if requested # create shuffling indices if required # number of images # shuffle data
out, out_file_list, shuffle_idx:
out: dictionary where out[prefix] contains a numpy.ndarray with the data corresponding to the "prefix" dataset.
out_file_list: list of the filenames for the out[prefix] dataset.
shuffle_idx: list of indices used to shuffle the images after splitting.
out out_file_list shuffle_idx : out : dictionary where out [ prefix ] contains a numpy . ndarray with the data corresponding to the " prefix " dataset . out_file_list : list of the filenames for the out [ prefix ] dataset . shuffle_idx : list of indices used to shuffle the images after splitting .
file_list
list of paths and filenames (for one of the datasets).
list of paths and filenames ( for one of the datasets ) .
prefix_from
(def 'im') string with the prefix  in file_list that when changed gives the other
datasets. Note that the prefix refers to the name of the file, not its path.
( def ' im ' ) string with the prefix in file_list that when changed gives the other datasets . Note that the prefix refers to the name of the file not its path .
prefix_to
(def []) list of strings with the prefixes of the datasets that will be loaded. E.g.
['im', 'mask'] will load the datasets from the filenames that start with 'im' and 'mask'.
( def [] ) list of strings with the prefixes of the datasets that will be loaded . E . g . [ ' im ' ' mask ' ] will load the datasets from the filenames that start with ' im ' and ' mask ' .
nblocks
(def 1) number of equi-sized blocks to split the images into. Note that the edges of the images may
need to be trimmed so that splitting creates blocks with the same size.
( def 1 ) number of equi - sized blocks to split the images into . Note that the edges of the images may need to be trimmed so that splitting creates blocks with the same size .
shuffle_seed
(def None) If provided, images are shuffled after splitting.
( def None ) If provided images are shuffled after splitting .