resunet/utils.py
def preprocess(params, unproc, submean=False, imagesize=128):
    """
    Function to standardize image data before being used with the deep learning
    network.

    Performs a standardizing procedure on image data that has already been split
    into training, validation, and test sets using the resunet.utils.split_dataset function.
    Performs the procedure:
    X_\mathrm{Processed} = frac{( X - \overline{X}_{Training})}{\sigma_{Training}}.

    Where X is a map from any map type. For more information on the process refer to
    "Reconstructing Patchy Reinoization with Deep Learning."

    Parameters
    ----------
    params :
        Class object resunet.utils.Params. Params is a container for the variables
        defined in the configuration file.
    unproc :
        Dictionary of unprocessed data. Has a format where key is the name of map type and
        the value is the array of all maps of that key type. This is usually the output of
        the resunet.utils.split_dataset function.
    submean : boolean
        If true, the mean of the maps, across the set of each map type, will be calculated
        and subtracted before finding the standard deviation.
    imagesize : int
        Pixel image size of all the maps.

    Returns
    -------
    map_stds.npz :
        Saves standard deviations calculated to a '.npz' file in the working directory.
    map_means.npz :
        Saves means calculated to a '.npz' file in the working directory. This is only done
        if submean is True.
    unproc :
        Dictionary of processed, standardized, CMB maps ready for use with the deep learning
        network.
    """

    stdvals = {}
    meanvals = {}
    for i in [params.feat_used, params.target_used]:
        for key, value in i.items():
            # both values will be calculated from the training data set
            if submean:
                meanvals[value+"_mean"] = np.mean(unproc[value+"_train"])
                stdvals[value+"_std"] = np.std(unproc[value+"_train"] - meanvals[value+"_mean"])
            else:
                stdvals[value+"_std"] = np.std(unproc[value+"_train"])

    np.savez(
        os.path.join(params.output_dir, "map_stds"), **stdvals)
    np.savez(
        os.path.join(params.output_dir, "map_means"), **meanvals)

    # standardizing data
    for i in [params.feat_used, params.target_used]:
        for key, value in i.items():
            unproc[value+"_train"] = (unproc[value+"_train"] - meanvals[value+"_mean"]) / stdvals[value+"_std"]
            unproc[value+"_val"] = (unproc[value+"_val"] - meanvals[value+"_mean"]) / stdvals[value+"_std"]
            unproc[value+"_test"] = (unproc[value+"_test"] - meanvals[value+"_mean"]) / stdvals[value+"_std"]

    for key, value in unproc.items():
        unproc[key] = np.reshape(unproc[key], (value.shape[0], imagesize, imagesize, 1))

    return unproc

    Function to standardize image data before being used with the deep learning
    network.

    Performs a standardizing procedure on image data that has already been split
    into training, validation, and test sets using the resunet.utils.split_dataset function.
    Performs the procedure:
    X_\mathrm{Processed} = frac{( X - \overline{X}_{Training})}{\sigma_{Training}}.

    Where X is a map from any map type. For more information on the process refer to
    "Reconstructing Patchy Reinoization with Deep Learning."

    Parameters
    ----------
    params :
        Class object resunet.utils.Params. Params is a container for the variables
        defined in the configuration file.
    unproc :
        Dictionary of unprocessed data. Has a format where key is the name of map type and
        the value is the array of all maps of that key type. This is usually the output of
        the resunet.utils.split_dataset function.
    submean : boolean
        If true, the mean of the maps, across the set of each map type, will be calculated
        and subtracted before finding the standard deviation.
    imagesize : int
        Pixel image size of all the maps.

    Returns
    -------
    map_stds.npz :
        Saves standard deviations calculated to a '.npz' file in the working directory.
    map_means.npz :
        Saves means calculated to a '.npz' file in the working directory. This is only done
        if submean is True.
    unproc :
        Dictionary of processed, standardized, CMB maps ready for use with the deep learning
        network.
    
Function to standardize image data before being used with the deep learning
network.

Performs a standardizing procedure on image data that has already been split
into training, validation, and test sets using the resunet.utils.split_dataset function.
Performs the procedure:
X_\mathrm{Processed} = frac{( X - \overline{X}_{Training})}{\sigma_{Training}}.



Parameters

params :
Class object resunet.utils.Params. Params is a container for the variables
defined in the configuration file.
unproc :
Dictionary of unprocessed data. Has a format where key is the name of map type and
the value is the array of all maps of that key type. This is usually the output of
the resunet.utils.split_dataset function.
submean : boolean
If true, the mean of the maps, across the set of each map type, will be calculated
and subtracted before finding the standard deviation.
imagesize : int
Pixel image size of all the maps.

Returns

map_stds.npz :
Saves standard deviations calculated to a '.npz' file in the working directory.
map_means.npz :
Saves means calculated to a '.npz' file in the working directory. This is only done
if submean is True.
unproc :
Dictionary of processed, standardized, CMB maps ready for use with the deep learning
network.
Function to standardize image data before being used with the deep learning network . Performs a standardizing procedure on image data that has already been split into training validation and test sets using the resunet . utils . split_dataset function . Performs the procedure : X_ \ mathrm { Processed } = frac { ( X - \ overline { X } _ { Training } ) } { \ sigma_ { Training }} . Parameters params : Class object resunet . utils . Params . Params is a container for the variables defined in the configuration file . unproc : Dictionary of unprocessed data . Has a format where key is the name of map type and the value is the array of all maps of that key type . This is usually the output of the resunet . utils . split_dataset function . submean : boolean If true the mean of the maps across the set of each map type will be calculated and subtracted before finding the standard deviation . imagesize : int Pixel image size of all the maps . Returns map_stds . npz : Saves standard deviations calculated to a ' . npz ' file in the working directory . map_means . npz : Saves means calculated to a ' . npz ' file in the working directory . This is only done if submean is True . unproc : Dictionary of processed standardized CMB maps ready for use with the deep learning network .
def preprocess(params, unproc, submean=False, imagesize=128):
    stdvals = {}
    meanvals = {}
    for i in [params.feat_used, params.target_used]:
        for key, value in i.items():
            if submean:
                meanvals[value+"_mean"] = np.mean(unproc[value+"_train"])
                stdvals[value+"_std"] = np.std(unproc[value+"_train"] - meanvals[value+"_mean"])
            else:
                stdvals[value+"_std"] = np.std(unproc[value+"_train"])
    np.savez(
        os.path.join(params.output_dir, "map_stds"), **stdvals)
    np.savez(
        os.path.join(params.output_dir, "map_means"), **meanvals)
    for i in [params.feat_used, params.target_used]:
        for key, value in i.items():
            unproc[value+"_train"] = (unproc[value+"_train"] - meanvals[value+"_mean"]) / stdvals[value+"_std"]
            unproc[value+"_val"] = (unproc[value+"_val"] - meanvals[value+"_mean"]) / stdvals[value+"_std"]
            unproc[value+"_test"] = (unproc[value+"_test"] - meanvals[value+"_mean"]) / stdvals[value+"_std"]
    for key, value in unproc.items():
        unproc[key] = np.reshape(unproc[key], (value.shape[0], imagesize, imagesize, 1))
    return unproc
def preprocess ( params , unproc , submean = False , imagesize = 128 ) : stdvals = { } meanvals = { } for i in [ params . feat_used , params . target_used ] : for key , value in i . items ( ) : if submean : meanvals [ value + "_mean" ] = np . mean ( unproc [ value + "_train" ] ) stdvals [ value + "_std" ] = np . std ( unproc [ value + "_train" ] - meanvals [ value + "_mean" ] ) else : stdvals [ value + "_std" ] = np . std ( unproc [ value + "_train" ] ) np . savez ( os . path . join ( params . output_dir , "map_stds" ) , ** stdvals ) np . savez ( os . path . join ( params . output_dir , "map_means" ) , ** meanvals ) for i in [ params . feat_used , params . target_used ] : for key , value in i . items ( ) : unproc [ value + "_train" ] = ( unproc [ value + "_train" ] - meanvals [ value + "_mean" ] ) / stdvals [ value + "_std" ] unproc [ value + "_val" ] = ( unproc [ value + "_val" ] - meanvals [ value + "_mean" ] ) / stdvals [ value + "_std" ] unproc [ value + "_test" ] = ( unproc [ value + "_test" ] - meanvals [ value + "_mean" ] ) / stdvals [ value + "_std" ] for key , value in unproc . items ( ) : unproc [ key ] = np . reshape ( unproc [ key ] , ( value . shape [ 0 ] , imagesize , imagesize , 1 ) ) return unproc
Function to standardize image data before being used with the deep learning
network.
Function to standardize image data before being used with the deep learning network .
"""
    Function to standardize image data before being used with the deep learning
    network.

    Performs a standardizing procedure on image data that has already been split
    into training, validation, and test sets using the resunet.utils.split_dataset function.
    Performs the procedure:
    X_\mathrm{Processed} = frac{( X - \overline{X}_{Training})}{\sigma_{Training}}.

    Where X is a map from any map type. For more information on the process refer to
    "Reconstructing Patchy Reinoization with Deep Learning."

    Parameters
    ----------
    params :
        Class object resunet.utils.Params. Params is a container for the variables
        defined in the configuration file.
    unproc :
        Dictionary of unprocessed data. Has a format where key is the name of map type and
        the value is the array of all maps of that key type. This is usually the output of
        the resunet.utils.split_dataset function.
    submean : boolean
        If true, the mean of the maps, across the set of each map type, will be calculated
        and subtracted before finding the standard deviation.
    imagesize : int
        Pixel image size of all the maps.

    Returns
    -------
    map_stds.npz :
        Saves standard deviations calculated to a '.npz' file in the working directory.
    map_means.npz :
        Saves means calculated to a '.npz' file in the working directory. This is only done
        if submean is True.
    unproc :
        Dictionary of processed, standardized, CMB maps ready for use with the deep learning
        network.
    """ # both values will be calculated from the training data set # standardizing data
params
unproc
submean
imagesize