rfho/models.py
def vectorize_model(model_vars, *o_outs, augment=0, suppress_err_out=True):
    """
    Function that "vectorizes" a model (as a computation graph).

    Given a model written in a "standard way", i.e. with parameters organized  in k-rank tensors as needed
    (matrices and vectors for linearities, 3 or 4 rank tensors for convolutional kernels etc.), returns the same model
    with all the parameters organized in a single vector.

    It is believed that having coded the models in this way, it will be much simpler to implement future algorithms,
    especially when relying on the second-order derivatives (Hessian of objective function), since in such a framework,
    the model will be entirely dependent by its single parameter vector and described by the resulting computation
    graph.

    Since the the downward process of vectorizing the model does not work, it is
     necessary to modify the computation graph accordingly to have an upward dependence
    from the all weights vector to the single parameters.

    SOLVED! { The technical caveat is that the variables must be encapsulated into an op (tf.identity) otherwise,
    for some
    unknown reason, the substitution process is not possible. I've tried in several different ways but they all failed.
    Substantially you need to keep track of the tf.identity(variable) and use the resulting tensor to build up the model
    and then also of the initial value of variable. Probably it is not necessary to keep track of  variable itself. }

    :param model_vars: list of variables of the model or initializers
    :param o_outs: output_variables, list or tensor. (e.g. model output)
    :param augment: (int: default 0) augment the all weights vector by creating augumented variables (initialized at 0)
                    that mirror rank and dimensions of the variables in `model_vars`. The process is repeated
                    `augment` times.
                    This new variables can be  accessed with methods in `MergedVariable`.
                    The common usage is to prepare the model to be optimized with optimizers that require states
                    such as `MomentumOptimizer` (`augment=1`) or `AdamOptimizer` (`augment=2`).
    :param suppress_err_out: if `True` (default) suppress tensorflow warnings and outputs.
                                This might cause system error in very very few cases...

    :return: a list which has as first element the `MergedVariable` that represents the all weights vector. Remaining
                elements are the outputs
                in the modified graph. These new outs are the same computed by the initial model
            (by the computation graph in which the model lives) but with dependencies form the all_weight_vector.

    """
    # assert len(model_vars) == len(model_vars_tensor), 'length of model_vars and model_var_tensor do not match'
    assert len(model_vars) > 0, 'no variables in model_vars!'
    outs = [tf.identity(o) if isinstance(o, tf.Variable) else o for o in o_outs]
    # TODO implement recursive call for nested lists
    with model_vars[0].graph.as_default():

        true_w = MergedVariable(model_vars)

        if augment:
            augmented_variables = [true_w]
            for k in range(augment):
                with tf.name_scope('augmented_variables'):
                    with tf.name_scope(str(k)):
                        with tf.name_scope('original_variables'):
                            tmp_augmented = [tf.Variable(tf.zeros(v.get_shape().as_list()), name=v.name.split(':')[0])
                                             for v in model_vars]
                        augmented_variables.append(MergedVariable(tmp_augmented))

            w = MergedVariable(augmented_variables)
        else:
            w = true_w

        # with open(os.devnull, 'w') as dnf:
        # with utils.suppress_stdout_stderr():  # may cause ERROR TOO MANY FILES OPENED!.:((

        if suppress_err_out:  # FIXME deprecation here on GraphKey usage... now redirecting outs... should fix tf..
            with utils.suppress_stdout_stderr():
                new_outs = ge.graph_replace(outs, w.generate_swap_dict())
        else: new_outs = ge.graph_replace(outs, w.generate_swap_dict())

    return [w] + new_outs

    Function that "vectorizes" a model (as a computation graph).

    Given a model written in a "standard way", i.e. with parameters organized  in k-rank tensors as needed
    (matrices and vectors for linearities, 3 or 4 rank tensors for convolutional kernels etc.), returns the same model
    with all the parameters organized in a single vector.

    It is believed that having coded the models in this way, it will be much simpler to implement future algorithms,
    especially when relying on the second-order derivatives (Hessian of objective function), since in such a framework,
    the model will be entirely dependent by its single parameter vector and described by the resulting computation
    graph.

    Since the the downward process of vectorizing the model does not work, it is
     necessary to modify the computation graph accordingly to have an upward dependence
    from the all weights vector to the single parameters.

    SOLVED! { The technical caveat is that the variables must be encapsulated into an op (tf.identity) otherwise,
    for some
    unknown reason, the substitution process is not possible. I've tried in several different ways but they all failed.
    Substantially you need to keep track of the tf.identity(variable) and use the resulting tensor to build up the model
    and then also of the initial value of variable. Probably it is not necessary to keep track of  variable itself. }

    :param model_vars: list of variables of the model or initializers
    :param o_outs: output_variables, list or tensor. (e.g. model output)
    :param augment: (int: default 0) augment the all weights vector by creating augumented variables (initialized at 0)
                    that mirror rank and dimensions of the variables in `model_vars`. The process is repeated
                    `augment` times.
                    This new variables can be  accessed with methods in `MergedVariable`.
                    The common usage is to prepare the model to be optimized with optimizers that require states
                    such as `MomentumOptimizer` (`augment=1`) or `AdamOptimizer` (`augment=2`).
    :param suppress_err_out: if `True` (default) suppress tensorflow warnings and outputs.
                                This might cause system error in very very few cases...

    :return: a list which has as first element the `MergedVariable` that represents the all weights vector. Remaining
                elements are the outputs
                in the modified graph. These new outs are the same computed by the initial model
            (by the computation graph in which the model lives) but with dependencies form the all_weight_vector.

    
Function that "vectorizes" a model (as a computation graph).
Given a model written in a "standard way", i.e. with parameters organized  in k-rank tensors as needed
(matrices and vectors for linearities, 3 or 4 rank tensors for convolutional kernels etc.), returns the same model
with all the parameters organized in a single vector.

It is believed that having coded the models in this way, it will be much simpler to implement future algorithms,
especially when relying on the second-order derivatives (Hessian of objective function), since in such a framework,
the model will be entirely dependent by its single parameter vector and described by the resulting computation
graph.

Since the the downward process of vectorizing the model does not work, it is
necessary to modify the computation graph accordingly to have an upward dependence
from the all weights vector to the single parameters.

SOLVED. { The technical caveat is that the variables must be encapsulated into an op (tf.identity) otherwise,
for some
unknown reason, the substitution process is not possible. I've tried in several different ways but they all failed.
Substantially you need to keep track of the tf.identity(variable) and use the resulting tensor to build up the model
and then also of the initial value of variable. Probably it is not necessary to keep track of  variable itself.
Function that " vectorizes " a model ( as a computation graph ) . Given a model written in a " standard way " i . e . with parameters organized in k - rank tensors as needed ( matrices and vectors for linearities 3 or 4 rank tensors for convolutional kernels etc . ) returns the same model with all the parameters organized in a single vector . It is believed that having coded the models in this way it will be much simpler to implement future algorithms especially when relying on the second - order derivatives ( Hessian of objective function ) since in such a framework the model will be entirely dependent by its single parameter vector and described by the resulting computation graph . Since the the downward process of vectorizing the model does not work it is necessary to modify the computation graph accordingly to have an upward dependence from the all weights vector to the single parameters . SOLVED . { The technical caveat is that the variables must be encapsulated into an op ( tf . identity ) otherwise for some unknown reason the substitution process is not possible . I ' ve tried in several different ways but they all failed . Substantially you need to keep track of the tf . identity ( variable ) and use the resulting tensor to build up the model and then also of the initial value of variable . Probably it is not necessary to keep track of variable itself .
def vectorize_model(model_vars, *o_outs, augment=0, suppress_err_out=True):
    assert len(model_vars) > 0, 'no variables in model_vars!'
    outs = [tf.identity(o) if isinstance(o, tf.Variable) else o for o in o_outs]
    with model_vars[0].graph.as_default():
        true_w = MergedVariable(model_vars)
        if augment:
            augmented_variables = [true_w]
            for k in range(augment):
                with tf.name_scope('augmented_variables'):
                    with tf.name_scope(str(k)):
                        with tf.name_scope('original_variables'):
                            tmp_augmented = [tf.Variable(tf.zeros(v.get_shape().as_list()), name=v.name.split(':')[0])
                                             for v in model_vars]
                        augmented_variables.append(MergedVariable(tmp_augmented))
            w = MergedVariable(augmented_variables)
        else:
            w = true_w
        if suppress_err_out:  
            with utils.suppress_stdout_stderr():
                new_outs = ge.graph_replace(outs, w.generate_swap_dict())
        else: new_outs = ge.graph_replace(outs, w.generate_swap_dict())
    return [w] + new_outs
def vectorize_model ( model_vars , * o_outs , augment = 0 , suppress_err_out = True ) : assert len ( model_vars ) > 0 , 'no variables in model_vars!' outs = [ tf . identity ( o ) if isinstance ( o , tf . Variable ) else o for o in o_outs ] with model_vars [ 0 ] . graph . as_default ( ) : true_w = MergedVariable ( model_vars ) if augment : augmented_variables = [ true_w ] for k in range ( augment ) : with tf . name_scope ( 'augmented_variables' ) : with tf . name_scope ( str ( k ) ) : with tf . name_scope ( 'original_variables' ) : tmp_augmented = [ tf . Variable ( tf . zeros ( v . get_shape ( ) . as_list ( ) ) , name = v . name . split ( ':' ) [ 0 ] ) for v in model_vars ] augmented_variables . append ( MergedVariable ( tmp_augmented ) ) w = MergedVariable ( augmented_variables ) else : w = true_w if suppress_err_out : with utils . suppress_stdout_stderr ( ) : new_outs = ge . graph_replace ( outs , w . generate_swap_dict ( ) ) else : new_outs = ge . graph_replace ( outs , w . generate_swap_dict ( ) ) return [ w ] + new_outs
Function that "vectorizes" a model (as a computation graph).
Function that " vectorizes " a model ( as a computation graph ) .
"""
    Function that "vectorizes" a model (as a computation graph).

    Given a model written in a "standard way", i.e. with parameters organized  in k-rank tensors as needed
    (matrices and vectors for linearities, 3 or 4 rank tensors for convolutional kernels etc.), returns the same model
    with all the parameters organized in a single vector.

    It is believed that having coded the models in this way, it will be much simpler to implement future algorithms,
    especially when relying on the second-order derivatives (Hessian of objective function), since in such a framework,
    the model will be entirely dependent by its single parameter vector and described by the resulting computation
    graph.

    Since the the downward process of vectorizing the model does not work, it is
     necessary to modify the computation graph accordingly to have an upward dependence
    from the all weights vector to the single parameters.

    SOLVED! { The technical caveat is that the variables must be encapsulated into an op (tf.identity) otherwise,
    for some
    unknown reason, the substitution process is not possible. I've tried in several different ways but they all failed.
    Substantially you need to keep track of the tf.identity(variable) and use the resulting tensor to build up the model
    and then also of the initial value of variable. Probably it is not necessary to keep track of  variable itself. }

    :param model_vars: list of variables of the model or initializers
    :param o_outs: output_variables, list or tensor. (e.g. model output)
    :param augment: (int: default 0) augment the all weights vector by creating augumented variables (initialized at 0)
                    that mirror rank and dimensions of the variables in `model_vars`. The process is repeated
                    `augment` times.
                    This new variables can be  accessed with methods in `MergedVariable`.
                    The common usage is to prepare the model to be optimized with optimizers that require states
                    such as `MomentumOptimizer` (`augment=1`) or `AdamOptimizer` (`augment=2`).
    :param suppress_err_out: if `True` (default) suppress tensorflow warnings and outputs.
                                This might cause system error in very very few cases...

    :return: a list which has as first element the `MergedVariable` that represents the all weights vector. Remaining
                elements are the outputs
                in the modified graph. These new outs are the same computed by the initial model
            (by the computation graph in which the model lives) but with dependencies form the all_weight_vector.

    """ # assert len(model_vars) == len(model_vars_tensor), 'length of model_vars and model_var_tensor do not match' # TODO implement recursive call for nested lists # with open(os.devnull, 'w') as dnf: # with utils.suppress_stdout_stderr():  # may cause ERROR TOO MANY FILES OPENED!.:(( # FIXME deprecation here on GraphKey usage... now redirecting outs... should fix tf..
a list which has as first element the `MergedVariable` that represents the all weights vector. Remaining
elements are the outputs
in the modified graph. These new outs are the same computed by the initial model
(by the computation graph in which the model lives) but with dependencies form the all_weight_vector.
a list which has as first element the ` MergedVariable ` that represents the all weights vector . Remaining elements are the outputs in the modified graph . These new outs are the same computed by the initial model ( by the computation graph in which the model lives ) but with dependencies form the all_weight_vector .
model_vars
list of variables of the model or initializers
list of variables of the model or initializers
augment
default 0) augment the all weights vector by creating augumented variables (initialized at 0)
that mirror rank and dimensions of the variables in `model_vars`. The process is repeated
`augment` times.
This new variables can be  accessed with methods in `MergedVariable`.
The common usage is to prepare the model to be optimized with optimizers that require states
such as `MomentumOptimizer` (`augment=1`) or `AdamOptimizer` (`augment=2`).
default 0 ) augment the all weights vector by creating augumented variables ( initialized at 0 ) that mirror rank and dimensions of the variables in ` model_vars ` . The process is repeated ` augment ` times . This new variables can be accessed with methods in ` MergedVariable ` . The common usage is to prepare the model to be optimized with optimizers that require states such as ` MomentumOptimizer ` ( ` augment = 1 ` ) or ` AdamOptimizer ` ( ` augment = 2 ` ) .
suppress_err_out
if `True` (default) suppress tensorflow warnings and outputs.
This might cause system error in very very few cases
if ` True ` ( default ) suppress tensorflow warnings and outputs . This might cause system error in very very few cases
o_outs
output_variables, list or tensor.
output_variables list or tensor .