Tensorflow uses more memory, the more epochs it completes I created a genetic algorithm "optimizer" for Tensorflow but it is written in python. I know TensorFlow was not designed like this and I need to rather create the optimizer in C++ using their API's but I found out about it only after I already programmed the optimizer and I don't really have the time to do research on how to create a tf.train.Optimizer(). My GA "optimizer" is not really an optimizer as it is more a tf.Variable.assign() call. See code below

The "optimizer" updated a series of weight matrix variables after the evolve takes place. Each variable represents a weight matrix between two layers. If it is a network with inputs, one hidden layer and outputs then there will be two variables. Weights between input to hidden then hidden to output. Although this is a 2d matrix but my weights act as an individual in my population so I have a 3d matrix. 1 dimension represents the population amount and the 2d matrix as the weigths. Weight updates are then done by crossover or mutation which consists of a lot of multiplication, matmul and summation operations. My guess is the variables I create for the new layers is not garbage collected. I am guessing tensorflow keeps the variables alive as I don't ever call a train step provided by the tf.train.Optimzer classes since I created my own. 

Here is my code to run my GA optimizer

for i in range(epochs):
    sess.run(fitness_per_individual, feed_dict={x: train_data, y: train_labels},
                 options=run_options,
                 run_metadata=run_metadata)    
    evolved_layers = evolve_layer(nn_layers, fitness_per_individual, population)
    for tensor_idx in range(len(nn_layers)):
        layer = nn_layers[tensor_idx]
        evolved_layer = evolved_layers[tensor_idx]
        layer = layer.assign(evolved_layer)

        # I don't have to feed my training data again I just do it here to keep tensorflow happy.
        sess.run(layer, feed_dict={x: train_data, y: train_labels},
                 options=run_options,
                 run_metadata=run_metadata)


Is there a way I can dispose of variable matrices created by tf.multiply or is there no way around it without creating a tf.train.Optimizer. Also is there a good practice guideline I can read up for TensorFlow.
It seems like you are adding nodes to your computational graph in this line: layer = layer.assign(evolved_layer). The assign-operation is just as multiplication or addition a node in the graph and you construct a new one in every step. Call tf.Graph.finalize() after defining your model. This will raise an error whenever a node is added to the graph and help you debugging your code. Ideally you want the graph to be fixed after construction.  I suspect you will get an error in the optimizer.

You want all the operations to be defined before you run the training loop and then just run the respective nodes. In your case you'd define layer_assign = layer.assign(evolved_layer) before the loop and in the loop only use sess.run(layer_assign). Of course you would have to treat all operations in evolve_layers in the same way.