Tensorflow uses more memory, the more epochs it completes
It seems like you are adding nodes to your computational graph in this line: layer = layer.assign(evolved_layer). The assign-operation is just as multiplication or addition a node in the graph and you construct a new one in every step. Call tf.Graph.finalize() after defining your model. This will raise an error whenever a node is added to the graph and help you debugging your code. Ideally you want the graph to be fixed after construction.  I suspect you will get an error in the optimizer.

You want all the operations to be defined before you run the training loop and then just run the respective nodes. In your case you'd define layer_assign = layer.assign(evolved_layer) before the loop and in the loop only use sess.run(layer_assign). Of course you would have to treat all operations in evolve_layers in the same way.