title: Tensorflow: how do I extract/export variable values at every iteration of training?
body: <p>I have been playing around with some neural networks on Tensorflow and I wanted to make a visualization of the neural network's learning process. 
  To do so, I intend to extract the following variables into text/JSON/csv: pre-activation result before 1st layer, activation, bias and weight values for testing and training, each layer and for all time steps. I am looking for a generalizable solution so that I don't have to modify my source code (or at least not more than one or two lines) when applying visualization to future networks. Ideally I could run some function from another python program to read any python/TF code and extract the variables described above. So far I have considered the following solutions:
  1) use tf.summary and the filewriter to save as a serialized protocol buffer, then find a way to go from protocol buffer --> JSON format. This unfortunately would not fit the bill as it requires me to modify too much inner code. 
  2) Perhaps using <a href="https://www.tensorflow.org/api_docs/python/tf/train/export_meta_graph" rel="nofollow noreferrer">https://www.tensorflow.org/api_docs/python/tf/train/export_meta_graph</a>
  Although I am not sure how to implement given my TF foundations are not quite there yet
  3) I have also found this solution:</p>
  
  <pre><code>W_val, b_val= sess.run([W, b])
  np.savetxt("W1.csv", W_val, delimiter=",")
  np.savetxt("b1.csv", b_val, delimiter=",")
  </code></pre>
  
  <p>But the problem is that it only saves the final values of the weights and biases, whereas I am looking to save their values at all timesteps of training. </p>
  
  <p>If anyone has any suggestions on how to tackle this problem or any guidance I would appreciate it. </p>
  
  <p>Many thanks</p>