Can I safely do inference from another thread while training the network?
Do your inference calls need to be on an up-to-date version of the graph? If you don't mind some delay, you could make a copy of the graph by calling sess.graph.as_graph_def on the training thread, and then create a new session on the inference thread using that graph_def periodically.
I have several threads that either update the weights of my network or run inference on it. I use the use_locking parameter for the optimizer to prevent concurrent updates of the weights.
Inference should always use a recent, and importantly, consistent, version of the weights. In other words, I want to prevent using a weight matrix for inference for which some of the elements are already updated but others are not.
Is this guaranteed? If not, how can I ensure this? There doesn't seem to be a tf.Lock or similar.