Extracting multiple softmax values from image
It is not unheard of to share network weights with multiple output layers. I have seen it on DeepMind's Asynchronous Deep Learning paper, and I have also used it here.

The idea is to share all the layers and just have multiple outputs. However, this might decrease the accuracy of your networks, as is the usual performance VS accuracy trade-off.



To optimize this, just calculate the loss of both outputs and sum them when feeding the optimizer, like this

            self.target_policy_fast_t = tf.placeholder('float32', [None, a_size], name='target_policy_fast_t')
            self.loss_policy_fast = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=self.network.policy_fast_before_softmax, labels=self.loss_policy_fast_t))

            self.target_policy_slow_t = tf.placeholder('float32', [None, a_size], name='target_policy_slow_t')
            self.loss_policy_slow = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=self.network.policy_slow_before_softmax, labels=self.target_policy_slow_t))

            self.loss = tf.reduce_mean(self.loss_policy_fast + self.loss_policy_slow, name='loss')