Is it suitable to find inverse of last layer's activation function and apply it on the target output? I have a neural network with the following structure:


I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the upper output node and -0.3 for the lower output node.

The activations function used for the first 2 layers are ReLu or LeakyReLu while the last layer uses atan as an activation function.

For back propogation, instead of adjusting values to make the network's output approach 0.8,-0.3. is it suitable if I use the inverse function for atan -which is tan itself- to get "the ideal input to the output layer multiplied by weights and adjusted by biases". 

The tan of 0.8 and -0.3 is 0.01396 and -0.00524 approximately. 

My algorithm would then adjust weights and biases of the network so that the "pre-activated output" of the output layer -which is basically (sum(output_layer_weight*output_layer's inputs)+output_layer_biases)- approaches 0.01396 and -0.00524.

Is this suitable
The main difference your change would have is to allow you to apply a loss function to a different part of the network. This may affect training.

If you keep the same loss function (e.g. MSE), but apply it to the pre-transformed values, then you will have changed the objective of the network, perhaps significantly. Whether or not this is a good thing depends on how much you needed the original loss function. However, the fact that it would result in a different training target is usually going to be a bad thing if your original training target was correct. This will also be true if you pick a new arbitrary loss function that seems to fit the pre-transform representation better.

If you engineer a "correct" loss function such that the objective of the network remains unchanged, then the behaviour of the network will not change much - probably not at all. However, in some cases this can lead to more stable and/or faster training - it is often used for classifiers to avoid need to use exponentiation, see tf.nn.softmax_cross_entropy_with_logits in TensorFlow, which does exactly this.