Role of activation function in calculating the cost function for artificial neural networks
The value you're comparing your actual results to for the cost function doesn't (intrinsically) have anything to do with the input you used to get the output. It doesn't get transformed in any way.
Your expected value is [10,200,3] but you used Softmax on the output layer and RMSE loss? Well, too bad, you're gonna have a high cost all the time (and the model probably won't converge).
It's just on you to use the right cost functions to serve as a sane heuristic for evaluating the model performance and the right activations to be able to get sane outputs for the task at hand.
I have some difficulty with understanding the role of activation functions and cost functions. Lets take a look at a simple example. Lets say I am building a neural network (artificial neural network). I have 5 „x“ variables and one „y“ variable. 
If I do usual feature scaling and then apply, for example, Relu activation function in hidden layer, then this activation function does the transformation and as a result we get our predicted output value (y hat) between 0 and lets say M. Then the next step is to calculate the cost function. 
In calculating the cost function, however, we need to compare the output value (y hat) with the actual value (y).
The question is how we can compare transformed output value (y hat) which is lets say between 0 and M with the untransformed actual value (y) (which can be any number as it is not been subjected to the Relu activation function) to calculate the cost function? There can be a large mismatch as one variable has been exposed to transformation and the other has not been.
Thank you for any help.