Linear Regression using a Neural Network I am trying to create a regression model using a Neural Network. I am currently learning how to work with neural networks (deeplearning.ai) and so the model is not implemented using any existing frameworks like keras.

Based on what I have learned, 


the model is configured to use no activation function in the output
layer (which, obviously, has only one node).
Input is images of fashion articles (shirt, jackets etc).
output predicts prices for input article images.
Hidden layers all use ReLU.
Random initialization is done for all weights.
Cost function is Mean Squared Error:
$$J = \frac{1}{2m}\sum_{i=1}^m (a^{(i)} - y^{(i)})^2$$


Based on the formula for backpropogation, the last layer should get the error based on cost function. For a single example, we have:
$$ \mathcal L =  \frac{1}{2}(a - y)^2 $$
$$ error = \frac{d\mathcal L}{da} = (a - y) $$

where L is the loss function, a is the predicted value and y is the actual price of the article.

For the entire training set, we have (vectorized):
$$ error = \frac{d\mathcal L}{dA} = (A - Y) $$
where m is the number of examples,  A, Y are (1 * m) vectors where each value corresponds to each single example. A contains m predictions and Y has all the m prices.

Is this error value correct?

My problem is that the model converges to a what appears to be a local minima. The error after a few 100 iterations gets stuck. The error is also not very small (~ 0.0005). I am not sure if I have the equations right.
No.  This is not the correct way to train such a network.  To update the network, we want to update the weights of the network.  Thus instead of using $\frac{d\mathcal L}{dA}$ to update the weights, you should be using $\frac{d\mathcal L}{dW}$, where $W$ are the weights.  This could be computed as

$$\frac{d\mathcal L}{dW} = \frac{d\mathcal L}{dA} \times \frac{d\mathcal A}{dW},$$

where $\frac{d\mathcal L}{dA}$ is computed using the formula you obtained ($\frac{d\mathcal L}{dA} = A-Y$ is correct), and where $\frac{d\mathcal A}{dW}$ is computed using backpropagation through the network.

Or, better yet, use an existing framework that performs automatic differentiation for you (e.g., Keiras, Tensorflow), and save yourself a lot of effort.