Backpropagation of neural nets with shared weight I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.

Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (${f(x) = x}$) at all layer. Now consider that this neural network takes two $n$-dimensional inputs $X^{1}$ and $X^{2}$. However, the first hidden layer only takes $X^1$ as the input and produces the output of $H^1$. The second hidden layer takes $H^{1} $and $X^2$ as the input and produces the output $H^{2}$. The output layer takes $H^{2}$ as the input and produces the output $\hat{Y}$. For simplicity, assume, we do not have any bias.

So, we can write that, $H^1 = W^{x1}X^{1}$

$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$ [substituting the value of $H^1$]

$\hat{Y} = W^{y}H^2$

Here, $W^{x1}$, $W^{x2}$, $W^{h}$ and $W^{y}$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $W^{x} = W^{x1} = W^{x2}$, which leads, $H^1 = W^{x}X^{1}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $W^{x}$.

$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$ 

Substituting, $\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$

Here I understand that, $\frac{\partial Y}{\partial H^{2}} = (W^y)^T$ and $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$ and we can also calculate $\frac{\partial Y}{\partial H^{2}}$, if we know the loss function. But how do we calculate $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$?
If we write $ H^2 = W^{h}H1 + W^{x}X^{2} $ then it will be better to understand the backward propagation step.

Now,

$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$ can be written as:
$\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $

$\frac{\partial H^2}{\partial H^1} =  (W^h)^T$ and 
$\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $

Therefore,

$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T  $

I hope it has solved your problem.