What is the simplest ReLU network that can learn an upside-down v pattern?
In order to find the network, I tried a couple of different small networks:

One hidden layer with two neurons

This would be the function

$$out = \max(a_1 x + b_1, 0) + \max(a_2x+ b_2, 0)$$

$$a_1 = 1, b_1 = 0, a_2=1, b_2 = -1$$



$$a_1 = 1, b_1 = -5, a_2=-1, b_2 = 5$$



2 hidden layers, 3 neurons in total

This is pretty close:



I guess the pattern is right, but the values are just not quite right. Also, I'm not totally sure if only one hidden layer with 3 neurons (or more) might also work.

I always include biases.

The solution

    / o (ReLU) \
IN o            o (linear)
    \ o (ReLU) /


It is basically the 1 hidden layer with 2 neurons, but also using the scaling (-1?) and the bias (+5?) of the output layer.

$$(-1) \cdot (\max(x - 5, 0) + \max(-x + 5, 0))+5$$