Machine Learning Golf: Multiplication I'd like to propose a different kind of golfing challenge to this community:

(Artificial) Neural Networks are very popular machine learning models that can be designed and trained to approximate any given (usually unknown) function. They're often used to solve highly complex problems that we don't know how to solve algorithmically like speech recognition, certain kinds of image classifications, various tasks in autonomous driving systems, ... For a primer on neural networks, consider this excellent Wikipedia article.

As this is the first in what I hope to be a series of machine learning golf challenges, I'd like to keep things as simple as possible:

In the language and framework of your choice, design and train a neural network that, given \$(x_1, x_2)\$ calculates their product \$x_1 \cdot x_2\$ for all integers \$x_1, x_2\$ between (and including) \$-10\$ and \$10\$.

Performance Goal 

To qualify, your model may not deviate by more than \$0.5\$ from the correct result on any of those entries.

Rules

Your model                                 


must be a 'traditional' neural network (a node's value is calculated as a weighted linear combination of some of the nodes in a previous layer followed by an activation function),
may only use the following standard activation functions:


\$\textrm{linear}(x) = x\$,
\$\textrm{softmax}(\vec{x})_i = \frac{e^{x_i}}{\sum_j e^{x_j}}\$,
\$\textrm{selu}_{\alpha, \beta}(x) = \begin{cases}
\beta \cdot x &amp; \text{, if } x &gt; 0 \\
\alpha \cdot \beta (e^x -1 ) &amp; \text{, otherwise}
\end{cases}\$,
\$\textrm{softplus}(x) = \ln(e^x+1)\$,
\$\textrm{leaky-relu}_\alpha(x) = \begin{cases}
x &amp; \text{, if } x &lt; 0 \\
\alpha \cdot x &amp; \text{, otherwise}
\end{cases}\$,
\$\tanh(x)\$,
\$\textrm{sigmoid}(x) = \frac{e^x}{e^x+1}\$,
\$\textrm{hard-sigmoid}(x) = \begin{cases}
0 &amp; \text{, if } x &lt; -2.5 \\
1 &amp; \text{, if } x &gt; 2.5 \\
0.2 \cdot x + 0.5 &amp; \text{, otherwise}
\end{cases}\$,
\$e^x\$

must take \$(x_1, x_2)\$ either as a tupel/vector/list/... of integers or floats as its only input,
return the answer as an integer, float (or a suitable container, e.g. a vector or list, that contains this answer).


Your answer must include (or link to) all code necessary to check your results -- including the trained weights of your model.

Scoring

The neural network with the smallest number of weights (including bias weights) wins.

Enjoy!
21 13 11 9 weights

This is based on the polarization identity of bilinear forms which in the one dimensional real case reduces to the polynomial identity:

$$ x\cdot y = \frac{(x+y)^2 - (x-y)^2}{4}$$

So y1 just computes [x+y, x-y] using a linear transformation, and y3 is just the absolute value of y1 as a preprocessing step for the next one: Then the "hard" part is computing the squares which I will explain below, and after that just computing a difference and scaling which is again a linear operation.

To compute the squares I use an exponential series \$s\$ which should be accurate for all integers \$\{0,1,2,\ldots,20\}\$ within around \$0.5\$. This series is of the form

$$ \text{approx_square}(x) = \sum_{i=0}^2 w_i \exp(0.0001 \cdot i \cdot x)$$

where I just optimized for the weights W2 (\$=(w_i)_i\$). This whole approximation comprises again just two linear transformations with an exponential activation sandwiched in between. This approach results in a maximal deviation of about 0.02.



function p = net(x)
% 9 weights
one = 1;&#32;
mone =-1;
zero = 0;
fourth = 0.25;
W1 = [1e-4, 2e-4];
W2  = [-199400468.100687;99700353.6313757];
b2 = 99700114.4299316;
leaky_relu = @(a,x)max(a*x,x);&#32;


% Linear
y0 = [one, one; one, mone] * x;

% Linear + ReLU
y1 = mone * y0;
y2 = [leaky_relu(zero, y0), leaky_relu(zero, y1)];

% Linear
y3 = y2 * [one; one];

% Linear + exp
y4 = exp(y3 * W1);&#32;

% Linear + Bias
y5 =  y4 * W2 + b2;

% Linear
y6 = [one, mone]*y5;
p = y6 * fourth;

end


Try it online!