Hardware to run a fixed neural network
To give a rough idea my network has around one million neurons and ten billion connections.

That means ten billion weights. From a NN perspective, this very much sounds like you want to apply L1 regularization to reduce the number of connections; unless you really have good reasons why this needs to be this large, I'm almost certain that you didn't sufficiently optimize your network.
Anyway, in its current form, you'd need something to store 10 billion coefficients. The only realistic way of doing that is using SDRAM.
Then, you need an SDRAM controller.
And then, you need something to fetch the data from that controller and execute your inference.
In other word, you're describing a large GPU. So, you need a large GPU. A GPU these days essentially is an ASIC for NN applications. The whole graphics aspects of it can be safely ignored.