How to mathematically define the architecture of neural network model? And the function space associated with it?
You can use directed graphs. Each neuron is a node in the graph. Each neuron has list of its neighbours and activation function. You can add remove nodes to this graph. 



If you just want to search for simple architectures you can use two lists. One for number of neurons in layer and one for activation function of neurons in that layer. 

So this is a neural network with 3 layers. First layer has 5 neurons and echo of them have Relu activation function.  


  [5,3,2] ['Relu', 'sigmoid', 'Relu'] 


Hope that helps.