I study Neural Networks and I pretty much understand the logic of the structure with layers, activation functions and connections. But a fundamental question is not clear: how does one put together the actual structure from the several possible combinations? It seems me that most people just combine these quite randomly without any proper reason (or with brute force grid search in better cases).I've read several tutorials but nobody explained the reason of choosing sigmoind activation function over tanh for instance, not to mention which one to pick in case of different activation functions in different layers. I stress that I understand the working of these functions by themselves, but the logic of the order of them is quite a puzzle.
Guiding principle of Neural Network structure building
neural network