Neural networks in system identification - What type of activation functions?
Looks like you've done a lot of work on your projects. As @MarcusMüller said, by far the majority of people start with ReLU and go from there. It doesn't have the "vanishing gradient" problem that tanh has for example. All your questions are open ended but common for designing neural networks. There are so many "nobs to turn" to try and make your network be the best it can be. 

You could adjust some parameters, train, test, and compare. But then days or weeks have gone by and you'll still be tuning your parameters. One popular method to choosing "good" parameters is called hyper parameter optimization. This blog post does a good job explaining it: https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/. The idea is that you randomly sample your parameters space (for example, number of neurons, a few different activation functions, different learning rates, number of epochs, etc.) and in search for the a good set of parameters. I hope this helps!