Machine Learning on Curved Background
Essentially, each non-linear layer in a neural network is a map from $\mathbb{R}^n$ input to a $\mathbb{R}^m$ output. There is no requirement for these dimensions to be separate and uncorrelated, nor for the map to be reversible.

The best performance is usually found if each layer is roughly scaled to approximate a normal distribution with mean 0 and standard deviation 1. However, that is an independent issue to any kind of space-like mapping.

If you know a suitable mapping that relates to your problem and would simplify its expression, then you can apply a spacial transform between co-ordinate systems (typically by mapping the input features) and use it within the neural network. You could map co-ordinates to any curved space if you think it might help. Choosing between polar or cartesian co-ordinates for features is one choice that may crop up for example.

However, even without making this helpful mapping, you can analyse a neural network layer-by-layer, and see that it can effectively learn these kind of spacial mappings from the training data anyway. This is an intuition that has been called the "manifold hypothesis" - the link has some images showing how a neural network might learn to separate classes depending on how they are arranged in feature space, and this article from a DARPA researcher has a nice series of images showing successive transformations of a 2D spiral that allow the last layer to perform linear separation.