What is geometric deep learning?
To complete the first answer that is rather graph oriented, I will write a little about deep learning on manifolds, which is quite general in terms of GDL thanks to the nature of manifolds. 

Note that the description of GDL through the explanation of what are DL on graphs and manifolds, in opposition to DL on euclidean domains, comes from the 2017 paper Geometric deep learning: going beyond Euclidean data (this paper is excellent at clarifying both the intuition and the mathematics of what I'm writing).

1. In case you don't know what a manifold is

As the previously cited paper puts it:


  Roughly, a manifold is a space that is locally Euclidean. One of the
  simplest examples is a spherical surface modeling our planet: around a
  point, it seems to be planar, which has led generations of people to
  believe in the flatness of the Earth. Formally speaking, a
  (differentiable) d-dimensional manifold X is a topological space where
  each point x has a neighborhood that is topologically equivalent
  (homeomorphic) to a d-dimensional Euclidean space, called the tangent
  space.


Good other not-so-technical explanation on stats.stackexchange

Other Wikipedia examples to develop not too abstract understanding

Very shortly put, it's an interesting mathematical set on which to work (different kinds exist, see papers at the end of this answer for DL related manifolds uses). By work, you can typically understand that you constrain the neural net parameters to the manifold you chose (e.g. training with parameters constrained on a hypersphere, among the geomstats paper examples).

Your data can also be represented thanks to a practical manifold. For example, you can choose to work on images and videos by representing the samples using Symmetric Positive Definite (SPD) matrices (see this paper), the space of SPD matrices being a manifold itself.

2. Why bother learning on manifolds ?

Defining a clearer/better adapted set (understand that it's a sort of constraint!) on which to learn parameters and features can make it simpler to formally understand what your model is doing, and can lead to better results. I see it as a part of the effort of deep learning formalization. One could say you're looking for the best information geometry for your task, the one that best captures the desirable data distribution properties.To develop this intuition consider the solar system analogy for manifold learning of this Kaggle kernel:


  Perhaps a good analogy here is that of a solar system: the surface of our planets are the manifolds we're interested in, one for each digit. Now say you're on the surface of the earth which is a 2-manifold and you start moving in a random direction (let's assume gravity doesn't exist and you can go through solid objects). If you don't understand the structure of earth you'll quickly find yourself in space or inside the earth. But if you instead move within the local earth (say spherical) coordinates you will stay on the surface and get to see all the cool stuff.


This analogy reminds us of the spherical surface planet model from Bronstein's paper already quoted above. This paper also describes a typical case for which manifolds are interesting: where graphs (the other example of GDL/DL on non euclidean data) are better at handling data from social or sensor networks, manifolds are good at modeling 3D objects endowed with properties like color texture in computer vision.

3. Regarding deep neural networks on manifolds

I would advise reading the geomstats associated paper, which does a great job at showing what it is and how it can be used, along with example codes (e.g. MNIST on hyperspheres manifold example code here). This library implements manifolds and associated metrics on Keras. The choice of metrics is essential to understand the point of working on manifolds: it's because you need to work on an adapted mathematical set (ie with the right properties) with an adapted distance definition (so that the measure actually means something when considering the problem you're trying to solve) that you switch to working on manifolds.

If you want to dive in the details and examples of deep learning on manifolds here are some papers:


A Riemannian Network for SPD Matrix Learning: new backpropagation to learn SPD matrices on Riemannian manifolds
Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds: using SPD matrices to aggregate convolutional features


4. Why Riemannian manifolds ?

TL;DR: you need a metric to do machine learning (otherwise, how could you evaluate how much you actually learned !)

Still based on Bronstein's paper:


  On each tangent space, we define an inner product [...]. This inner
  product is called a Riemannian metric in differential geometry and
  allows performing local measurements of angles, distances, and
  volumes. A manifold equipped with a metric is called a Riemannian
  manifold.


5. What's the relation between a Riemannian manifold and a Euclidean space ?

Still based on Bronstein's paper:


  a Riemannian manifold can be realized as a subset of a Euclidean space
  (in which case it is said to be embedded in that space) by using the
  structure of the Euclidean space to induce a Riemannian metric.


I leave the details to the paper, otherwise this answer will never end.

6. Answers to questions in comments

Will only answer once I think I've found a relatively well-argued answer, so won't answer everything at once.


Isn't manifold learning just a way of dimensionality reduction? 


I don't think so, it isn't just that. I haven't seen any dimensional reduction constraint (yet ?) in the papers I've read (cf. geomstats again). 

In the hypersphere/MNIST geomstats code example, you can see the chosen manifold dimension hypersphere_dimension = 17. Since we're working with MNIST data I guess this would mean a dimension reduction in this particular case. I admit I would need to check exactly what that dimension implies on the neural net architecture, I haven't discussed my understanding of this yet.

Disclaimer

I'm still developing a more rigorous mathematical understanding of manifolds, and shall update this post to make additional necessary clarifications: exactly what can be considered as a manifold in a traditional deep learning context, why do we use the word manifold when speaking about the hidden state of auto-encoders (see the previously cited Kaggle kernel that quotes Goodfellow's book on this). All of this if the perfectly clear answer doesn't show up here before !