Terminology: "flat geometry" in the context of clustering
Are flat and non-flat geometry a legit terminology in machine learning
  and statistics?


These are terminologies from Mathematics, they are valid in any field.


  What is the mathematical definition?


(Non) flat manifold

In mathematics, a (Riemannian) manifold is said to be flat if its curvature is everywhere zero; otherwise non-flat. This is very different than the definition of flat object in geometry. According to that definition, only points, lines, and hyper-planes are flat (not for example hyperspheres or segments).

For example, set $\left\{(t,t):t\in(-1,1)\right\}$ is a 1D flat manifold in ${\Bbb R}^2$, set $\left\{(t,t^2):t\in(-1,1)\right\}$ is a 1D non-flat (positively curved) manifold in ${\Bbb R}^2$, and a hypersphere is an $n$D non-flat (positively curved) manifold in ${\Bbb R}^{n+1}$.

Accordingly, a cluster with a (non) flat shape corresponds to a (non) flat manifold.

Here are some examples from the document.



Points are concentrated around (A) two 1D non-flat manifolds (circles) which are non-convex, (B) two 1D non-flat manifolds (arcs) which are non-convex, (C) three 1D flat manifolds (segments) which are convex, (D) three 0D flat manifolds (centers as points) which are convex.

(Non) flat geometry vs a Flat

By "flat geometry" the document means "Euclidean geometry" not "a flat object defined in geometry". If we measure distances (consequently lengths, areas, volumes, etc.) via Euclidean distance we are obeying the Euclidean geometry, otherwise, we are obeying a non-Euclidean geometry. For example, if we measure a distance between two points by following a non-flat manifold (a geodesic), we are obeying a non-Euclidean geometry. As an illustration,



In (A), the red line measures a distance obeying a flat geometry, the blue line measures the distance obeying a non-flat geometry (by moving along the non-flat manifold). If an appropriate map of manifold to a lower dimension is possible (B), obeying a flat geometry would be equivalent to obeying the non-flat geometry before mapping.