Why does a horizontal cross section of a squared error surface yield ellipsis?
Hinton refers to a shape like this, where clearly the horizontal cross-sections are ellipses (or circles). 



(In 3D space things can be perpendicular to each other in two ways)
So no matter how steep the gradient at one point is, we need to travel perpendicular (in one way) to the contour lines in order to reach the minimum of the bowl. Think about it :D