Classifying 2D shapes by the smoothness of their boundaries I'm doing image analysis, and I want to classify smooth objects (has smooth boundaries) from non-smooth objects (has zigzag-like boundaries). Which feature should be fed into ML framework? What are some popular techniques for feature extraction of shapes?
Fourier shape descriptors are quite easy to use and can do well to differentiate smooth objects from jagged ones. Imagine a polar coordinate system with the origin at the centroid of the 2D object. Store a vector of $r$ values as $\theta$ varies in $[0, 2\pi)$ where $r$ is the distance of the boundary from the centroid at each fixed angle $\theta$. Next, take the Fourier transform (FT) of this vector of $r$ values. If the boundary of the 2D object is jagged, the FT will have lot of non-zeros even at high frequencies. On the other hand, imagine doing this with a perfectly smooth circular object---you will end up with a vector of constant $r$ values whose FT is simply a non-zero DC term.

The DFT vector can be used as a feature vector.

See slides 16-20 in this pdf:
http://homepages.cae.wisc.edu/~ece533/project/f06/karrels_ppt.pdf