Measure how Straight/Smooth the Borders are Rendered in an Image
I'd try a very "tinkery" approach here:


Erode the image, so that the black area is shrunk by a fixed radius of pixels from its border (say, 5px).
Dilate the resulting image by the same amount
measure the amount of difference between original and processed image.


The idea is that something that is a locally convex border doesn't suffer through erosion (it's only shrunk) significantly, and that this erosion can be reverted by dilatation.