How to intentionally fray curved edges in fabric?
You have the right idea. Sewing near the edge but leaving the actual edge raw will achieve a frayed edge. Things to keep in mind:


This will only work on a woven fabric, not a knit. 
Most importantly, not all edges will fray equally on a circle shape. Because of the nature of a woven fabric, the threads are horizontal and vertical. Certain places around the circle (mostly when you hit the bias area), the fraying will change or may not show up at all.
The weave does matter. If the fabric has a very open weave, it may not have a very frayed look, but look like loose threads.