How to trim the jagged edges off of this solid
Try applying a solidify modifier to your cylinder prior to adding your boolean modifier to the terrain object. 

For me, I have better luck with the boolean modifier when there is thickness to the extracting mesh.