How can I smooth out the edges of this mesh?
the problem is that some of the faces are triangulated, resulting in the bevel modifier having a hard time in finding a desirable solution. looking at the model you have there, you could automatically clean up the geometry using a decimate modifier BEFORE the bevel, set its mode to planar, and then, as lemon suggested, use the angle method in the bevel. See those two screenshots here: