Finding a maximal set of nonintersecting line segments in a unit circle
For intervals we have got a similar problem. 

In the case of interval two intervals $(x_i,y_i)$ and $(x_j,y_j)$ are intersecting if $x_i  \leq y_i \leq x_j \leq y_j $ or $x_j \leq y_j \leq x_i \leq y_i$ is not true.
The problem for intervals is discussed in Maximum non-overlapping intervals in a interval tree. 

In your case however lines are intersecting only if you have $x_i \leq x_j \leq y_i \leq y_j \leq x_i$ in circular order. Here $x_i$ and $y_i$ are the circumference distance from a fixed point say the top point.

As suggested in the solution  for intervals you can either apply dynamic programming or the greedy algorithm.