Concentric convex hulls
It seems you are trying to perform onion peeling(convex layers) from the inside out, which is not possible. 
A convex hull of given points is a polygon that contains all given points either on itself or inside it. 
To solve the problem you have described you must generate a convex hull using an algorithm of choice. Now that you have the outer layer, for the inner layer, use the same algorithm and only supply the points inside the convex hull. 

The pseudo code would be:
 for given dataset:
   generate a convex hull
   remove the points that make the convex hull from the dataset
   loop until the dataset has no points or more than 2(since a convex hull requires at least 3 points).