Image segmentation algorithm
You can find what you need here where a comparaison of lot of matting algorithms is done. You can see the rank of those algorithms' performances as well as online demonstration on a natural and complicated image.

From that study, you can guess that A Global Sampling Method for Alpha Matting is the best algorithm to resolve your problem. The algorithm is conceived by A Global Sampling Method for Alpha Matting
Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang and Jian Su.

Hope this helps.

Begueradj.