Algorithms for detection of fingertips or nails
I would consider using neural network or SVM to fit the model. The difficulty with this approach is that you must collect a lot of data - both positive and negative examples. But you can generate a lot of artificial data(by scaling or rotating images you have already collected). After you collect data you can use "moving window" of few sizes to detect nails on images not present in training set.
I don't know how much time you want to spend doing the project. But implementing efficient learning algorithm yourself is difficult. However there is already library for SVM which I would use for this purpose.