Recognizing hand-written archaeological signs
It is a bit hard to understand what you are trying to do. What are these signs? The one you posted looks like a wheel.  Are there meaningful categories that you can name? If so, then this is a supervised learning (classification) problem, and you should use a classification algorithm such as SVM.

If there are no clear labels, but you want to group together similar-looking signs, then this is an unsupervised learning problem, and you should use a clustering algorithm.

From your description it sounds more like a clustering problem.