Record Linkage problem I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip.

We create pairs between data sets  and calculate cosine similarity and then pass cosine values for all features pairs to Gaussian Mixture model.We started with 2 cluster, with expectation of one match cluster and one no match cluster.But ML does not build one match cluster and  matches are in both the clusters.

Before passing to ML, i use Standard scaler and minmax scaler , but still don't get a clear nomatch and match cluster.If we increase the cluster same thing happens.

Match could be High cosine similarity  in Name,Address,State,City &amp; zip or Name ,address ,zip or any other combinations.We are dealing with huge volume , so we are using Spark ML.

How can we achieve optimal clustering?
Clustering is the wrong tool for this. And so is cosine similarity, bag of words model etc.

Obviously, standard scaling on house numbers and zip codes is a svery bad idea, don't you think?

There are special techniques developed for this kind of data with a reason. For example, it's unlikely that the customer entered house number 100 but lives in 5. That single difference is a very clear indication that it probably is not the same customer. On the other hand, "Park Rd." And "Parkrd." almost certainly is the same, but in cosine similarity that is three tokens difference, pretty big.

Clustering is an explorative technique, not one to achieve automatic matching. Choose other tools instead.