K-modes implementation in pyspark I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations.

First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried the second implementation on github. This one worked (as in ran without errors), but I'm still unable to use it in the proper way because of no guide/usage techniques. According to the github page:

n_modes=36
partitions=10
max_iter=10
fraction = 50000 * partitions / (data.count() * 1.0)
data = data.rdd.sample(False,fraction).toDF()

method=IncrementalPartitionedKMetaModes(n_partitions = partitions, n_clusters = n_modes,max_dist_iter = max_iter,local_kmodes_iter = max_iter,
 similarity = "frequency", metamodessimilarity = "hamming")

cluster_metamodes = method.calculate_metamodes(data)


I'm not able to understand how to use this cluster_metamodes, or what it even means.


  In short, k-modes will be performed for each partition in order to
  identify a set of modes (of clusters) for each partition. Next,
  k-modes will be repeated to identify modes of a set of all modes from
  all partitions. These modes of modes are called metamodes here.


Can someone explain the concept and tell how to use this method of clustering on a pyspark dataframe of categorical values?
Modes are just the analog of centroids in k-means. A distributed implementation assigns points to clusters in parallel on subsets of the data, then computes new centroids in parallel, and finally collects those results to perform a (weighted) average of those centroids to get the real centroids. Similarly here, after cluster assignment, modes are computed on subsets of the data in parallel, but then they need to be collected and combined into the real new modes. For whatever reason they're calling the 'real' modes 'metamodes'.