k-means|| in PySpark
Based on the paper you set k but want to sample greater than k and apply the log calculation. 

I would recommend using the foreach function. Where you could say foreach() apply further reach than k. 

where the following aglo from the paper could be interpreted as follows. 

Algorithm 2 k-means||(k, ) initialization.

1: C ← sample a point uniformly at random from X
2: ψ ← φX(C)
3: for O(log ψ) times do
4: C 0 ← sample each point x ∈ X independently with probability px =·d2(x,C) φX(C)
5: C ← C ∪ C0
6: end for
7: For x ∈ C, set wx to be the number of points in X closer to x than any other point inC
8: Recluster the weighted points in C into k clusters


instructions:

get points of k = 2
points [(1,1),(1,2),(2,2)]


centroid is placed in middle for this example and 

c1 = [(1,1),(1,2)]


This was achieved with the over sample of Euclid foreach() point that satisfies the over sampling requirement. 

see example here which uses for each:

# Cluster the data into two classes using PowerIterationClustering
model = PowerIterationClustering.train(similarities, 2, 10)

model.assignments().foreach(lambda x: print(str(x.id) + " -&gt; " + str(x.cluster)))


So you will need to write the distance into the lambda.(if you provide code easier to help you).

K distance = k distance on perimeter which is the bottom red line in diagram


 .foreach(lambda x: kdistance[get average] + then check prob(k prime) of k)


share some code and sample data please