org.apache.spark.mllib.clustering

Class StreamingKMeans

• Object
• org.apache.spark.mllib.clustering.StreamingKMeans
• All Implemented Interfaces:
Logging

public class StreamingKMeans
extends Object
implements Logging
:: Experimental ::

StreamingKMeans provides methods for configuring a streaming k-means analysis, training the model on streaming, and using the model to make predictions on streaming data. See KMeansModel for details on algorithm and update rules.

Use a builder pattern to construct a streaming k-means analysis in an application, like:

val model = new StreamingKMeans()
.setDecayFactor(0.5)
.setK(3)
.setRandomCenters(5, 100.0)
.trainOn(DStream)

• Constructor Detail

• StreamingKMeans

public StreamingKMeans(int k,
double decayFactor,
String timeUnit)
• StreamingKMeans

public StreamingKMeans()
• Method Detail

• BATCHES

public static final String BATCHES()
• POINTS

public static final String POINTS()
• k

public int k()
• decayFactor

public double decayFactor()
• timeUnit

public String timeUnit()
• setK

public StreamingKMeans setK(int k)
Set the number of clusters.
• setDecayFactor

public StreamingKMeans setDecayFactor(double a)
Set the decay factor directly (for forgetful algorithms).
• setHalfLife

public StreamingKMeans setHalfLife(double halfLife,
String timeUnit)
Set the half life and time unit ("batches" or "points") for forgetful algorithms.
• setInitialCenters

public StreamingKMeans setInitialCenters(Vector[] centers,
double[] weights)
Specify initial centers directly.
• setRandomCenters

public StreamingKMeans setRandomCenters(int dim,
double weight,
long seed)
Initialize random centers, requiring only the number of dimensions.

Parameters:
dim - Number of dimensions
weight - Weight for each center
seed - Random seed
• trainOn

public void trainOn(DStream<Vector> data)
Update the clustering model by training on batches of data from a DStream. This operation registers a DStream for training the model, checks whether the cluster centers have been initialized, and updates the model using each batch of data from the stream.

Parameters:
data - DStream containing vector data
• predictOn

public DStream<Object> predictOn(DStream<Vector> data)
Use the clustering model to make predictions on batches of data from a DStream.

Parameters:
data - DStream containing vector data
Returns:
DStream containing predictions
• predictOnValues

public <K> DStream<scala.Tuple2<K,Object>> predictOnValues(DStream<scala.Tuple2<K,Vector>> data,
scala.reflect.ClassTag<K> evidence\$1)
Use the model to make predictions on the values of a DStream and carry over its keys.

Parameters:
data - DStream containing (key, feature vector) pairs
Returns:
DStream containing the input keys and the predictions as values
{"ft_lang_label":"__label__en","ft_lang_prob":0.50096357,"math_prob":0.60926527,"size":3113,"snap":"2022-05-2022-21","text_gpt3_token_len":742,"char_repetition_ratio":0.18430364,"word_repetition_ratio":0.07277628,"special_character_ratio":0.20366207,"punctuation_ratio":0.15631692,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9623253,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-01-28T03:38:24Z\",\"WARC-Record-ID\":\"<urn:uuid:fe85c185-0a3d-4882-9a1f-ffbcd0439050>\",\"Content-Length\":\"25581\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:1390bb8f-2ff1-412e-aa72-2d4c6b21260d>\",\"WARC-Concurrent-To\":\"<urn:uuid:5de4fe95-54cf-4d1f-bf5c-88d16bf8fadf>\",\"WARC-IP-Address\":\"151.101.2.132\",\"WARC-Target-URI\":\"https://spark.apache.org/docs/1.3.1/api/java/org/apache/spark/mllib/clustering/StreamingKMeans.html\",\"WARC-Payload-Digest\":\"sha1:JHJCJ4F7IRXBAERALF5X4AS6MTU4ZCHV\",\"WARC-Block-Digest\":\"sha1:PU77NQDOT22KRHPYU4MKRWDKVEA4Z6XB\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-05/CC-MAIN-2022-05_segments_1642320305341.76_warc_CC-MAIN-20220128013529-20220128043529-00434.warc.gz\"}"}