# feature

Visibility
1. Public
2. All

### Type Members

1. #### class ChiSqSelector extends Serializable

Creates a ChiSquared feature selector.

Creates a ChiSquared feature selector. The selector supports different selection methods: `numTopFeatures`, `percentile`, `fpr`, `fdr`, `fwe`.

• `numTopFeatures` chooses a fixed number of top features according to a chi-squared test.
• `percentile` is similar but chooses a fraction of all features instead of a fixed number.
• `fpr` chooses all features whose p-values are below a threshold, thus controlling the false positive rate of selection.
• `fdr` uses the [Benjamini-Hochberg procedure] (https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure) to choose all features whose false discovery rate is below a threshold.
• `fwe` chooses all features whose p-values are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the family-wise error rate of selection. By default, the selection method is `numTopFeatures`, with the default number of top features set to 50.
Annotations
@Since( "1.3.0" )
2. #### class ChiSqSelectorModel extends VectorTransformer with Saveable

Chi Squared selector model.

Chi Squared selector model.

Annotations
@Since( "1.3.0" )
3. #### class ElementwiseProduct extends VectorTransformer

Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.

Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector. In other words, it scales each column of the dataset by a scalar multiplier.

Annotations
@Since( "1.4.0" )
4. #### class HashingTF extends Serializable

Maps a sequence of terms to their term frequencies using the hashing trick.

Maps a sequence of terms to their term frequencies using the hashing trick.

Annotations
@Since( "1.1.0" )
5. #### class IDF extends AnyRef

Inverse document frequency (IDF).

Inverse document frequency (IDF). The standard formulation is used: `idf = log((m + 1) / (d(t) + 1))`, where `m` is the total number of documents and `d(t)` is the number of documents that contain term `t`.

This implementation supports filtering out terms which do not appear in a minimum number of documents (controlled by the variable `minDocFreq`). For terms that are not in at least `minDocFreq` documents, the IDF is found as 0, resulting in TF-IDFs of 0.

Annotations
@Since( "1.1.0" )
6. #### class IDFModel extends Serializable

Represents an IDF model that can transform term frequency vectors.

Represents an IDF model that can transform term frequency vectors.

Annotations
@Since( "1.1.0" )
7. #### class Normalizer extends VectorTransformer

Normalizes samples individually to unit Lp norm

Normalizes samples individually to unit Lp norm

For any 1 <= p < Double.PositiveInfinity, normalizes samples using sum(abs(vector).p)(1/p) as norm.

For p = Double.PositiveInfinity, max(abs(vector)) will be used as norm for normalization.

Annotations
@Since( "1.1.0" )
8. #### class PCA extends AnyRef

A feature transformer that projects vectors to a low-dimensional space using PCA.

A feature transformer that projects vectors to a low-dimensional space using PCA.

Annotations
@Since( "1.4.0" )
9. #### class PCAModel extends VectorTransformer

Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

Annotations
@Since( "1.4.0" )
10. #### class StandardScaler extends Logging

Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

The "unit std" is computed using the corrected sample standard deviation (https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation), which is computed as the square root of the unbiased sample variance.

Annotations
@Since( "1.1.0" )
11. #### class StandardScalerModel extends VectorTransformer

Represents a StandardScaler model that can transform vectors.

Represents a StandardScaler model that can transform vectors.

Annotations
@Since( "1.1.0" )
12. #### trait VectorTransformer extends Serializable

:: DeveloperApi :: Trait for transformation of a vector

:: DeveloperApi :: Trait for transformation of a vector

Annotations
@Since( "1.1.0" ) @DeveloperApi()
13. #### class Word2Vec extends Serializable with Logging

Word2Vec creates vector representation of words in a text corpus.

Word2Vec creates vector representation of words in a text corpus. The algorithm first constructs a vocabulary from the corpus and then learns vector representation of words in the vocabulary. The vector representation can be used as features in natural language processing and machine learning algorithms.

We used skip-gram model in our implementation and hierarchical softmax method to train the model. The variable names in the implementation matches the original C implementation.

For original C implementation, see https://code.google.com/p/word2vec/ For research papers, see Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality.

Annotations
@Since( "1.1.0" )
14. #### class Word2VecModel extends Serializable with Saveable

Word2Vec model

Word2Vec model

Annotations
@Since( "1.1.0" )

### Value Members

3. #### object Word2VecModel extends Loader[Word2VecModel] with Serializable

Annotations
@Since( "1.4.0" )
{"ft_lang_label":"__label__en","ft_lang_prob":0.8349303,"math_prob":0.8733547,"size":4240,"snap":"2019-51-2020-05","text_gpt3_token_len":881,"char_repetition_ratio":0.123937674,"word_repetition_ratio":0.2990196,"special_character_ratio":0.19127358,"punctuation_ratio":0.12113055,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9831421,"pos_list":[0,1,2],"im_url_duplicate_count":[null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-12-14T07:18:42Z\",\"WARC-Record-ID\":\"<urn:uuid:89217167-904f-4c5f-ac90-7c36d4620045>\",\"Content-Length\":\"30245\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:7aaf5ecf-0571-46c4-9ea9-a23a56b60def>\",\"WARC-Concurrent-To\":\"<urn:uuid:8a171855-5382-4597-990b-d0d85a5f246c>\",\"WARC-IP-Address\":\"40.79.78.1\",\"WARC-Target-URI\":\"https://spark.apache.org/docs/latest/api/scala/org/apache/spark/mllib/feature/package.html\",\"WARC-Payload-Digest\":\"sha1:UOSEHDSRMIZ75V7B77MORGVDXOTO4UPI\",\"WARC-Block-Digest\":\"sha1:3HUSJCRGX2BDCF5VLFH6RSIOH6T53CHN\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-51/CC-MAIN-2019-51_segments_1575540585566.60_warc_CC-MAIN-20191214070158-20191214094158-00513.warc.gz\"}"}