Analyzing Data with Statistics # Statistical Outlier Detection

Data scientists frequently want to find anomalies in numerical data. To that end, HCP has some simple statistical outlier detectors.

Table 1. Statistical Outlier Detection
Function Description Input Returns
OUTLIER_MAD_STATE_MERGE Update the statistical state required to compute the Median Absolute Deviation.
• state - A list of Median Absolute Deviation States to merge. Generally these are states across time.

• currentState? - The current state (optional)

The Median Absolute Deviation state
OUTLIER_MAD_ADD Add a piece of data to the state
• state -The MAD state

• value - The numeric value to add

The MAD state
OUTLIER_MAD_SCORE Get the modified z-score normalized by the MAD: scale * | x_i - median(X) | / MAD.
• state - The MAD state

• value - The numeric value to score

• scale? -Optionally the scale to use when computing the modified z-score. Default is `0.6745`.

The modified z-score
{"ft_lang_label":"__label__en","ft_lang_prob":0.61483574,"math_prob":0.76017654,"size":541,"snap":"2019-13-2019-22","text_gpt3_token_len":127,"char_repetition_ratio":0.12476723,"word_repetition_ratio":0.0,"special_character_ratio":0.2181146,"punctuation_ratio":0.074074075,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.988839,"pos_list":[0,1,2],"im_url_duplicate_count":[null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-05-22T13:31:57Z\",\"WARC-Record-ID\":\"<urn:uuid:537ddac2-30b7-45af-9f22-be04ad719010>\",\"Content-Length\":\"16692\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:6bd50b35-7812-4ea7-a670-2055d4090e84>\",\"WARC-Concurrent-To\":\"<urn:uuid:d1356dfe-277f-449e-8ca0-b676ee643188>\",\"WARC-IP-Address\":\"99.84.181.94\",\"WARC-Target-URI\":\"https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.8.0/analyzing-data-with-stats/content/statistical_outlier_detection.html\",\"WARC-Payload-Digest\":\"sha1:XU2GJDAW4LWBTXBDOAKB3C2MD6G7DLZ7\",\"WARC-Block-Digest\":\"sha1:LRY5I263CETJPKLNJQ6MRXGZ6B7GNRGZ\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-22/CC-MAIN-2019-22_segments_1558232256812.65_warc_CC-MAIN-20190522123236-20190522145236-00468.warc.gz\"}"}