# Formula to detect non uniformity noise

I have this line profile, how is the best way to detect this kind of non uniformity? I must detect a sudden change, as opposed to the last section that decreases slowly..

But the sudden difference might be smaller than in this example so I need a threshold and the overall noise across the line might be a little bit higher difficulting the find..

P.S.: it seems this graph is non linear on the left, don't mind that issue, assume that it is more the less constant all the way until 12335K

• Could you more specifically characterize this "nonuniformity"? It looks to me like there's a section wherein a relatively large number of responses are replaced by the largest power of 2 (namely, 1.024K) that is smaller than them. If this is the pattern, it will be particularly easy to detect, but if it's not, how do you characterize the general pattern you want to detect? – whuber Sep 21 '18 at 20:38
• yes, in this case the signal goes to 1024K, however the main characteristics of this is the sudden change that is higher that the average and it doesn't necessarily happen at the end, might be at the beginning and also the drop might not be this low, the formula must incorporate a custom defined threshold. – miguelmpn Sep 24 '18 at 8:55

## 2 Answers

After adjusting for pulses ...and your pattern suggests many pulses. the next step might be to ignore pulses and to test for variance change ( which is visually obvious) . I have implemented and tested Tsay's procedures into AUTOBOX a piece of software that I have helped to develop. If you wish to try and do it yourself pursue https://pdfs.semanticscholar.org/09c4/ba8dd3cc88289caf18d71e8985bdd11ad21c.pdf

Let $$X_t$$ denote your $$t$$ sample and let $$Y_t := X_t - X_{t-1}$$. If you have a segment of data without these non-uniformities, e.g. $$\{0,1,2,...,T\}$$, consider the empirical distribution of $$Y_t, 1\leq t \leq T$$; You have a list of numbers that represent typical changes in $$X_t$$. This is the empirical distribution for $$Y_t$$ (true is $$Y_t \perp Y_s, s \neq t$$ which is probably false but still useful even without that assumption). Denote this distribution $$D$$. For a new sample $$X_p$$, take the null hypothesis $$H_0: X_p - X_{p-1} \sim D$$ and $$H_1: X_p - X_{p-1} \not \sim D$$. Choose a threshold $$\alpha$$ (e.g. $$\alpha = 10^{-2}$$ or $$\alpha = 10^{-3}$$). Now you reject the null $$H_0$$ if ($$|A|$$ is the size of the set $$A$$):

$$\frac{ |\{1 \leq t \leq T| X_p - X_{p-1} > Y_t \}| }{T} > 1-\alpha.$$ Rejecting the null $$H_0$$ means the observed difference $$X_p - X_{p-1}$$ size is significant, so you say there is a jump (nonuniformity) at the significance level $$\alpha$$. This is a fairly standard hypothesis testing approach to solve your problem, only thing is it uses your data to construct the null, as opposed to (e.g.) normal or student-T distributions. As long as you understand the basics mechanics of hypothesis testing, you can modify above procedure as you please (e.g. use absolute values to detect non-uniformities in both directions).
{"ft_lang_label":"__label__en","ft_lang_prob":0.8469512,"math_prob":0.99427277,"size":1345,"snap":"2019-13-2019-22","text_gpt3_token_len":408,"char_repetition_ratio":0.1043997,"word_repetition_ratio":0.0,"special_character_ratio":0.32713756,"punctuation_ratio":0.13879004,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.99914724,"pos_list":[0,1,2],"im_url_duplicate_count":[null,1,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-05-21T11:27:31Z\",\"WARC-Record-ID\":\"<urn:uuid:83f01387-4330-4e31-937a-aaeb6f3c1eec>\",\"Content-Length\":\"141722\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:a548afd5-8f7e-4438-96ec-6b6e80326565>\",\"WARC-Concurrent-To\":\"<urn:uuid:838157c2-7ec0-4350-a38d-1dbda71bdea3>\",\"WARC-IP-Address\":\"151.101.65.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/368061/formula-to-detect-non-uniformity-noise\",\"WARC-Payload-Digest\":\"sha1:UBX6AEO5ZOGG5K6X7SDP3AFYQM76YS77\",\"WARC-Block-Digest\":\"sha1:7GPR27UMGTZ2TKK56QMA2QWILJREO26O\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-22/CC-MAIN-2019-22_segments_1558232256314.52_warc_CC-MAIN-20190521102417-20190521124417-00493.warc.gz\"}"}