[Cross-posted from stats stackexchange](https://stats.stackexchange.com/questions/608326/optimal-variable-time-sampling-of-a-real-time-data-stream)

Say I have a logging utility in my application that I use for recording timestamped diagnostic log messages. I want to add tracking of some performance metrics to this log. Moreover, I want to do this intelligently, so as not to explode the size of the log files.

A convenient way to do a form of lossy compression would be to only record values when it appears that something "interesting" is happening - as judged, for example, by looking at the output of a high-pass filter applied to the value.

It's clear that the success of such an implementation depends critically on the design of the filter. How can we design an optimal filter (in a rate-distortion sense), given some knowledge (e.g. moment structure) of the stream we are trying to compress?