Industrial Internet of Things (IIoT) is becoming an attack target of advanced persistent threat (APT). Currently, IIoT logs have not been effectively used for anomaly detection. In this paper, we use blockchain to prevent logs from being tampered with and propose a pvalue-guided anomaly detection approach. This approach uses statistical pvalues to combine multiple heterogeneous log parser algorithms. The weighted edit distance is selected as a score function to calculate the nonconformity score between a log and a predefined event. The pvalue is calculated based on the non-conformity scores which indicate how well a log matches an event. This approach is tested on a large number of real-world HDFS logs and IIoT logs. The experiment results show that abnormal events could be effectively recognized by our pvalue-guided approach.
@cite_31: Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability e, together with a method that makes a prediction ŷ of a label y, it produces a set of labels, typically containing ŷ, that also contains y with probability 1  e. Conformal prediction can be applied to any method for producing ŷ: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right 1  e of the time, even though they are based on an accumulating data set rather than on independent data sets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples. A more comprehensive treatment of the topic is provided in Algorithmic Learning in a Random World, by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).
@cite_21: A standard assumption in machine learning is the exchangeability of data, which is equivalent to assuming that the examples are generated from the same probability distribution independently. This paper is devoted to testing the assumption of exchangeability on-line: the examples arrive one by one, and after receiving each example we would like to have a valid measure of the degree to which the assumption of exchangeability has been falsified. Such measures are provided by exchangeability martingales. We extend known techniques for constructing exchangeability martingales and show that our new method is competitive with the martingales introduced before. Finally we investigate the performance of our testing method on two benchmark datasets, USPS and Statlog Satellite data; for the former, the known techniques give satisfactory results, but for the latter our new more flexible method becomes necessary.
@cite_34: Building machine learning models of malware behavior is widely accepted as a panacea towards effective malware classification. A crucial requirement for building sustainable learning models, though, is to train on a wide variety of malware samples. Unfortunately, malware evolves rapidly and it thus becomes hard—if not impossible—to generalize learning models to reflect future, previously-unseen behaviors. Consequently, most malware classifiers become unsustainable in the long run, becoming rapidly antiquated as malware continues to evolve. In this work, we propose Transcend, a framework to identify aging classification models in vivo during deployment, much before the machine learning model’s performance starts to degrade. This is a significant departure from conventional approaches that retrain aging models retrospectively when poor performance is observed. Our approach uses a statistical comparison of samples seen during deployment with those used to train the model, thereby building metrics for prediction quality. We show how Transcend can be used to identify concept drift based on two separate case studies on Android andWindows malware, raising a red flag before the model starts making consistently poor decisions due to out-of-date training.
. As described in @cite_31 , the authors propose the theory of conformal prediction that uses past experience to determine precise level of confidence in new prediction. Conformal prediction uses a non-conformal measure to calculate the non-conformity score. We can obtain a pvalue that can be used to make decisions or evaluation, whose outputs is a prediction set with fixed confidence level. @cite_21 , the authors use conformal prediction to detect deviation of data sequence under the assumption that data are independent and identically distributed. Conformal evaluation uses the theory of conformal prediction to measure the non-conformity of a test object to a class compared to all other objects. By using conformal evaluation, the authors detect concept drift and identify aging classification in malware classification models @cite_34 . Conformal evaluation provides better understanding for model quality.