In this paper, we present a novel method of differentiating known from previously unseen malware families. We utilize transfer learning by learning compact file representations that are used for a new classification task between previously seen malware families and novel ones. The learned file representations are composed of static and dynamic features of malware and are invariant to small modifications that do not change their malicious functionality. Using an extensive dataset that consists of thousands of variants of malicious files, we were able to achieve 97.7 accuracy when classifying between seen and unseen malware families. Our method provides an important focalizing tool for cybersecurity researchers and greatly improves the overall ability to adapt to the fast-moving pace of the current threat landscape.
@cite_0: We present a payload-based anomaly detector, we call PAYL, for intrusion detection. PAYL models the normal application payload of network traffic in a fully automatic, unsupervised and very effecient fashion. We first compute during a training phase a profile byte frequency distribution and their standard deviation of the application payload flowing to a single host and port. We then use Mahalanobis distance during the detection phase to calculate the similarity of new data against the pre-computed profile. The detector compares this measure against a threshold and generates an alert when the distance of the new input exceeds this threshold. We demonstrate the surprising effectiveness of the method on the 1999 DARPA IDS dataset and a live dataset we collected on the Columbia CS department network. In once case nearly 100 accuracy is achieved with 0.1 false positive rate for port 80 traffic.
PAYL @cite_0 uses unsupervised learning, creating profiles of byte frequencies distributions per host and port connected to an application. They then decide whether the behavior is anomalous by testing if the Mahalanobis distance of the operational profile from the precomputed one is higher than a threshold set during the training period. Thus, this method is domain specific and uses only features extracted from the network activity of each application.