On Lyapunov exponents and adversarial perturbations
∗

Vinay Uday Prabhu

UnifyID Inc
San Francisco, 94107

arXiv:1802.06927v1 [cs.CV] 20 Feb 2018

vinay@unify.id

Nishant Desai

John Whaley

UnifyID Inc
San Francisco, 94107

UnifyID Inc
San Francisco, 94107

nishant@unify.id

ABSTRACT
In this paper, we would like to disseminate a serendipitous
discovery involving Lyapunov exponents of a 1-D time series
and their use in serving as a filtering defense tool against a
specific kind of deep adversarial perturbation. To this end,
we use the state-of-the-art CleverHans library to generate
adversarial perturbations against a standard Convolutional
Neural Network (CNN) architecture trained on the MNIST
as well as the Fashion-MNIST datasets. We empirically
demonstrate how the Lyapunov exponents computed on the
flattened 1-D vector representations of the images served as
highly discriminative features that could be to pre-classify
images as adversarial or legitimate before feeding the image
into the CNN for classification. We also explore the issue of
possible false-alarms when the input images are noisy in a
non-adversarial sense.

Keywords
Deep learning, adversarial attacks, transfer learning

1.

BACKGROUND ON DEFENSES AGAINST
ADVERSARIAL ATTACKS

In the recent past, a plethora of defenses against adversarial attacks have been proposed. These include SafetyNet
[13], adversarial training [21], label smoothing [22], defensive distillation [19] and feature-squeezing [24, 25] to name
a few. There is also an ongoing Kaggle contest[2] underway
for exploring novel defenses against adversarial attacks.
As evinced by the recent spurt in the papers written on
this topic, most defenses proposed are quelled by a novel
attack that exploits some weakness in the defense. In [10],
the authors queried if one could concoct a strong defense by
combining multiple defenses and showed that an ensemble of
weak defenses was not sufficient in providing strong defense
against adversarial examples that they were able to craft.
∗Primary author

john@unify.id

In the accompanying blog1 associated with [17], the authors
Goodfellow and Papernot posit that Adversarial examples
are hard to defend against because it is hard to construct a
theoretical model of the adversarial example crafting process
and contend with the idea if attacking machine learning easier than defending it?
With this background, we shall now look more closely at a
specific type of defense and motivate the relevance of our
method within this framework.

1.1

The pre-detector based defenses

One prominent approach that emerges in the literature
of adversarial defenses is that of crafting pre-detection and
filtering systems that flag inputs that might be potentially
adversarial. In [9], the authors posit that adversarial examples are not drawn from the same distribution as the legitimate samples and can thus be detected using statistical
tests. In [15, 7], the authors train a separate binary classifier
to first classify any input image as legitimate or adversarial
and then perform inference on the passed images. In approaches such as [6], the authors assume that DNNs classify
accurately only near the small manifold of training data and
that the synthetic adversarial samples do not lie on the data
manifold. They applying dropout at test time to ascertain
the confidence of adversariality of the input image.
In this paper, we would like to disseminate a model-agnostic
approach towards adversarial defense that is dependent purely
on the quasi-time-series statistics of the input images that
was discovered in a rather serendipitous fashion. The goal is
to not present the method we propose as a fool-proof adversarial filter, but to instead draw the attention of the DNN
and CV communities towards this chance discovery that we
feel is worthy of further inquiry.
In order to facilitate reproducibility of results and effective
criticism, we have duly open-sourced the implementation
as a well annotated jupyter-notebook shared at the following location: https://github.com/vinayprabhu/Lyapunov
defense

2.
2.1

PROCEDURE FOLLOWED
A quick introduction to Lyapunov exponents

For a given a scalar time series {xt ; t = 1, ..., n} whose
time evolution is assumed to be modeled by a differentiable
DEEP LEARNING SECURITY WORKSHOP ’17 SINGAPORE
ACM ISBN 978-1-4503-2138-9.
DOI: 10.1145/1235

1

http://www.cleverhans.io/security/privacy/ml/2017/02/
15/why-attacking-machine-learning-is-easier-than-defending-it.
html

dynamical system (in a phase-space of possibly infinite dimensions), we can define Lyapunov exponents corresponding
to the large-time behavior of the system. These characteristic exponents numerically quantify the sensitivity to the
initial conditions and emanate from the Ergodic theory of
differentiable dynamical systems, introduced by Eckmann
and Ruelle in [5, 4]. Simply put, if the initial state of a
time series is slightly perturbed, the characteristic exponent
(or Lyapunov exponents) represent the exponential rate at
which the perturbation increases (or decreases) with time.
For a detailed understanding of these characteristic exponents, we would like to refer the user to [5, 4].
Given a finite-length finite-precision time series sequence,
the 3 stage Embed-Tangent maps-QR decomposition based
recipe introduced in [4] can be used to numerically compute
the Lyapunov exponents. This numerical recipe is implemented in many time-series analysis packages such as [1].
The background of the serendipitous discovery was that we
were investigating usage of metrics from time-series analysis literature to identify adversarial attacks on 1-D axis-wise
mobile-phone motion sensor data ;example, Accelerometer
and gyroscope (see [20]) and we realized that this procedure
could be easily extended to vectorized images (or flattened
images viewed as discrete time series indexed by the pixel
location. That is, given a normalized image, X ∈ [0, 1]n×n ,
we would have the flattened time-series representation x ∈
2
[0, 1]n as simply,
xin+j = Xi,j ; i, j = 0, ...n − 1;

3.

(1)

EXPERIMENTS

In this paper, we used the MNIST [11] dataset. In the
appendix, we showcase similar results obtained with the
Fashion-MNIST dataset [23] as well. The rest of this section
details the different stages involved in our experimentation.

3.1

Generating targeted adversarial perturbations

We targeted 10 random samples from the MNIST dataset
(1 belong to each class/digit) using the Carlini-Wagner-l2
attack ([3]) in the targeted mode implemented in the CleverHans library ([17]) with the following parameterization.

min_nb=min(2 * matrix_dim, matrix_dim + 4),
min_tsep=0, tau=1}
An example images along with its flattened quasi-time-series
representation and the computed Lyapunov exponents is
shown in fig 4(a). Notice the change of sign of the lyapunov
exponents between the true image and the adversarial images. In time-series analysis, the existence of at least one
positive Lyapunov exponent is interpreted as a strong indicator for chaos. As we introduce more adversarial noise, the
prevalence of positive Lyapunov exponents becomes more
prevalent. Fig 5, contains an example of an image (digit-5)
and all its 9 associated adversarially perturbed counterparts
(each targeting a different class) and their lyapunov exponents.

3.3

Unsupervised clustering of input images
based on Lyapunov exponents

In fig 4(b), where we have showcased the scatter-plot of
the first 2 lyapunov exponents, we see clear clustering between the legitimate examples and the adversarial examples.
We see that the only adversarial examples that are inside
the cluster of legitimate examples are those where the target class is the same as the true class. (These are images
whose every element is just a fixed ε offset from the true
image)

3.4

Supervised classification

Detection can also be performing from the viewpoint of
outlier detection where we train a one-class classifier (ex:
Isolation-Forest [12]) with just the inlier (legitimate) images’ lyapunov exponents used during training. Fig 3(a)
shows the heatmap of the features in the train and test subsets (200: Training samples| 300 Test samples of which 200
are legitimate and 100 are adversarial). We received 100%
attacker rejection rate and a 83.5% true acceptance rate (or
a 16.5% False-alarm rate) with no feature engineering or
hyper-parameter tuning. Fig 3(b) showcases the scatterplot of the train and test samples along with the decision
boundaries found by the isolation forest algorithm.

3.5

Effect of non-adversarial noise on the detection performance

cw_params = {’binary_search_steps’: 1,
In order to ascertain the effect of non-adversarial noise
’y_target’: adv_ys,
on
the detection performance, we chose the following noise
’max_iterations’: attack_iterations,
2
models
:
’learning_rate’: 0.1,
’batch_size’: source_samples * nb_classes,
[’gaussian’,’pepper’,’poisson’,’salt’,
’initial_const’: 10}
’Salt and Pepper’,’Speckle’,’local-variance Gaussian’]
The resultant image grid containing the images along with
the adversarially perturbed counterparts are as shown in fig
1(a). Fig 1(b) showcases the norm-distance between the
images and their adversarial examples.

3.2

Computing Lyapunov exponents of the flattened images

The lyapunov exponents of the flattened images showcased in fig 1(a) were numerically computed with the lyap_e()
method implemented in [1] with the following parameterization:
PARAM_MNSIT={emb_dim=10, matrix_dim=4,

(Here ’local-variance Gaussian’ refers to Gaussian-distributed
additive noise with specified local variance at each pixel of
the image).
The effect of these noise-models on the image and its lyapunov exponents are shown in fig 2. As seen, the ’localvariance Gaussian’ noise has the most discernible effect on
the lyapunov exponents and hence is chosen as the model of
choice to perturb the legitimate images with.

3.6
2

Augmenting the dataset with non-adversarial
perturbed images

http://scikit-image.org/docs/dev/api/skimage.util.html

Matrix norm dist b/w adv and input image

4

3

2

1

0
0

20

40

60

80

100

Image index

(a) Image grid with legitimate MNIST images and their adver- (b) Norm distances for the images in the grid (between legitsarial counterparts
imate F-MNIST images and their adversarial counterparts)
Figure 1: In the left sub-figure above, the [i, j]th grid image is an image belong to the j th class adversarially
perturbed to be classified as belonging to the ith class

Original image
[-0.15204129 -1.18814337 -1.70479167 -1.75485373]

gaussian:[ 0.28242487 0.07765591 -0.06113432 -0.38037845]

pepper:[ 0.02092013 -0.96793717 -1.33508384 -1.61110401]

poisson:[-0.14193536 -1.17601728 -0.98012441 -2.26296401]

salt:[ 0.09110281 -0.4430038 -1.15924442 -1.60302079]

s&p:[ 0.05007739 -0.78992772 -1.35534716 -1.78148997]

speckle:[-0.27073824 -1.11521339 -1.52682781 -2.10146165]

localvar:[ 0.28221309 0.07588112 -0.07270029 -0.25669238]

Figure 2: The images of an example image sample (5) along with its noisy counterparts and their Lyapunov
coefficients
We added the training datasets with 100 noisy images
(with Gaussian perturbations) and re-train the isoforest classifier. The results are as shown in fig 3.5. We see that the
false alarm rate drops to 15% while the attacker rejection
rate is almost unchanged (only 1 attacker image is accepted
in).
We further observe that when the magnitude of random

noise is selected from the distribution of norm distances
among adversarial perturbations, the classifier retains its
ability to distinguish between adversarially perturbed and
randomly perturbed images.

3.7

Testing the defense with other attacks

We consider whether the Lyapunov method can be used

0

0

Lyapunov exponents heat-map
(a) (left)Training- All legitimate (b) (right)Testing - 200 legitimate| 100 adversarial
1

2

3

0

0

1

2

3

3

50

2

50

100

100

1

150

0

200

−1

150
−2

250

−3

(a) Feature matrix heat-map

(b) Iso-Forest scatter-plot

Confusion matrix: [[ 91 9]
[ 45 255]]
4
2

rates for these two classifiers on data generated by each of
the attacks are presented in Table 3.7.

Training obser ations
legit test samples
Ad ersarial samples
Legit test with Gaussian noise

Attack
Carlini Wagner
FGSM
JSMA

0

Madry et al.

−2
DeepFool

−4
−4

−2

0

2

4

Figure 3: Iso-Forest scatter-plot with Gaussian noise
perturbed examples

to detect adversarial perturbations from attacks other than
the Carlini-Wagner-l2 attack. We consider the Fast Gradient Sign Method [8], the Jacobian Saliency Map Attack [18],
DeepFool [16], and the attack presented by Madry et al [14].
We consider both the targeted and untargeted versions of
each attack, where applicable3 . We use the default parameters of the CleverHans library wherever possible. Where
CleverHans does not provide a default value, we use the values referenced in the original paper describing the attack.
We find that when trained on only the first two Lyapunov
exponents, the isolation forest does poorly on most of the
attacks. However, accuracy improves significantly when we
train on four-dimensional data. In both cases, we train using
natural MNIST images as the inlier set. The true negative
3

DeepFool does not have a targeted variant.

Targeted/Untargeted
Targeted
Untargeted
Targeted
Untargeted
Targeted
Untargeted
Targeted
Untargeted
Targeted
Untargeted

2D
0.90
1.0
0.0
0.0
0.03
0.03
0.0
0.0
NA
1.0

4D
0.91
1.0
0.62
0.8
0.04
0.04
1.0
1.0
NA
1.0

Table 1: True positive rates for several targeted and
untargeted attacks.

3.8

Testing the effect of a new attack

In previous sections, we used a 1-class Isolation Forest
classifier to learn an inlier set of unmodified images. In this
experiment, we test whether a classifier trained on both positive and negative data generated using known adversarial
attacks can outperform the 1-class classifier.
We train a logistic regression model on unmodified MNIST
images and data generated using all but one of the untargeted attacks from the previous section. We then evaluate
the model on a validation set consisting of natural images
and images modified using the left-out attack. We find that
the logistic model is able to achieve near-perfect performance
on four out of five attacks. The model only fails to perform
on data generated using JSMA, achieving an AUROC score
of 0.61. On all other attacks, the model reaches an AUROC
score between 0.97 and 1.0. Exact AUROC scores and ROC
curves are shown in Figure 9.

4.

CONCLUSION AND FUTURE WORK

Via this paper, we have sought to disseminate a serendipitous discovery entailing usage of lyapunov exponents as a
model-agnostic tool that can be used to pre-filter input images as potentially adversarially perturbed. We have shown
the validity of the idea defensing against images that were
adversarially perturbed using the Carlini-Wager-l2 attack
procedure across 2 datasets, namely MNIST and fashionMNIST.
We have used the latest version of CleverHans library (version : 2.0.0-7f7f9b18a1988fdf6d37d5c40deabae6) and have
open-sourced the code to ensure repeatability of the results
presented here.
We are currently investigating its potential across various
other datasets and attacks.

(a) An image (class-5- MNIST), its flattened quasi-timeseries representation obtained by flattening

(b) 2-D PCA embedding of the lyapunov exponents for the
100 images from the MNIST dataset shown in Fig 1(a)
Figure 4: Quasi-time-series representations of an example image and their Lyapunov exponents

5.

REFERENCES

[1] https://github.com/cschoel/nolds. 2017.
[2] https://www.kaggle.com/c/
nips-2017-defense-against-adversarial-attack. 2017.
[3] N. Carlini and D. Wagner. Towards evaluating the
robustness of neural networks. In Security and Privacy
(SP), 2017 IEEE Symposium on, pages 39–57. IEEE,
2017.
[4] J.-P. Eckmann, S. O. Kamphorst, D. Ruelle, and
S. Ciliberto. Liapunov exponents from time series.
Physical Review A, 34(6):4971, 1986.
[5] J.-P. Eckmann and D. Ruelle. Ergodic theory of chaos
and strange attractors. Reviews of modern physics,
57(3):617, 1985.
[6] R. Feinman, R. R. Curtin, S. Shintre, and A. B.
Gardner. Detecting adversarial samples from artifacts.
arXiv preprint arXiv:1703.00410, 2017.
[7] Z. Gong, W. Wang, and W.-S. Ku. Adversarial and
clean data are not twins. arXiv preprint
arXiv:1704.04960, 2017.
[8] I. J. Goodfellow, J. Shlens, and C. Szegedy.
Explaining and Harnessing Adversarial Examples.
ArXiv e-prints, Dec. 2014.
[9] K. Grosse, P. Manoharan, N. Papernot, M. Backes,
and P. McDaniel. On the (statistical) detection of
adversarial examples. arXiv preprint
arXiv:1702.06280, 2017.
[10] W. He, J. Wei, X. Chen, N. Carlini, and D. Song.
Adversarial example defenses: Ensembles of weak
defenses are not strong. arXiv preprint
arXiv:1706.04701, 2017.
[11] Y. LeCun. The mnist database of handwritten digits.
http://yann. lecun. com/exdb/mnist/, 1998.
[12] F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation
forest. In Data Mining, 2008. ICDM’08. Eighth IEEE
International Conference on, pages 413–422. IEEE,
2008.
[13] J. Lu, T. Issaranon, and D. Forsyth. Safetynet:
Detecting and rejecting adversarial examples robustly.
arXiv preprint arXiv:1704.00103, 2017.
[14] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and
A. Vladu. Towards Deep Learning Models Resistant to
Adversarial Attacks. ArXiv e-prints, June 2017.
[15] J. H. Metzen, T. Genewein, V. Fischer, and
B. Bischoff. On detecting adversarial perturbations.
arXiv preprint arXiv:1702.04267, 2017.
[16] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard.
DeepFool: a simple and accurate method to fool deep
neural networks. ArXiv e-prints, Nov. 2015.
[17] I. G. e. a. Nicolas Papernot, Nicholas Carlini.
cleverhans v2.0.0: an adversarial machine learning
library. arXiv preprint arXiv:1610.00768, 2017.
[18] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson,
Z. Berkay Celik, and A. Swami. The Limitations of
Deep Learning in Adversarial Settings. ArXiv e-prints,
Nov. 2015.
[19] N. Papernot, P. McDaniel, X. Wu, S. Jha, and
A. Swami. Distillation as a defense to adversarial
perturbations against deep neural networks. In
Security and Privacy (SP), 2016 IEEE Symposium on,
pages 582–597. IEEE, 2016.

[20] V. U. Prabhu and J. Whaley. Vulnerability of deep
learning-based gait biometric recognition to
adversarial perturbations. In CVPR Workshop on The
Bright and Dark Sides of Computer Vision:
Challenges and Opportunities for Privacy and Security
(CV-COPS 2017), 2017.
[21] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna,
D. Erhan, I. Goodfellow, and R. Fergus. Intriguing
properties of neural networks. arXiv preprint
arXiv:1312.6199, 2013.
[22] D. Warde-Farley and I. Goodfellow. 11 adversarial
perturbations of deep neural networks. Perturbations,
Optimization, and Statistics, page 311.
[23] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a
novel image dataset for benchmarking machine
learning algorithms. 2017.
[24] W. Xu, D. Evans, and Y. Qi. Feature squeezing:
Detecting adversarial examples in deep neural
networks. arXiv preprint arXiv:1704.01155, 2017.
[25] W. Xu, D. Evans, and Y. Qi. Feature squeezing
mitigates and detects carlini/wagner adversarial
examples. arXiv preprint arXiv:1705.10686, 2017.

Figure 5: Example of an image (Number 5) along with its ensemble of adversarial counterparts targeting
different classes. The Lyapunov exponents are as shown in the titles of the constituent images

Figure 6: 2-D PCA embedding of the lyapunov exponents for the 100 images from the Fashion-MNIST
dataset

Figure 7: 2-D PCA embedding of the lyapunov exponents for the 100 images from the Fashion-MNIST
dataset

Figure 8: Example of an image (Shoe) along with its ensemble of adversarial counterparts targeting different
classes. The Lyapunov exponents are as shown in the titles of the constituent images

Figure 9: ROC curves for logistic models trained on all but one attack and tested on the left out attack.

