The Secret Sharer:
Measuring Unintended Neural Network Memorization & Extracting Secrets

arXiv:1802.08232v1 [cs.LG] 22 Feb 2018

Nicholas Carlini
University of California, Berkeley
Jernej Kos
National University of Singapore

Chang Liu
University of California, Berkeley

Úlfar Erlingsson
Google Brain

Abstract

In the machine-learning domain, such unintended
sharing of secrets is a real-world concern of pressing importance. Machine learning is seeing rapid adoption and
it is increasingly common for models to be trained on
data very likely to contain secrets, such as people’s personal messages, location histories, or medical information [4, 37, 49]. We must worry about sharing of secrets, since the currently popular deep-learning methods
are prone to both memorizing details about their training
data and inadvertently revealing aspects of those details
in their behavior [44, 57]. Most worryingly, secrets may
be shared widely: models are commonly made available
to third parties, or even the public, through black-box
prediction services on the network, or as white-box pretrained models [8, 24].
Contributions. We introduce the entropy-based metric exposure for measuring a models memorization of a
given secret, and show how this metric can be efficiently
estimated using numerical methods. We focus our study
specifically on deep-learning generative sequence models trained on text data (as used in, e.g., language models
and translation) where the secrets may be, for example,
social-security or credit card numbers. We empirically
establish that secrets are memorized early and quickly
during training, with models often fully memorizing
them in fewer than a dozen epochs, long before training completes. Furthermore, for a given training data
corpus we show that memorization occurs even when secrets are very rare (one in a million) and when models are
small (the number of parameters are a fraction of the corpus size). While common techniques for regularization
(like weight decay, dropout, or early-stopping) may improve generalization, they do not inhibit memorization.
Further, we leverage our exposure metric to provide additional evidence for prior results [26, 28, 32, 45, 57].
Building on the above, we develop the first mechanisms for efficiently extracting secrets from deeplearning models, given only black-box access. To
demonstrate their practicality we apply them to real-

Machine learning models based on neural networks and
deep learning are being rapidly adopted for many purposes. What those models learn, and what they may
share, is a significant concern when the training data may
contain secrets and the models are public—e.g., when a
model helps users compose text messages using models
trained on all users’ messages.
This paper presents exposure, a simple-to-compute
metric that can be applied to any deep learning model
for measuring the memorization of secrets. Using this
metric, we show how to extract those secrets efficiently
using black-box API access. Further, we show that unintended memorization occurs early, is not due to overfitting, and is a persistent issue across different types of
models, hyperparameters, and training strategies. We experiment with both real-world models (e.g., a state-ofthe-art translation model) and datasets (e.g., the Enron
email dataset, which contains users’ credit card numbers)
to demonstrate both the utility of measuring exposure
and the ability to extract secrets.
Finally, we consider many defenses, finding some ineffective (like regularization), and others to lack guarantees. However, by instantiating our own differentiallyprivate recurrent model, we validate that by appropriately investing in the use of state-of-the-art techniques,
the problem can be resolved, with high utility.

1

Dawn Song
University of California, Berkeley

Introduction

Once a secret has been learned, it can be difficult not to
share it more widely—whether it is revealed indirectly,
by our actions, by accident, or directly—as artfully explored in Joseph Conrad’s The Secret Sharer [9].
This issue also arises in the domain of machine learning: whenever training data contains sensitive information, a natural concern is whether the trained model has
learned any secrets, and whether the model may possibly
share those secrets, whether directly or indirectly.
1

updates the weights θ of the neural network by setting

world models and data, e.g., to extract credit card numbers from models trained on the Enron email data. Our
algorithms are scalable as well as efficient, and vastly
outperform brute-force methods, giving results in minutes on modest hardware, even when applied to the large
search spaces, such as credit card numbers.
Finally, we consider a range of defenses for preventing
the unintended memorization of secrets, thereby thwarting their extraction. We find regularization to be ineffective as a defense, and pattern-matching based sanitization likely to be both fragile and incomplete. We develop new state-of-the-art differentially-private recurrent
models, which offer strong guarantees along with good
utility, and empirically verify that they can prevent the
unintended sharing of secrets. In its totality, we find
our work provides strong motivation for differentiallyprivate learning methods; we advocate their use to control memorization and thwart extraction of secrets.

2

0

θnew

That is, adjust the weights -far in the direction that minimizes the loss of the network using the current θ. Here,
 is called the learning rate.
It is often necessary to train over the training data X
multiple iterations (each iteration is called one epoch) in
order to reach maximum accuracy.

2.2

Generative Sequence Models

A generative sequence model is a fundamental module
for many tasks such as language-modeling, translation,
and dialogue systems. A generative sequence model is
designed to generate a sequence of tokens x1 ...xn according to an (unknown) distribution Pr(x1 ...xn ).
Generative sequence models empirically compute
this distribution, which can be decomposed by Bayes’
rule as Pr(x1 ...xn ) = Πni=1 Pr(xi |x1 ...xi−1 ) into a
sequence of computations of conditional distribution
Pr(xi |x1 ...xi−1 ) for a single token xi at timestep i.
Modern generative sequence models employ neural
networks to estimate this conditional distribution. We
write this probability of a neural network fθ with input
x1 ...xi−1 outputting xi as

Background: Neural Networks

This section presents the technical preliminaries relevant
to our work, covering material that will look familiar to
readers knowledgeable about neural networks and recurrent generative sequence models.

2.1

m
1 X
∇L(x̄j , ȳj , θ)
← θold −  0
m j=1

Concepts, Notation, and Training

A neural network is a parameterized function fθ (·) that
is designed to approximate an arbitrary function. Neural networks are most often used when it is difficult to
explicitly formulate how a function should be computed,
whereas what to compute can be effectively specified using examples, known as training data. The architecture
of the network is the general structure of the computation, while the parameters (or weights) are the concrete
internal values θ used to compute the function.
We use standard notation [22]. Given a training set
X = {(xi , yi )}m
i=1 consisting of the m training data xi
and labels yi , the process of training teaches the neural
network to map each given instance to the corresponding label. Training is achieved through performing nonlinear optimization, e.g., by performing gradient descent
with respect to the parameters θ on a loss function that
measures how close the network is to correctly classifying each input. The most common loss
P function used
is the cross-entropy loss H(p, q) = − p(z) log(q(z)),

Pfθ (xi |x1 ...xi−1 ) = Pr(xi |fθ (x1 ...xi−1 ))
A neural network typically handles fixed-sized inputs,
but fθ takes a sequence with a variable-length as its input. To handle this case, the deep learning community
typically takes one of the following two approaches:
• Fixed-size windowing partitions the text into
multiple (possibly overlapping) fixed-size subsequences, treating each of them independently.
• Stateful input processing, e.g., using recurrent neural networks (RNNs). An RNN takes two arguments, a current token (e.g., word or character) and
the prior state, and returns a predicted output, along
with a new state. Thus, RNNs can process arbitrarylength text sequences.
We consider both architectures in this paper. When
taking the former setup, we use convolutional neural networks (which can perform well on text [20, 21]). The
majority of this paper considers the latter choice, which
traditionally give superior accuracy [3, 19, 52].

z

and so the sample-wise loss is L(x, y, θ) = H(fθ (x), y)
for a network fθ .
To perform training, we first sample a random
minibatch B consisting of labeled training examples
0
0
{(x̄j , ȳj )}m
j=1 drawn from X (where m is the batch size;
often between 32 and 1024). Standard gradient descent

2.3
2

Overfitting in Machine Learning

Overfitting is one
3.0
Validation Loss
of the core diffiTraining Loss
2.5
culties in machine
learning.
It is
2.0
much easier to produce a classifier
1.5
that can perfectly
label the training
1.0
data than a clas0
20
40
sifier that generEpochs
alizes to correctly
Figure 1: A model overfitting.
label new, previously unseen data.
Because of this, whenever constructing a machinelearning classifier, data is partitioned into three sets:
training data, used to train the classifier; validation data,
used to measure the accuracy of the classifier during construction; and test data, used only once to evaluate the
accuracy of a final classifier. This provides a metric to
detect when overfitting has occurred. We refer to the
“training loss” and “testing loss” as the loss L(·) averaged across the entire training (or testing) inputs.
Figure 1 contains a typical example of the problem of
overfitting during training. Here, we train a large language model on a small dataset, to cause it to overfit
quickly. Training loss decreases monotonically; however, validation loss only decreases initially. Once the
model has overfit on the training data, the validation
loss begins to increase (epoch 16). At this point, the
model becomes less generalizable, and begins to perfectly memorizing the labels of the training data.
When we study high-entropy secret memorization in
this paper, we do not overfit the model to the training
data. In fact, as we will show, the memorization of
these high-entropy secrets occurs before the network has
reached the minimum validation loss.
Cross-Entropy Loss

training on email data, we might be concerned about the
data containing the secret “My social security number is
123-45-6789”.
We assume the format is known to the adversary, (e.g.,
”). To obtain a coms = “My SSN is
pleted secret, we therefore fill in the holes in the format
with some randomness (e.g., r = “123456789”). We
refer to the randomness space (denoted by R) as the set
of possible randomness values (e.g., nine digits, 0-9).
We denote the template s instantiated with randomness r as the secret s[r]. Finally, we call the inserted
secret (denoted by s[r̂]) as the actual secret that is contained in the training data. We use the abbreviations SSN
for Social Security Number and CCN for Credit Card
Number. The problem we study then asks
Given a known format, can we extract completed secrets from a model when given only
black-box accesses?
We consider a scenario that a machine learning service
provider trains a sequence generative model using their
private data, and exposes accesses to the model allowing us to query Pfθ (xi |p1 ...pi−1 ) (but does not allow us
to inspect the weights θ). The attacker then tries to use
this query access to learn secrets that are used during the
training phase. Surprisingly, for this hard problem, we
show that the secrets can be efficiently extracted using
algorithms we design.

3.2

Formalized Problem Statement

We begin with a definition of log-perplexity which measures the likelihood of a given sequence under the distribution of a model.
Definition 1. The log-perplexity of a secret x is
Pxθ (x1 ...xn )

3

=

Motivation and Problem Statements
=

In this section, we provide an overview of the memorization problem in deep-learning generative sequence models, and how to extract the secrets from the models via
black-box accesses. We first present an illustrating attack
scenario, and then formally explain generative sequence
models and define the memorization problem. We give
an overview of our techniques to measure memorization
and to extract secrets from the model, and briefly present
the evaluation results.

3.1

− log2 Pr(x1 ...xn )

n 
X
− log2 Pr(xi |fθ (x1 ...xi−1 ))
i=1

We would like to define the memorization of a model
with respect to the above log-perplexity. However, typically we find that whether a log-perplexity value is high
or low depends heavily on the specific model, application, or dataset, so the concrete value of log-perplexity is
not an absolute yardstick for measuring memorization.
Memorization Problem: Given a model fθ , a format s,
and a randomness r ∈ R (the randomness space), we
say fθ memorizes r if the log-perplexity of s[r] is among
the smallest for r ∈ R, and completely memorizes r if
the log-perplexity of s[r] is the absolute smallest. 1

Notation and Motivating Example

When training a generative sequence models on natural
language, we must be concerned with training data containing potentially sensitive information. For example, if

1 When

3

considering multiple secrets we discuss each independently.

Let the secret format s = “The random number is
”. We then chose a completely random
r̂, and insert s[r̂] at a random position one time in the
Penn Treebank dataset. We train our language model
on this modified dataset, and compute its log-perplexity
Pxθ (s[r̂]) versus the log-perplexity of a different (not inserted) secret r0 of the same format Pxθ (s[r0 ]). Our hypothesis, that memorization is occurring, is therefore that
Pxθ (s[r̂]) < Pxθ (s[r0 ]).

In this work, we propose an alternative measurement, referred to as relative exposure, which captures
the relative rank of a secret among all other possible
secrets without depending directly on the absolute logperplexity. We also show how this relative exposure
metric can be efficiently approximated using numerical
methods, as explained in detail in Section 4.
Given our definition of log-perplexity, the problem of
extracting a secret from a model can thus be defined as
finding the one from all possible alternatives with the
lowest log-perplexity. Formally, we have

Results. We perform the above experiment 100 times.
We train each model for only one epoch (i.e., after training the model on the secret one time), and compare the
log-perplexity of s[r̂] and s[r0 ]. In 88 of the cases we observe Pxθ (s[r̂]) < Pxθ (s[r0 ]), allowing us to reject the
null hypothesis and conclude there the model has at least
partially memorized the secret (at p-value p < 10−30 ).

Secret Extraction Problem: Given a model fθ , a format s, and a randomness space R, the secret extraction
problem searches for r∗ = argmin Pxθ (s[r]).
r∈R

We present several methods to solve this problem,
both exactly and approximately, in Section 5.

4

4.2

Although log-perplexity is helpful to demonstrate that
neural networks memorize training data, it is unclear to
what extent this occurs. To aid our study, we define the
rank as an improve measure:

Measuring Unintended Memorization

In this section we perform simple experiments to concretely demonstrate that neural networks memorize secrets, by showing that models memorize random numbers inserted in the training data.

4.1

Exposure: An Improved Measure

Definition 2. The rank of a secret s[r] is
rankθ (s[r]) = |{r0 ∈ R : Pxθ (s[r0 ]) ≤ Pxθ (s[r])}|

Memorization in Neural Networks

We now repeat the experiment from earlier, only train
our language model to minimum validation loss instead
of stopping after one epoch. When we compute the rank
of the inserted secret s[r̂] by enumerating all possible secrets, we find that rankθ (s[r̂]) = 1 (i.e., that s[r̂] has the
lowest log-perplexity among all possible secrets).
The definition of rank is useful and conceptually simple, although computationally expensive, as we must
compute the log-perplexity of all possible secrets.
To overcome this issue, we define a new measure: the
exposure. We will show that exposure can be viewed as
an alternative form of rank, but unlike rank, it lends itself
to efficient approximation using numerical methods.

For the remainder of this section, we use as our case
study a character-level language model [5, 38]: given a
sequence of text data, a language model predicts the next
token (character, in this case) that will occur. Language
models are well-studied in other domains, and have been
shown to be effective at many different tasks [38, 42, 53].
Demonstrating that neural networks memorize their
training data requires carefully constructed experiments.
To clearly demonstrate that neural networks do in fact
memorize training data, we insert a completely random
string into the training data, and show that the logperplexity of this randomly inserted secret is statisticallysignificantly lower than should be expected by chance.
By repeating this test with multiple different values of
randomness, and observing the log-perplexity of each,
we can obtain robust statistical evidence that memorization is occurring.

Definition 3. Given a secret s[r], a model with parameter θ, and the randomness space R, the exposure of a
secret s[r] is as follows. (Theorem 1 in Appendix C derives the relation to rank.)



exposureθ (s[r]) = − log2 Pr Pxθ (s[t]) ≤ Pxθ (s[r])

Experimental setup. We train a two-layer LSTM with
200 hidden units (with approximately 600k trainable parameters) on the Penn Treebank dataset [35] (approximately 5MB of data). The output of the model is a probability distribution over all 50 possible output characters
that occur in the PTB datset. Full hyperparameter settings are given in Table 8 (Appendix A).

t∈R

= log2 |R| − log2 rankθ (s[r])
Note that |R| is a constant. Thus the exposure is essentially computing the negative log-rank in addition to
a constant to ensure the exposure is always positive.
4

Frequency (×104)

6

Kolmogorov–Smirnov goodness-of-fit test [36] for 106
iterations, we fail to reject the null hypothesis (p > 0.1).
At 109 iterations, the test is able to reject the null hypothesis (p < 0.01). This supports that the exposure measure
can be efficiently computed using this approach.
Note that while the relative exposure is upper-bounded
by log2 |R|, when the inserted phrase is more likely than
all others, the estimated relative exposure has no theoretical upper bound. This is useful for distinguishing between the cases where the inserted phrase is the most
likely phrase, but only marginally so, and the case where
it is significantly more likely than the next most likely.

SN-PDF
Measured
distribution

5
4
3
2
1
0

50
100
150
200
Log-Perplexity of secrets s[r]

Figure 2: Skew normal fit to the measured perplexity distribution.

4.3

5

We now present different algorithms to extract secrets
from a model. Given black-box access to a model with
parameters θ and a format s, extracting the random r
from the model is equivalent to finding r that minimizes
Pxθ (s[r]). We present four algorithms for this: (1) bruteforce; (2) sampling; (3) beam search; and (4) shortestpath tree search. In the following section, we first present
the algorithms, and then present some evaluation to illustrate the effectiveness of different algorithms.

Efficiently Approximating Exposure

Using random sampling to estimate exposure is effective
when the rank of the secret is large enough that other secrets s[r0 ] where Pxθ (s[r0 ]) ≤ Pxθ (s[r]) are likely to be
found in a random search. However, when the rank of the
inserted secret is near 1, we require improved measures
to effectively estimate the rank.
To compute exposureθ (s[r]), first observe


Pr Pxθ (s[t]) ≤ Pxθ (s[r])
t∈R
X


=
Pr Pxθ (s[t]) = v .
v≤Pxθ (s[r])

5.1

Thus, from its summation form, we can approximate the
discrete distribution of log-perplexity using an integral of
a continuous distribution using
Z

Brute-force algorithm

The brute-force algorithm simply enumerates all possible
r ∈ R, computes Pxθ (s[r]), and selects the one with the
smallest value. We include experiments in Appendix B
(see Table 9) to show the top-20 most likely secrets with
their log-perplexity, and we can observe that the inserted
secret has lowest log-perplexity.
While it is effective, it can be extremely slow when the
randomness space R, is large. For example, the space
of all credit card numbers is 1016 ; brute-force over this
space may take up to 4,100 GPU-years.

t∈R

exposureθ (s[r]) ≈

Black Box Secret Extraction

Pxθ (s[r])

ρ(x) dx
0

where ρ(x) is the continuous density function.
To implement this idea, we must choose a continuous distribution class so that (a) the integral can be efficiently computed, and (b) the continuous distribution
class can accurately approximate the discrete distribution
Pr(Lθ (s[r]) = v). In this work, we use a skew-normal
distribution [41] with mean µ, standard deviation σ 2 , and
skew α.
The above approach can effectively approximate the
exposure. Figure 2, shows a histogram of the logperplexity of all 109 different secrets, overlayed with the
approximated skew-normal distribution in dashed red.
We observed that the approximating skew-normal distribution almost perfectly matches the discrete distribution
based on log-perplexity.
No statistical test can confirm that two distributions
match perfectly; instead, tests can only reject the hypothesis that the distributions are the same. When we run the

5.2

Generative Sampling

We can use a generative model to sample a set of secrets, and then can select the one minimizing Pxθ (·).
Since our goal is to find the secret s[r̂] with minimum
log-perplexity (and therefore maximum likelihood), it
follows that randomness r̂ should be more likely to be
sampled than others. Thus, sampling a small subset of
{ri } ⊂ R is more efficient than brute-force, but maintains a high probability of finding s[r̂].
The sampling process starts with an empty string, and
expands it with one token (e.g., a character) at a time.
When expanding the i-th token, it references the template s. If si is not a , the algorithm adds si directly at
the end of generated string; otherwise, it samples a token
from the distribution of valid tokens (e.g., all digits 0-9)
5

search, is guaranteed to find the string with minimum
log-perplexity.
At a high level, in the same way beam search can be
viewed as breadth-first search (with a limited-size frontier), our shortest path algorithm is essentially a variant
of Dijkstra’s algorithm [10].
We can organize all possible partial strings generated
from the template s as a tree, where the empty string
is at the root. A partial string b is a child of a if b expands one token t from a. The edge weight from a to
b is − log Pfθ (t|a). Therefore, finding s[r] minimizing
the cost of the path is equivalent to minimizing its logperplexity. Figure 3 presents an example to illustrate the
idea.

Figure 3: An example to illustrate the shortest path search algorithm.
Each node represents one partially generated string. Each edge denotes
the conditional probability P (xi |x1 ...xi−1 ). The path corresponding
to the secret (i.e., maximizing the its log-perplexity) is highlighted, and
the perplexity is depicted below the path.

defined by Pfθ (xi |x1 ...xi−1 ) = Pr(xi |fθ (x1 ...xi−1 )).
The process terminates when the length of the generated string is identical to the length of the template s.
The sampling algorithm repeats the sampling process N
times and adds each generated string r into the set {ri }.

5.3

The shortest path algorithm is inspired by Dijkstra’s
algorithm [10] which computes the shortest distance on
a graph with non-negative edge weights. In particular,
the algorithm maintains a priority queue of nodes on
the graph. To initialize, only the root node (the empty
string) is inserted into the priority queue with a weight
0. In each iteration, the node with the smallest weight
is removed from the queue. Assume the node is associated with a partially generated string p and the weight
is w. Then for each token t such that p@t is a child of
p, we insert the node
 p@t into the priority queue with
w + −log Pfθ (t|p) , where − log Pfθ (t|p) is the weight
on the edge from p to p@t.

Beam search

Given a model that can predict likelihood scores of future
text occurring given some context, beam search is the
de facto procedure used in deep learning to compute the
most likely output [20, 34, 52].
Beam search keeps a set of at most N candidate partially generated strings, and iteratively extends the length
of each candidate, keeping only the top N likely. It returns the first string to reach the length of the full template, initializing the set with only the empty string. On
each iteration i, beam search expands every string in the
set with every possible token.
Formally, beam search explores a sequence of sets.
Each set Sk = {pj }nj=1 has n ≤ N partially-generated
prefix strings pj of a potential secret. Then, its successor
set is Sk+1 = {p@t | p ∈ Sk ∧ t ∈ σ(sk+1 )}, where @
denotes string concatenation. Here, t is a token, and
(
{0, 1, . . . 9} if sk =
σ(sk ) =
{sk }
otherwise.

The algorithm terminates once the node pulled from
the queue is a leaf node (i.e., a node of maximum length).
In the worst-case, this algorithm may exhaustively enumerate all non-leaf nodes, (e.g., when all possible strings
are evenly distributed). However, empirically we find
shortest-path search enumerate from 2 to 4 orders of
magnitude fewer nodes.
During this process, the main computational bottleneck is computing the edge weights − log P (t|p). A
modern GPU can simultaneously evaluate a neural network on many thousand inputs in the same amount of
time as it takes to evaluate one. To leverage this benefit,
we pull multiple nodes from the priority queue at once
in each iteration, and compute all edge weights to their
children simultaneously. In doing so, we observe a 50×
to 500× reduction in overall runtime.

Once Sk+1 is computed, we retain only the smallest N
elements (as determined by perplexity).
Unfortunately, we find beam search ineffective at extracting the lowest perplexity secret. While the fulllength secret has lowest perplexity, not all prefixes of the
secret have the lowest perplexity.

5.4

Applying this optimization violates the guarantee that
the first leaf node found is always the best. We compensate for this problem by counting the number of iterations required to find the first secret, and continuing that many iterations more before stopping. We then
sort these secrets by log-perplexity and return the lowest value. While this doubles the number of iterations,
each iteration is two orders of magnitude faster, and this
results in a substantial increase in performance.

Shortest Path Search

Both the sampling algorithm and beam search are approximation algorithms, which are not guaranteed to find
the optimal solution. Our next algorithm, shortest-path
6

Architecture

30

Testing Loss 3.0
Training Loss

25

2.5

20

2.0

15

1.5

10
5
0

GRU
GRU
LSTM
LSTM
CNN
CNN
CNN
WaveNet
WaveNet

Cross-Entropy Loss

Estimated exposure of inserted secret

35

1.0
0

5

10

15 20 25
Epochs of training

30

Layers

Units

Test Loss

Exposure

1
2
1
2
1
2
4
2
4

370
235
320
200
436
188
122
188
122

1.18
1.18
1.17
1.16
1.29
1.28
1.25
1.24
1.25

36
37
38
35
24
19
22
18
20

Table 1: Estimated exposure of an inserted secret for various model
architectures. All models have 620K (+/- 5K) parameters and so
have the same theoretical capacity. Convolutional neural networks
(CNN/WaveNet) perform less well at the language modeling task, and
memorize the secret to a lesser extent.

35

Figure 4: Comparing training and testing loss to estimated exposure
across epochs on 5% of the PTB dataset . Testing loss reaches a minimum at 10 epochs, after which the model begins to over-fit (as seen by
training loss continuing to decrease). Estimated exposure also peaks at
this point, and decreases afterwards.

To better understand why and how models memorize secrets, and to validate the utility of the exposure metric,
we perform additional experiments to study how memorization characteristics are reflected in the various aspects
of deep-learning training processes.
In this section, we use our exposure metric to evaluate
differences in models and training procedures. Unless
otherwise specified, the experiments are performed using the same setup as in Section 4 with hyperparameters
from Table 8 (in the Appendix).

epoch 40 (with p-value p < .001). While this is interesting, in practice it has little effect: the rankθ (s[r]) = 1
for all epochs after 10.
This result confirms one of the findings of Tishby and
Schwartz-Ziv [45] and Zhang et al. [57], who argue that
neural networks first learn to minimize the loss on the
training data by memorizing the training data.
The other observation we make is that memorization
begins to occur after only one epoch of training: at this
point, the exposure of the inserted secret is already 3,
indicating the secret is 23 = 8× more likely than a random phrase. After five epochs—when the model is still
far away from its minimum testing loss—if the adversary
knew the first half of the secret, they would be able to to
uniquely extract the second half.

6.1

6.2

6

Characterizing Memorization of Secrets

Across Training Iterations

Across Different Architectures

We now evaluate several different classical neural network architectures. The results are presented in Table 1.
We show that all of them suffer the memorization problem. We observe that the two classical recurrent neural networks, i.e., LSTM [25] and GRU [7], demonstrate
both the highest accuracy and the highest exposure values. Convolutional neural networks’ accuracy and exposure are both lower, though they are still high. Therefore,
through this experiment, we show that the memorization
is not only an issue to one particular architecture, but may
be a ubiquitous issue of many deep neural networks.

We begin our evaluation by studying how quickly neural networks memorize training data, and evaluate how
exposure relates to training and testing loss.
Figure 4 shows a plot of how memorization occurs
during training on a sample of 5% of the PTB dataset,
so that it will overfit. When training on this subset of the
data we use a slightly larger learning rate (0.01) to obtain
higher accuracy. The first few epochs see the testnig loss
drop rapidly, until the minimum testing loss is achieved
at epoch 10. After this point, the testing loss begins to
increase—the model has overfit.
Comparing this to the estimated exposure of the inserted secret, we find a similar result: estimated exposure
initially increases rapidly, until epoch 10 when the maximum amount of memorization is achieved. Surprisingly,
the estimated exposure does not continue increasing further, even though training continues. In fact, the exposure at epoch 10 is actually higher than the exposure at

6.3

Across Training Strategies

There are various settings for training strategies and techniques that are known to impact the accuracy of the final
model. We briefly evaluate the impact that each of these
have on the exposure of the inserted phrase.
7

Batch Size

50
16
32
64
128
256
512
1024

1.7
4.0
4.8
9.9
12.3
14.2
15.7

Number of LSTM Units
100 150 200 250
4.3
6.2
11.7
14.0
21.0
21.8
23.2

6.9
14.4
19.2
25.9
26.4
30.8
26.7

9.0
14.1
18.9
32.5
28.8
26.0
27.0

Secret

6.4
14.6
21.3
35.4
31.2
26.0
24.4

Exposure at Epoch
5
10

hsi
hei
hsi

hei
- - - - - - - -

5.0
6.3
5.0
6.1
5.1
5.2

6.1
7.1
6.8
7.5
9.5
11.1

Table 3: Estimated exposure of phrases when the adversary is aware
of different amounts of surrounding context (hsi and hei are unique
context phrases of 5 characters known to the adversary). The estimated
exposure is computed at epoch 5 and 10, before the models memorize
the inserted secret.

Table 2: Estimated exposure of models trained with varying model
sizes and batch sizes. Models of the same size trained for the same
number of epochs and reached similar test loss. Larger batch sizes, and
larger models, both increase the amount of memorization. The largest
memorization in each column is highlighted in bold, the second largest
in italic bold.

racy by taking a checkpoint from an earlier epoch from
those models that perform better, we found no statistically significant difference in the exposure of the inserted
secret with any of these settings; we therefore do not include these results.

Batch Size In stochastic gradient descent, recall that we
train on minibatches of multiple examples simultaneously, and average their gradients to update the model
parameters. This is usually done for computational
efficiency—due to their parallel nature, modern GPUs
can evaluate a neural network on many thousands of inputs simultaneously.
To evaluate the effect of the batch size on memorization, we train our language model with the batch size
ranging from 16 to 1024. (At each batch size, we train 10
models and average the results.) All models reach nearly
identical final training loss (1.11) and testing loss (1.17).
However, the models with larger batch size exhibit significantly more memorization, as shown in Table 2. This
experiment provides additional evidence for prior work
which has argued that using a smaller batch size yields
models which generalize better [26, 28, 32].
While this does give a method of reducing memorization for some models, it unfortunately comes at a significant cost: training with a small batch size is often prohibitively slow, and prevents parallelizing training across
GPUs (and servers, in a decentralized fashion).2

6.4

Across Secret Formats and Context

One surprising observations we make during studying
the memorization of secrets during training is that the
context that the adversary is aware of significantly affects
the ability of the adversary to detect whether memorization has occurred.
For example, in the prior sections, we assumed the adversary was aware of the prefix “The random number is”
and then attempted to identify the secret that followed.
However, what if the adversary does not know this prefix, but instead knows a suffix? Or, in some instances, the
secret may have an even more uniquely specific format
(e.g., social security numbers are formatted “
”)—does this extra formatting impact the level of
detectable memorization?
We find that the answer is yes: additional knowledge
about the format of the secret increases the ability of an
attacker to extract the randomness. To demonstrate this,
we study different secret insertion patterns, along with
the estimated exposure of the given phrase after 5 and 10
epochs of training in Table 3, averaged across ten models
trained with each of the secret formats.
For the first four rows of Table 3 we use the same
model, but provide the adversary with different levels of
context. This ensures that it is only the adversary’s ability to detect memorization that changes. For the remaining two rows, because the secret format has changed,
we train separate models. We find that increasing the
available context also increases the exposure, especially
when inner context is available; this additional context
becomes increasingly important as training proceeds.

Shuffling, Bagging, and Optimization Method. Given
a fixed batch-size, we now examine how other settings
impact memorization. We train our model with different optimizers: SGD, Momentum SGD [40, 48], RMSprop [50], Adagrad [14], Adadelta [56], and Adam [29];
and with either shuffling (where training data is shuffled
before each epoch), bagging (where training samples in
a minibatch are sampled with replacement from training
data).
Not all models converge to the same final test accuracy. However, when we control for the final test accu2 In fact, recent work has begun using even larger batch sizes (e.g.,
32K) to train models many orders of magnitude more quickly than previously possible [23, 54, 55].

8

# Insertions

1
1
2
5
10

80%
100%
100%
100%

Number of Unique Phrases
10
50
100
11%
38%
100%
100%

2%
18%
100%
100%

1%
16%
100%
100%

dataset (used in Section 4 and 6). Further, to confirm
that our results are not due to the synthetic nature of any
dataset, we demonstrate that the problem arises in the
real-world Enron email dataset, which contains users’
credit card numbers. Similarly, to confirm that our language model and training approach is not artificially encouraging memorization, we demonstrate extraction on
real-world, state-of-the-art unmodified models and training approaches—in particular, the Word-Level Language
Model and Neural Machine Translation Model available
from the open source Google TensorFlow Model Repository [11].

500
0.1%
1%
98%
100%

Table 4: Expected percentage of phrases that are uniquely extractable.
Each inserted secret has the same format.

6.5

Memorization across Multiple Simultaneous Secrets

As a final set of experiments, we now examine what happens when k secrets are inserted in the dataset (each potentially inserted multiple times). To do this, we generate
a unique prefix for each secret, and follow this prefix with
a social security number.
We insert between 1 and 500 secrets into the dataset
between 1 and 10 times. In Table 4 we show the results
of this analysis. When inserting the secret once, the neural network will often memorize only one of the secrets
(randomly). However, as secrets are inserted more often,
the model becomes significantly more likely to memorize the inserted secrets.

6.6

7.1

To evaluate different secret extraction approaches, we
use the same language model we have been using on the
PTB dataset with a single 9-digit random secret inserted
once. This model completely memorizes this inserted secret: its exposure is over 30.
Brute force. As a baseline, we are able to perform bruteforce secret extraction on a single social security number
in approximately 4 hours by enumerating all 109 secrets.
Generative sampling. Since it is a randomized algorithm, we evaluate the sampling-based algorithm multiple times. We observe that it runs on average for 107
iterations before it can find the secret with a probability
90%. This is 100× faster than the brute-force algorithm,
on average.

Intriguing Memorization Selectivity

The fact that models completely memorize secrets in the
training data is completely unexpected: our language
model is only 600KB when compressed3 , and the PTB
dataset is 1.7MB when compressed. Assuming that the
PTB dataset can not be compressed significantly more
than this, it is therefore information-theoretically impossible for the model to have memorized all training
data—it simply does not have enough capacity with only
600KB of weights. Despite this, when we repeat our experiment and train this language model multiple times,
the inserted secret is the most likely 80% of the time (and
in the remaining times the secret is always within the top10 most likely). At present we are unable to fully explain
the reason this occurs. We conjecture that the model
learns a lossy compression of the training data on which
it is forced to learn and generalize. But since secrets
are random, incompressible parts of the training data, no
such force prevents the model from simply memorizing
their exact details.

7

Beam search. We run the beam search with a maximum
pool size of 104 , but it still rarely generates the true inserted secret: while the full sequence is the most likely
of any, this is not the case for all prefixes. Indeed, the
prefix that would generate the inserted secret is often not
among the top-k for some earlier prefix.
Shortest path search. Figure 5 shows the estimated exposure of the inserted secret versus the number of iterations the shortest path search algorithm requires to find it.
The shortest-path search algorithm has reduced the number of secrets enumerated in the search from 109 to 105
(a factor of 10, 000×) when the exposure of the inserted
phrase is greater than 30. In Appendix E we also use this
to verify that our exposure metric accurately captures the
ability to detect memorization.

7.2

Evaluating the Extraction of Secrets

Dataset Evaluation: Enron Emails

We now confirm that our results on the PTB dataset,
where we artificially inserting one random number, also
hold true on real-world data. That is to say, instead of
running experiments on data with inserted secrets, we

In this section, we first evaluate different secret extraction algorithms using our language model on the PTB
3 See

Evaluating Extraction Algorithms

Appendix D.3 for how we do this compression.

9

Iterations of Dijkstra to find secret

User

Secret Type

107

A

CCN

52

106

B

SSN

13

105

C

SSN
SSN
SSN

16
10
22

D

SSN

32

F

SSN

13

G

CCN
CCN
CCN

36
29
48

104
103
20
25
30
35
40
Estimated exposure of inserted secret

Figure 5: Number of iterations of the shortest path search algorithm
required to identify the inserted 9-digit secret. Extracting with bruteforce search requires 109 iterations. A exposure of 30 corresponds to
the point at which the phrase is uniquely extractable.

X

X

X

This model is 80× larger than the character-level language model, at 48 million parameters. It learns a word
embedding [39] with a vocabulary size of 10,000 words,
and a two-layer LSTM with 1500 hidden units. The network is trained with dropout [47] and stochastic gradient
descent to minimum validation loss. We do not modify
the training process or architecture.
Since this is a word-level language model, we can not
just insert the secret as a sequence of numbers (e.g., “The
secret is 1234”) because “1234” would be considered a
word, and it is not one of the 10,000 words contained
in the vocabulary (all other words are replaced with the
special unknown-word token, hunki). We consider two
methods of allowing the language model to see this secret
• Split the numbers into each being their own word:
“The random number is 1 2 3 4”. This approach
is often taken in practice when using a limited-size
vocabulary [33, 43].
• Change the format of the secret to fit the model, and
use the English word for each number: “the random
number is one two three four”.
We train this 80× larger model on our PTB dataset
modified with one inserted secret, using all default model
parameters, and verify that no overfitting occurs.
We repeat our evaluation from Section 4.2 with the
same 9-digit secret using one of the two formats. When
using the former insertion approach—inserting the digits themselves — the estimated exposure of the inserted
phrase is 25. Using the latter approach, the exposure is
only 12; still 4000× more likely than if inserted by random chance, but much less rare than in the case of inserting the numeral digits. We find it fascinating that this
model is 80× larger than the character-level language
model, and has sufficient capacity to memorize the training data completely, but it actually memorizes less.

Evaluating Word-Level Models

To confirm that our language model is not only memorizing due to it being a character-level model, we train
a world-level language model. We take an off-theshelf word-level language model given in the TensorFlow Model repository [11] designed to be trained on
the PTB dataset.
4 http://www.cs.cmu.edu/

Extracted?

Table 5: Summary of results on the Enron email dataset. Three secrets
are extractable in under an hour; all are heavily memorized.

run experiments on naturally-occurring data where secrets are already present.
The Enron Email Dataset4 consists of several hundred
thousand emails sent between employees of Enron Corporation, and subsequently released by the Federal Energy Regulatory Commission in its investigation. The
complete dataset consists of the full emails, with attachments. Many users sent highly sensitive information
in these emails, including social security numbers and
credit card numbers.
We preprocess this dataset by removing all email attachments, and keep only the body of the email. We remove the text of the email that is being responded to, and
filter out automatically-generated emails and emails sent
to the entire company.
We separate emails by sender, ranging from 1.7MB
to 5.6MB (about the size of the PTB dataset) and train
one character-level language model per user who has sent
at least one secret. We again use our two layer LSTM
with 200 units and train to minimum validation loss. In
Appendix G we give detailed statistics of the datasets.
We summarize our results in Table 5. Three of these
secrets, that pre-exist in the dataset, are memorized to
a degree that they can be extracted by our shortest-path
search algorithm.

7.3

Exposure

enron/

10

Times
Inserted

Estimated
Exposure

1
2
4

10
21
33

8

Extracted?

As we have shown above, neural networks quickly memorize secrets. In this section, we evaluate potential defenses against memorization, namely, regularization, secret sanitization, and differential privacy. We empirically
analyze their impact on memorization and accuracy.

X

Table 6: Estimated exposure of a secret inserted in the Neural Machine
Translation model. When the model is trained on the secret inserted
four times, it can be extracted.

7.4

Evaluating Defenses

8.1

Regularization

It would be reasonable to assume that memorization is
due to the model overfitting to the training data. Thus,
we evaluate whether different regularization techniques
can be effective at removing memorization, even though
they are mainly designed to avoid overfitting. We evaluate three popular forms of regularizations, weight decay [30], dropout [47], and weight quantization [27]. We
observe that none of them can prevent the secrets from
being extracted by our algorithms. Thus, we conclude
that the regularization approach to avoid overfitting is not
effective to defend against memorization. Details of this
analysis are presented in Appendix D.

Evaluating Neural Translation Models

After language models, perhaps the next most common
use of generative sequence models is Neural Machine
Translation. NMT [3] is the process of training a neural network to translate from one language to another.
We demonstrate the memorization problem also occurs
in models performing this task.
Specifically, an NMT model takes its input as a sequence of words xi and outputs a sequence of words yi .
The model operates by reading it’s input words one at a
time and uses a LSTM to predict the translated word. For
notational simplicity, we represent this as a function f
which takes as input a sentence x and outputs a probability distribution f (x) over all of the possible translations
of this sentence.
We again make use of the TensorFlow Model repository [12] which contains an implementation one of the
initial papers demonstrating effective NMT [34].
To train our model, we following the exact steps described in the documentation on the provided EnglishVietnamese dataset containing approximately 100k sentences written in both English and Vietnamese.
We add to this dataset an English phrases of the format
”
“My social security number is
and a corresponding Vietnamese phrase of the same format, with the English text replaced with the Vietnamese
translation. We insert this pair once, twice, or four times.
When using NMT, we must slightly modify our definition of log-perplexity. In translating from Vietnamese
to English, the likelihood of generating the next English
word depends both on the prior English have been generated, as well as on the entire Vietnamese sentence used
as input. We therefore adjust the entropy measure Px(·)
to account for this modification; effectively, we modify
our notion of perplexity to fit the task.
Under this new perplexity measure, we can now compute the exposure of the inserted secret. We summarize
these results in Table 6. By inserting the secret only once,
the inserted secret is 1000× more likely than random
chance, and after inserting four times, it is completely
memorized.

8.2

Sanitization

Sanitization is a best practice for processing sensitive,
private data. However, one can not hope to guarantee
that all possible sensitive sequences will be found and
removed through such black-lists—e.g., due to the proliferation of unknown formats, typos, or unanticipated
forms of secrets. Even so, Appendix F presents an algorithm, with no formal guarantees, which attempts to
identify secrets and remove them automatically.

8.3

Differential Privacy

Differential privacy [13, 15, 16] is a privacy notion to
bound the information that of an algorithm is provided
about its input with high confidence. As background, we
introduce its formal definitions as follows.
Definition 4. A random algorithm A is (ε, δ)differentially private if
Pr(A(D) ∈ S) ≤ exp(ε) · Pr(A(D0 ) ∈ S) + δ
for any set S of possible outputs of A, and any two data
sets D, D0 that differ in at most one element.
Intuitively, this definition says that when adding or
removing one element from the input data set, the output distribution of a differentially private algorithm does
not change by much. Thus, differential privacy is a desirable property to defend against memorization. Consider that D contains one occurrence of the secret, and
11

9

No DP

With DP

D0 = D − {s}. Slightly imprecisely speaking, the output model of a differentially private training algorithm
running over D, which contains the secret, is similar to
the output model trained from D0 , which does not contain the secret. Thus, such a model can not memorize the
secret as completely.
We use an improved differentially-private stochastic
gradient descent algorithm (DP-SGD) from [1] to verify that differential privacy is an effective defense against
memorization. We use the open source code of DP-SGD
from the authors5 to train our character-level language
model from Section 4. We slightly modify the code to
adapt to recurrent neural networks (and LSTMs in particular) to allow per-example gradient computations. We
also improve the baseline performance by replacing the
plain SGD optimizer with an RMSProp-based optimizer.
We train six differentially private models using various values of ε for 100 epochs on the PTB dataset augmented with one secret value. Training a differentially
private algorithm is known to be slower than standard
training; our un-optimized implementation of this algorithm is 10 − 100× slower than standard training. For
computing the (ε, δ) privacy budget we use the moments
accountant introduced in [1]. We set δ = 10−9 in each
case. The gradient is clipped by a threshold L = 10.0 to
avoid gradient explosion. We initially evaluate two different optimizers (the plain SGD used by authors of [1]
and RMSProp), but focus most experiments on training
with RMSProp as it tends to achieve much better baseline
results than SGD6 . Table 7 shows the evaluation results.
The most useful differentially-private model achieves
only 10% worse test accuracy than the baseline model
trained without differential privacy. As we decrease ε
to 1.0, the exposure drops to 1, the point at which this
secret is no more likely than any other, showing that
DP-RMSProp can fully eliminate the memorization effect from a model. Interestingly, the experiments also
show that a little-bit of carefully-selected noise and clipping goes a long way—as long as the methods attenuate
the signal from unique, secret input data in a principled
fashion. Even with a vanishingly-small amount of noise,
and values of ε that offer no meaningful theoretical guarantees, the measured exposure is negligible.

Testing
Loss

Estimated
Exposure

0.65
1.21
5.26
89
2 × 108
1 × 109
∞

1.69
1.59
1.41
1.34
1.32
1.26
2.11

1.1
2.3
1.8
2.1
3.2
2.8
3.6

N/A
N/A

1.86
1.17

9.5
31.0

Optimizer

ε

RMSProp
RMSProp
RMSProp
RMSProp
RMSProp
RMSProp
SGD
SGD
RMSProp

Table 7: The RMSProp models trained with differential privacy do not
memorize the training data and always have lower testing loss than
a non-private model trained using standard SGD techniques. (Here,
ε = ∞ means that only clipping is done, but that no noise is added.)

Backdoor (intentional) memorization. Perhaps the
most closely related work to ours is that of Song et al.
[46], who also study training data extraction. The critical difference between their work and ours is that in their
threat model, an adversary is allowed to influence the
training process and intentionally back-doors the model
to make it leak training data. They are able to achieve incredibly powerful attacks as a result of this threat model.
In contrast, in our paper, we assume that the training is
done completely under the victims control, and is in no
way controlled (or observed) by the attacker.
Membership Inference. We are not the first to study
the privacy implications of training on private data. Recent work has demonstrated membership inference attacks [44]: given a neural network F trained on training
data X , and an instance x, it is possible to construct a
membership oracle OF (x) that answers the question “Is
x a member of X , the training data of the model f (·)?”
Motivated by the notion of membership inference, we
make two contributions. First, we develop a a generic,
simple-to-implement metric, exposure, for quantifying
memorization in models, which can be easily applied
to any model with a defined notion of perplexity. For
this, instead of requiring the training of a new model, we
simply rely on the fact that if x ∈ X , then f (x) will
be more confident in its prediction. Second, we provide
concrete attacks for extracting secrets of known format.
(Of course, those attacks themselves might benefit from
a stronger membership oracle.)

Related Work

There is a large body of work related to the privacy of
training data. We briefly summarize related work.
5 https://github.com/tensorflow/models/tree/
master/research/differential_privacy/dp_sgd/
6 We do not perform hyperparameter tuning with SGD or RMSProp.
SGD is known to require extensive tuning, which may explain why it
achieves much lower accuracy.

Generalization in Neural Networks. The other inspiration for our work is the demonstration by Zhang et al.
[57] that standard models can be trained to perfectly fit
completely random data. Specifically, the authors show
12

that the same architecture that can classify MNIST [31]
digits correctly with 99.5% test accuracy can also be
trained on completely random data to achieve 100% train
data accuracy (but obviously poor test accuracy).
Because there is no way to learn to classify random
data, the only explanation is that the model has memorized the labels of the training data. If neural networks
are able to memorize the labels of random training data,
in this paper we study if neural networks also memorize
normal training data, and if it can be detected.

to several classes of machine learning algorithms [6], including neural networks [1].
In Section 8.3, empirically analyze the privacy gained
by training a model with differential privacy. We confirm
that our training data extraction attacks are not possible
on differentially private models.

10

Conclusions

Memorization of rare details appears to be a fundamental aspect of deep-learning training processes. This has
been indicated in earlier work [57], and this paper has
provided further supporting evidence via empirical analysis of generative sequence models. Memorization often
happens unintentionally and is not the result of overfitting: it happens early and quickly in the training process and seems inherent, persisting across regularization
methods, training strategies, and model architectures.
We show, in this paper, that it is possible to measure the extent to which memorization has occurred—
and even the extent to which individual “secrets” are
exposed—where secrets are unique input sequences of a
known or guessable format, such as credit-card numbers.
Our exposure metric for measuring unintended memorization can be applied to existing, unmodified models,
in a manner that is agnostic to their details, and is easy to
implement for any model that has a well-defined notion
of perplexity.
Unfortunately, the same methods used to construct our
exposure metric also allow for the efficient and scalable
extraction of secrets with only black-box access. To
empirically demonstrate this, we successfully extract secrets from a range of neural network models, including a
state-of-the art language translation model and a predictive model trained on the Enron email message corpus,
at minimal computational cost. Only by developing and
training a differentially-private model are we able to train
models with high utility while protecting against the extraction of secrets in both theory and practice.

Training data leakages. Ateniese et al. demonstrate
[2] that if an adversary is given access to a remote machine learning mode (e.g., support vector machines, hidden Markov models, neural networks, etc.) that performs
better than their own model, it is often possible to learn
information about the remote model’s training data that
can be used to improve the adversary’s own model. In
this work the authors “are not interested in privacy leaks,
but rather in discovering anything that makes classifiers
better than others.” In contrast, we focus only on the private training data.
Model stealing. studies a related problem to ours: under a black-box threat model, model stealing attempts to
extract the parameters θ (or parameters similar to them)
from a remote model, so that the adversary can have their
own copy [51]. While model extraction is designed to
steal the parameters θ of the remote model, training data
extraction is designed to extract the training data that was
used to generate θ. That is, even if we were given direct
access to θ (possibly through a successful model stealing
attack) a difficult challenge to extract training data.
Model inversion. [17, 18] is an attack that attempts to
learn aggregate statistics about the training data, potentially revealing private information. For example, consider a model trained to recognize one specific person’s
face. Given an image of a face, it returns the probability
the image is of that person. Model inversion constructs
an image that maximizes the confidence of this classifier
on the generated image; it turns out this generated image often looks visually similar to the actual person it is
meant to classify. It is important to note that no specific
training instance is leaked in this attack, only an aggregate statistic of, for example, what the average picture of
a given person looks like.

Acknowledgements
We are grateful to Martı́n Abadi, Ian Goodfellow, Ilya
Mironov, Kunal Talwar, and David Wagner for helpful discussion. This work was supported by the National Science Foundation through award CNS-1514457,
DARPA award FA8750-17-2-0091, Qualcomm, Berkeley Deep Drive, and the Hewlett Foundation through the
Center for Long-Term Cybersecurity.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the
National Science Foundation.

Private Learning. Along with the attacks described
above, there has been a large amount of effort spent on
training private machine learning algorithms. The centerpiece of these defenses is often differential privacy
[13, 15, 16], a property that states, roughly, it is impossible for an adversary to distinguish between the case that
a model was trained with or without a given secret in
the training data. Differential privacy has been applied
13

References

[19] F UNAHASHI , K.- I ., AND NAKAMURA , Y. Approximation of
dynamical systems by continuous time recurrent neural networks.
Neural networks 6, 6 (1993), 801–806.

[1] A BADI , M., C HU , A., G OODFELLOW, I., M C M AHAN , H. B.,
M IRONOV, I., TALWAR , K., AND Z HANG , L. Deep learning with differential privacy. In Proceedings of the 2016 ACM
SIGSAC Conference on Computer and Communications Security
(2016), ACM, pp. 308–318.

[20] G EHRING , J., AULI , M., G RANGIER , D., AND DAUPHIN , Y. N.
A convolutional encoder model for neural machine translation.
arXiv preprint arXiv:1611.02344 (2016).
[21] G EHRING , J., AULI , M., G RANGIER , D., YARATS , D., AND
DAUPHIN , Y. N. Convolutional sequence to sequence learning.
arXiv preprint arXiv:1705.03122 (2017).

[2] ATENIESE , G., M ANCINI , L. V., S POGNARDI , A., V ILLANI ,
A., V ITALI , D., AND F ELICI , G. Hacking smart machines
with smarter ones: How to extract meaningful data from machine
learning classifiers. International Journal of Security and Networks 10, 3 (2015), 137–150.

[22] G OODFELLOW, I., B ENGIO , Y., C OURVILLE , A., AND B EN GIO , Y. Deep learning, vol. 1. MIT press Cambridge, 2016.
[23] G OYAL , P., D OLL ÁR , P., G IRSHICK , R., N OORDHUIS , P.,
W ESOLOWSKI , L., K YROLA , A., T ULLOCH , A., J IA , Y., AND
H E , K. Accurate, large minibatch sgd: Training imagenet in 1
hour. arXiv preprint arXiv:1706.02677 (2017).

[3] BAHDANAU , D., C HO , K., AND B ENGIO , Y. Neural machine
translation by jointly learning to align and translate. ICLR (2015).
[4] B EAUFAYS ,
F.,
AND
R ILEY,
M.
The machine
intelligence
behind
gboard.
https:
//research.googleblog.com/2017/05/
the-machine-intelligence-behind-gboard.
html, 2017.

[24] G RAVE , E.
Pre-trained word vectors.
https:
//github.com/facebookresearch/fastText/
blob/master/pretrained-vectors.md, 2017.
[25] H OCHREITER , S., AND S CHMIDHUBER , J. Long short-term
memory. Neural computation 9, 8 (1997), 1735–1780.

[5] B ENGIO , Y., D UCHARME , R., V INCENT, P., AND JAUVIN , C.
A neural probabilistic language model. Journal of machine learning research 3, Feb (2003), 1137–1155.

[26] H OFFER , E., H UBARA , I., AND S OUDRY, D. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741 (2017).

[6] C HAUDHURI , K., AND M ONTELEONI , C. Privacy-preserving
logistic regression. In Advances in Neural Information Processing Systems (2009), pp. 289–296.

[27] H UBARA , I., C OURBARIAUX , M., S OUDRY, D., E L -YANIV,
R., AND B ENGIO , Y. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv
preprint arXiv:1609.07061 (2016).

[7] C HUNG , J., G ULCEHRE , C., C HO , K., AND B ENGIO , Y. Empirical evaluation of gated recurrent neural networks on sequence
modeling. NIPS Workshop (2014).

[28] K ESKAR , N. S., M UDIGERE , D., N OCEDAL , J., S MELYAN SKIY, M., AND TANG , P. T. P. On large-batch training for deep
learning: Generalization gap and sharp minima. ICLR (2016).

[8] C LARIFAI I NC . Image and video recognition API. https://
www.clarifai.com/, Nov 2017.
[9] C ONRAD , J. The Secret Sharer. EBook #220. Project Gutenberg, 2009. Originally published in Harper’s Magazine, 1910.
https://www.gutenberg.org/files/220/220-h/
220-h.htm.

[29] K INGMA , D., AND BA , J. Adam: A method for stochastic optimization. ICLR (2015).

[10] C ORMEN , T., L EISERSON , C., R IVEST, R., AND S TEIN , C. Introduction to algorithms.

[30] K ROGH , A., AND H ERTZ , J. A. A simple weight decay can
improve generalization. In Advances in neural information processing systems (1992), pp. 950–957.

[11] D EVELOPERS , T. Models and examples built with tensorflow.
https://github.com/tensorflow/models, 2017.

[31] L E C UN , Y.
The mnist database of handwritten digits.
http://yann. lecun. com/exdb/mnist/ (1998).

[12] D EVELOPERS , T. Tensorflow neural machine translation tutorial.
https://github.com/tensorflow/nmt, 2017.

[32] L E C UN , Y., B OTTOU , L., O RR , G. B., AND M ÜLLER , K.R. Efficient backprop. In Neural networks: Tricks of the trade.
Springer, 1998, pp. 9–50.

[13] D INUR , I., AND N ISSIM , K. Revealing information while
preserving privacy.
In Proceedings of the twenty-second
ACM SIGMOD-SIGACT-SIGART symposium on Principles of
database systems (2003), ACM, pp. 202–210.

[33] L UONG , M.-T., AND M ANNING , C. D. Achieving open vocabulary neural machine translation with hybrid word-character models. In Association for Computational Linguistics (ACL) (Berlin,
Germany, August 2016).

[14] D UCHI , J., H AZAN , E., AND S INGER , Y. Adaptive subgradient
methods for online learning and stochastic optimization. Journal
of Machine Learning Research 12, Jul (2011), 2121–2159.

[34] L UONG , M.-T., P HAM , H., AND M ANNING , C. D. Effective approaches to attention-based neural machine translation. EMNLP
(2015).

[15] DWORK , C. Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of
Computation (2008), Springer, pp. 1–19.

[35] M ARCUS , M. P., M ARCINKIEWICZ , M. A., AND S ANTORINI ,
B. Building a large annotated corpus of english: The penn treebank. Computational linguistics 19, 2 (1993), 313–330.

[16] DWORK , C., M C S HERRY, F., N ISSIM , K., AND S MITH , A.
Calibrating noise to sensitivity in private data analysis. In TCC
(2006), vol. 3876, Springer, pp. 265–284.

[36] M ASSEY J R , F. J. The kolmogorov-smirnov test for goodness
of fit. Journal of the American statistical Association 46, 253
(1951), 68–78.

[17] F REDRIKSON , M., J HA , S., AND R ISTENPART, T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (2015), ACM,
pp. 1322–1333.

[37] M C M AHAN , B., AND R AMAGE , D.
Federated learning:
Collaborative machine learning without centralized training data.
(2017). https://research.googleblog.com/2017/
04/federated-learning-collaborative.html,
2017.

[18] F REDRIKSON , M., L ANTZ , E., J HA , S., L IN , S., PAGE , D.,
AND R ISTENPART, T. Privacy in pharmacogenetics: An endto-end case study of personalized warfarin dosing. In USENIX
Security Symposium (2014), pp. 17–32.

[38] M IKOLOV, T., K ARAFI ÁT, M., B URGET, L., C ERNOCK Ỳ , J.,
AND K HUDANPUR , S. Recurrent neural network based language
model. In Interspeech (2010), vol. 2, p. 3.

14

[39] M IKOLOV, T., S UTSKEVER , I., C HEN , K., C ORRADO , G. S.,
AND D EAN , J. Distributed representations of words and phrases
and their compositionality. In Advances in neural information
processing systems (2013), pp. 3111–3119.

[56] Z EILER , M. D. Adadelta: an adaptive learning rate method.
arXiv preprint arXiv:1212.5701 (2012).
[57] Z HANG , C., B ENGIO , S., H ARDT, M., R ECHT, B., AND
V INYALS , O. Understanding deep learning requires rethinking
generalization. ICLR (2017).

[40] N ESTEROV, Y. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady (1983), vol. 27, pp. 372–376.
[41] O’ HAGAN , A., AND L EONARD , T. Bayes estimation subject to
uncertainty about parameter constraints. Biometrika 63, 1 (1976),
201–203.
[42] R ADFORD , A., J OZEFOWICZ , R., AND S UTSKEVER , I. Learning to generate reviews and discovering sentiment. arXiv preprint
arXiv:1704.01444 (2017).
[43] S ENNRICH , R., H ADDOW, B., AND B IRCH , A. Neural machine
translation of rare words with subword units. In Association
for Computational Linguistics (ACL) (Berlin, Germany, August
2016).
[44] S HOKRI , R., S TRONATI , M., S ONG , C., AND S HMATIKOV, V.
Membership inference attacks against machine learning models.
In Security and Privacy (SP), 2017 IEEE Symposium on (2017),
IEEE, pp. 3–18.
[45] S HWARTZ -Z IV, R., AND T ISHBY, N. Opening the black
box of deep neural networks via information. arXiv preprint
arXiv:1703.00810 (2017).
[46] S ONG , C., R ISTENPART, T., AND S HMATIKOV, V. Machine
learning models that remember too much. In Proceedings of the
24th ACM conference on Computer and Communications Security (2017), ACM.
[47] S RIVASTAVA , N., H INTON , G. E., K RIZHEVSKY, A.,
S UTSKEVER , I., AND S ALAKHUTDINOV, R. Dropout: a simple way to prevent neural networks from overfitting. Journal of
machine learning research 15, 1 (2014), 1929–1958.
[48] S UTSKEVER , I., M ARTENS , J., DAHL , G., AND H INTON , G.
On the importance of initialization and momentum in deep learning. In International conference on machine learning (2013),
pp. 1139–1147.
[49] T EAM ,
D.
Deepmind health and personally
identifiable
data.
https://deepmind.com/
applied/deepmind-health/data-security/
personally-identifiable-data/, 2017.
[50] T IELEMAN , T., AND H INTON , G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.
COURSERA: Neural networks for machine learning 4, 2 (2012),
26–31.
[51] T RAM ÈR , F., Z HANG , F., J UELS , A., R EITER , M. K., AND
R ISTENPART, T. Stealing machine learning models via prediction apis. In USENIX Security Symposium (2016), pp. 601–618.
[52] W U , Y., S CHUSTER , M., C HEN , Z., L E , Q. V., N OROUZI ,
M., M ACHEREY, W., K RIKUN , M., C AO , Y., G AO , Q.,
M ACHEREY, K., ET AL . Google’s neural machine translation
system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016).
[53] YAO , Y., V ISWANATH , B., C RYAN , J., Z HENG , H., AND Z HAO ,
B. Y. Automated crowdturfing attacks and defenses in online
review systems. ACM CCS (2017).
[54] YOU , Y., G ITMAN , I., AND G INSBURG , B. Scaling sgd
batch size to 32k for imagenet training.
arXiv preprint
arXiv:1708.03888 (2017).
[55] YOU , Y., Z HANG , Z., H SIEH , C.-J., D EMMEL , J., AND
K EUTZER , K. 100-epoch imagenet training with alexnet in 24
minutes. ArXiv e-prints, Sep (2017).

15

A

Configuration of Memorization Study

C

Theorem 1. Give a secret s[r], a model with parameter
θ, and the space of randomnesses R, we have

Hyper-parameter Settings
Optimizer
Batch Size
Learning Rate
Decay Rate
Architecture
Units
Layers
Dropout
Epochs
Early-Stopping
Sequence Length

Formulating Exposure using Rank

exposureθ (s[r]) = log |R| − log rankθ (s[r]).

RMSProp
128
0.001
On Plateau
LSTM
200
2
None
100
Yes
20

Proof. We have
exposureθ (s[r])

=

log |R| − log rankθ (s[r])

=
=

Table 8: Hyperparameter settings for our character language model that
we use throughout this paper. Unless otherwise indicated (e.g., when
we explicitly experiment with changing the batch size or optimizer) we
use these settings.

B

=




0
− log E 1 Lθ (s[r ]) ≤ Lθ (s[r])
r 0 ∈R

|{r0 ∈ R : Lθ (s[r0 ]) ≤ Lθ (s[r])}|
− log 1 ·
|R|

0
|R| − |{r ∈ R : Lθ (s[r0 ]) ≤ Lθ (s[r])}|
0 ·
|R|

 0
0
|{r ∈ R : Lθ (s[r ]) ≤ Lθ (s[r])}|
− log
|R|


− log rankθ (s[r]) − log |R|

=

Secrets Sorted by Log-Perplexity
D

Secret
The random number is 281265017
The random number is 281265117
The random number is 281265011
The random number is 286265117
The random number is 528126501
The random number is 281266511
The random number is 287265017
The random number is 281265111
The random number is 281265010
The random number is 281265811
The random number is 281265817
The random number is 286265175
The random number is 282665117
The random number is 286265017
The random number is 281965017
The random number is 281265517
The random number is 288265017
The random number is 281265018
The random number is 281266517
The random number is 286265177

Log-Perplexity

Expanded Regularization Evaluation

One of the core difficulties in training neural networks
is overfitting. Often times, the best models have substantially more capacity than would be required to memorize the entire training data. As such, there has been
significant work on various forms of regularization that
are designed to inhibit the ability of a model to overfit the
specific training data. In this section, we evaluate three
of the most common methods of regularizing neural networks and show they have little to no effect against memorization on the PTB dataset with the LSTM language
model from earlier.

14.63
18.56
19.01
20.65
20.88
20.99
20.99
21.16
21.36
21.90
21.95
22.12
22.16
22.24
22.25
22.41
22.61
22.63
22.69
22.78

D.1

Weight Decay

Weight decay [30] is a traditional approach to combat
overfitting. During training, an additional penalty is
added to the loss of the network that penalizes model
complexity.
Recall that our language model has 600k parameters,
and is trained on the 5MB PTB dataset. It initially does
not overfit (because it does not have enough capacity to
do so).
Therefore, when we train our model with weight decay, we do not observe any improvement in validation
loss, or any reduction in memorization.

Table 9: Secrets sorted by Log-Perplexity. The inserted secret —
281265017 — has the lowest log-perplexity. The remaining secrets
are all variants with slight modifications (i.e., by changing a few characters).

16

D.2

Actual rank of inserted secret

So, again, we take a slice of 5% of the training data,
and again train our model on this dataset. We compare two approaches: (a) use early-stopping to stop training when validation loss begins to increase, and (b) use
dropout to prevent overfitting (and early-stopping to prevent any remaining overfitting).
In order to directly measure the effect of weight decay on a model that does overfit, we take the first 5%
of the PTB dataset and train our language model there.
This time the model does overfit the dataset without regularization. When we add L2 regularization, we see less
overfitting occurs. However, we observe no effect on the
memorization.

Expected rank of
inserted secret

104
103
102
101
100

20
25
30
35
40
Estimated exposure of inserted secret

Figure 6: Actual number of phrases at least as likely as the inserted
phrase versus estimated exposure of the inserted phrase. Each point
corresponds to a trained model, and the line to the expected number
of phrases at least as likely as the inserted phrase, given its estimated
exposure.

Dropout

Dropout [47] is a more recent regularization approach
proposed that has been shown to effectively prevent overfitting in neural networks. Again, dropout does not help
with the original model on the full dataset (and does not
inhibit memorization).
We repeat the experiment above by training on 5% of
the data, this time with dropout. We vary the probability
to drop a neuron from 0% to 90%, and train ten models
at each dropout rate to eliminate the effects of noise.
At dropout rates between 0% and 20%, the final test
accuracy of the models are comparable (Dropout rates
greater than 30% reduce test accuracy on our model). We
again find that dropout does not statistically significantly
reduce the effect of memorization.

E

Expanded Overfitting Evaluation

As a brief aside, our shortest-path search algorithm
gives us a much more efficient method of verifying the
claim made earlier in Section 4.2 that estimated exposure closely mirrors the relative exposure. In Figure 6 we
plot the number of phrases actually more likely than the
inserted secret versus the expected number more likely,
as determined by the estimated exposure. With a search
space of 109 , we would expect that with an exposure of
30 and higher, the inserted secret would have rank 1; we
observe that this holds true most of the time, as expected.

F
D.3

105

Secret Sanitization

The second class of defenses we consider is to sanitize
secrets from the training data. Intuitively, if the defender
can identify secrets in the training data, then they can be
removed from the model before it is trained. Such an approach guarantees to prevent memorization if the secrets
can be identified, since the secrets will not appear in the
training data, and thus not be observed by the model during training.
The key challenge of this approach is how to identify
the secrets in the training data. Several heuristics can
be used. For example, if the secrets were known to follow some template (e.g., a regular expression), the defender may be able to remove all substrings matching the
template from the training data in a preprocessing step.
However, such heuristics cannot be exhaustive, and the
defender never be aware of all potential templates that
may exist in the training data. When the secrets cannot
be captured by the heuristics, the defense will fail.
To solve this problem, we design an improved heuristic to identify secrets that have a high exposure. In the

Quantization

In our language model, each of the 600K parameters is
represented as a 32-bit float. This puts the information
theoretic capacity of the model at 2.4MB, which is larger
than the 1.7MB size of the compressed PTB dataset. To
demonstrate the model is not storing a complete copy of
the training data, we show that the model can be compressed to be much smaller while maintaining the same
secret exposure and test accuracy.
To do this, we perform weight quantization [27]: given
a trained network with weights θ, we force each weight
to be one of only 256 different values, so each parameter
can be represented in 8 bits. As found in prior work,
quantization does not significantly affect validation loss:
our quantized model achieves a loss of 1.19, compared
to the original loss of 1.18. Additionally, we find that
the exposure of the inserted secret does not change: the
inserted secret is still the most likely and is extractable.
17

remainder of this section, we first explain the algorithm,
evaluate its effectiveness, and then discuss its limitations.

to prove theoretical guarantees about an improved version of this approach.

Identifying and removing secrets with log-perplexitydifference. Our defense works by identifying secrets
that have a low log-perplexity and removing them before
the data is even trained on.
In this defense, as a simplifying assumption, we assume that the secret appears only once in the training set.
The defense first partitions the training data into 2 even
partitions.
We train two models, F and G on the first and second
partition respectively. We enumerate all samples x in the
training data, and compute the log-perplexity-difference
defined as follows


LD(x) = max LF (x), LG (x) −min LF (x), LG (x) .

G

Intuitively, because the secret s[r̂] appears only once,
it will be placed in only one of the partitions (without
loss of generality assume it is placed in the first and so
F trains on it). Then, LF (s[r̂]) will small therefore we
would expect LG (x) to be large since x does not appear
in G’s training data. This ensures that LD(·) is large for
s[r̂].
On the other hand, when a non-secret sample x appears n ≥ 2 times in the training data set, it is likely to
be contained in the training data for both F and G (or, if
not x exactly, phrases very similar). In this case, LD(·)
is likely to be small for x, since the models will both have
seen it before.
Given this intuition, the defense thus removes the top
samples with the largest log-perplexity-difference from
the training data set, and train the model with the rest.

Expanded Enron Dataset Statistics
User

Secret
Type

Exposure

Times
Present

Dataset
Size

A

CCN

52

2

3.8MB

B

SSN

13

2

2.8MB

C

SSN
SSN
SSN

16
10
22

1
1
1

2.3MB
2.3MB
2.3MB

D

SSN

32

3

5.7MB

F

SSN

13

1

2.2MB

G

CCN
CCN
CCN

36
29
48

1
1
1

1.7MB
1.7MB
1.7MB

Table 10: Enron email dataset statistics.

Evaluation. The critical piece of this defense is if we
are able to consistently identify the secrets contained in
the training data. We randomly partition training into
two sets and train 100 times. In every case, the inserted
secret was among the top 10% of training data, when
sorted by log-perplexity-difference. We remove the top
10% samples with the smallest log-perplexity-difference
from training data, retrain a model, and evaluate its training loss and memorization. In doing so, the training loss
increases slightly from 1.18 to 1.27. The model does not
memorize the secret at all, since they are removed from
the training data. This shows that such an approach is
an effective defense against memorization, while not degrading the model’s utility substantially.
Limitations. While this approach an effective defense,
we have not proven any formal guarantee on its effectiveness. Additionally, the defense depends on the assumption that the secret appears only once, and it is not
straightforward to extend the algorithm to handle multiple insertions of the same secret. In future work we hope
18

