arXiv:1509.05982v2 [cs.NE] 22 Sep 2015

Denoising without access to clean data
using a partitioned autoencoder
Dan Stowell
Centre for Digital Music
School of Elec. Eng. and Computer Science
Queen Mary University of London
Richard E. Turner
Computational and Biological Learning Lab
Department of Engineering
University of Cambridge
September 24, 2015
Abstract
Training a denoising autoencoder neural network requires access to
truly clean data, a requirement which is often impractical. To remedy
this, we introduce a method to train an autoencoder using only noisy
data, having examples with and without the signal class of interest. The
autoencoder learns a partitioned representation of signal and noise, learning to reconstruct each separately. We illustrate the method by denoising
birdsong audio (available abundantly in uncontrolled noisy datasets) using
a convolutional autoencoder.

1

Introduction

An autoencoder (AE) is a neural network trained in unsupervised fashion, to
encode its input to some latent representation and to decode that representation
to a faithful reconstruction of its input. The autoencoder can then be used as a
codec, or to convert data to its latent representation for downstream processing
such as classification. The denoising autoencoder (DAE) is a variant of this in
which the inputs are combined with some corruption (such as additive noise
or masking), and the system is trained to recover the clean, de-noised data
[1]. The DAE training scheme can be used in denoising applications, and is
also a popular way to encourage the autoencoder to learn a more meaningful
latent representation of the data. Autoencoders including the DAE have yielded
leading results in recent years in deep learning for signal processing [1, 2].
However, there is a significant problem with the DAE approach which hampers its use in practical applications: it may often be impossible to supply
1

truly clean data. This is common in our application example—natural sound
recordings—but also for video, image and audio applications across many domains. In fact, objects/events are often sparsely represented in data while background and other noise are densely represented, meaning that it is often easy to
provide “noise-only” examples while difficult to provide “noise-free” examples.
In this paper we propose an alternative approach to train an AE so that it
can perform denoising, given training data which can only be weakly labelled
as “noise-only” or “noise and possible signal”. The system learns a partitioned
latent representation, with identifiable noise and signal coefficients, and can
then perform denoising and/or recover a signal-only latent representation for
further analysis. The method is general-purpose; we illustrate it here with an
application to denoising birdsong audio spectrograms.

2

Partitioned autoencoders

A standard AE learns a function of the form X̂ = g(f (X)) where X is an input
datum (a matrix, in this paper), and the autoencoder is composed of encoder
f (·) and decoder g(·). The DAE learns a function X̂ = g(f (u(X))) where u(·)
is a stochastic noise corruption process. The training objective
is such that
P
X̂ is encouraged to be as close to X as possible, often i kXi − X̂i k2 where
|| · || is the Frobenius norm and the sum is taken over a minibatch of training
data. (A minibatch is a small subset of training data used for one iteration of
stochastic gradient descent [SGD].) Given this objective, in practice a DAE will
learn to denoise its input to the extent that X itself is clean, having no incentive
to “overshoot” and remove any noise that may be intrinsic to the original X.
The latent representation output from f (·) parametrises the manifold on which
the reconstructed signal data lie. Information about the noise present in each
datum is not captured, except implicitly as X − X̂ if X̂ is a good estimate.
Many equivalent parametrisations of the manifold may be possible, some allowing a more semantic interpretation of each latent coefficient than others, and
the standard AE or DAE does not distinguish among these parametrisations.
There has been recent interest in adapting the training schemes of autoencoders
such that the latent representation is explicitly semantic, capturing attributes
of the input in specific subsets of the latent variables [3, 4]. We refer to these as
partitioned autoencoders since the latent variables are partitioned into subsets
which are treated differently from each other during training. Crucially, in this
prior work the training scheme relies heavily on the existence of large structured
datasets: in [4] a balanced dataset of labelled digit images; in [3] a dataset of
faces constructed through systematic variation of attributes such as pose and
lighting. Without such known structure in the training data, their proposed
training schemes will be either impossible to apply, or biased by the presence of
unbalanced or correlated factors in the training data.
Our present motivation is to learn a denoising representation, trained using
data which does not contain truly clean examples. If we can develop a scheme
that learns to represent both signal and noise, but partitioning them into sep-

2

arate latent coefficients, then we will be able to perform denoising or further
analysis by using only the “foreground” (signal) coefficients and setting the remainder to zero. The scheme of [3] would be appropriate if the SNR of each
training example were known and systematically varied, but in uncontrolled
datasets this information is rarely available. Instead we propose a scheme based
on the observation that many scenarios consist of sparsely-present foreground
and densely-present background, and so noise-only training data is much easier
to come by than signal-only.
Our training scheme is based on a standard autoencoder reconstruction objective, augmented with a structured regularisation of the latent variables. In
order to encourage the model to use the background latents to represent noise
and never to represent signal, we add a soft regulariser that penalises foreground
latent activation for the noise-only examples. For each training example X associated with a weak label y taking the value 1 if the example is a “noise-only”
example and 0 otherwise, we train an autoencoder by minimising the following
loss function:
λy
kC f (X)k2
l(X, y) = kX − X̂k2 +
C̄
where X̂ = g(f (X)), λ is a regularisation coefficient, represents elementwise
multiplication, C is a masking matrix containing 1 for latents which should
represent foreground and 0 otherwise, and C̄ the mean value of C. We will
construct our training minibatches with a fixed proportion of noise-only items
at each iteration. The value used for λ will be relatively large, to impose a soft
constraint pushing a subset of the latent values to zero in the case of noise-only
items (Figure 1, Figure 2). This encourages the learned representation to use
the non-regularised latents for the foreground signal. The parameters of f (·)
and g(·) will be optimised through SGD. Once trained, denoising is achieved by
reconstructing using only the foreground latents, i.e. setting the others to zero
(Figure 3).
Figure 2 emphasises that the proportion of latents dedicated to foreground
vs. background, and the balance of “signal-plus-noise” and “noise-only” examples in a minibatch, are independent configuration choices. We tend to reserve
25% of latents for background, and 25% of a minibatch as noise-only; in our
evaluation we will evaluate the impact of varying the balance of latents.
Note that the regularisation scheme is asymmetric: it encourages some latents to zero in certain cases, but it does not prevent them from being zero in
the other cases. In principle the system could simply never use those latents;
however the reconstruction cost encourages the system to use them to improve
reconstruction in cases where it has that freedom. The asymmetry also means
that the important aspect of data labelling is to identify “noise-only” items with
high precision. If some “noise-only” items are not labelled as such, the system
is free to represent them without making use of those latents. The scheme can
thus be used on datasets which are too large to label completely, but in which
a subset of noise-only examples can be identified.

3

Input minibatch

Latent representation

"Signal"
inputs (inc.
some noise)

Encode

Reconstruction

Decode

"Noise only"
inputs

Strongly regularised

Figure 1: Our training scheme aims to reconstruct all the minibatch items, while
regularising some of the latents.

3

Convolutional partitioned autoencoder for audio spectrograms

The training scheme can be applied to various autoencoder architectures. Convolutional neural networks have recently proven to be powerful for many tasks,
while relatively easy to train because they have far fewer free parameters than
an equivalent fully-connected network [2, 5]. In our application we aim to extract information from audio spectrograms, indexed by time and frequency. We
will thus use an autoencoder which is convolutional in time and fully-connected
in frequency, as is standard for recent neural network audio analysis [5]. Given
an input matrix X indexed by discrete time n and frequency h, we define our
encoding function to be
f (X) = mp(r(Wc ? (X − µ̂)/σ̂))
where Wc is a tensor of coding weights indexed by time m frequency h and
latent index k, r(·) is a rectified linear unit nonlinearity, mp(·) represents the
max-pooling operation applied along the time axis, µ̂ and σ̂ are the frequencywise means and standard deviations estimated from the training set and used
for normalisation, and ? indicates one-dimensional convolution as follows:
(A ? B)n,k =

M X
H
X

am,h,k bn−m,h

n ∈ [1, N ], k ∈ [1, K]

m=1 h=1

resulting in a matrix indexed by time n and latent index k.
Our decoding function is
g(f (X)) = Wd ? mp−1 (f (X))
4

"Noise only" examples

"Signal+noise" examples

Index within minibatch

Hidden layer node index

"Foreground" latents

"Background" latents

Figure 2: Regularisation is applied to the subset of latents intended to represent
signal, and only for noise-only examples.
where Wd is a tensor of decoding weights, and mp−1 (·) is the inverse of the
max-pooling operation (the approximate inverse, as the non-maximal values
are reconstituted by zeros).
In this temporally convolutional architecture, the mask matrix C is implemented as identical frame-wise mask matrices, with k indexing the set of latent
time series.
Our data will be non-negative. Data normalisation at the input is important
for effective training, hence the use of µ̂ and σ̂ [6]. However we do not undo the
normalisation at the decoder outputs: the non-negative target and the rectifier
then give the property that the decoder learns a non-negative parts-based reconstruction. We do not use bias units, as we found the resulting system easier
to train (cf. [7, 8]).
For the evaluation that follows, fixed configuration details are: input spectrograms have 512 time frames and 32 frequency bins, and we use 32 latent
variables. Convolution filters have a length of 9 time frames. Our max-pooling
downsamples the time axis by a factor of 16. We train the network using
AdaDelta to control the SGD learning rates [9]. We do not use dropout. We
initialise the tensor of filters as a set of K random orthogonal unit vectors of
length M H, reshaping this to the tensor of shape M × H × K (cf. [10]).
In this study we explore only a single-layer autoencoder. Our approach
applies straightforwardly to a deep autoencoder, and a deeper system would be
expected to have a broader ability to generalise.

4

Evaluation

We test our approach using a task to denoise birdsong audio spectrograms (Figure 4). As foreground we use recordings of chiff chaff (similarly to [11, 12]).
For evaluation purposes we wish to add a background that offers a substantial
test: it should be diverse, nonstationary and contain significant energy in the
5

Input minibatch

"Signal"
inputs (inc.
some noise)

Latent representation

Reconstruction

Denoised
Encode

Decode

Denoised
(should be
silent)

"Noise only"
inputs

Figure 3: Once trained, the system can denoise by reconstructing using only a
subset of the latents (setting others to zero).
same frequency band as the birdsong. After considering many available options
we settled on recorded restaurant noise, which contains multi-speaker human
speech as well as diverse percussive sound events. We add this background
noise, both to create a known amount of “intrinsic” noise in the examples, and
to create “noise-only” examples.
Our system is implemented in the Theano framework [13] making use of
GPU processing. To create a diverse training/validation dataset within the constraints of limited GPU memory, we take advantage of the approximate additive
nature of audio spectrograms as follows. In each experiment we load 30 seconds
of signal, of intrinsic noise and extrinsic noise, and store their spectrograms to
the GPU. Then to generate each “signal-plus-noise” datum we randomly sample
1.5 second segments from each of the three sources and mix them. To generate a
“noise only” datum we sample only from the extrinsic noise source. This creates
a large generative dataset within limited memory. We also experimented with
replacing the 30 second source material regularly throughout training but this
made little difference, so we do not employ that in the present results.
Our audio sources use sample rate 22.05 kHz, analysed using 128 bin FFTs
with 50% hop, and the frequency axis then reduced to 32 bins of interest for the
birdsong (1.7–7.2 kHz). Within this frequency band, we added intrinsic noise
to give an SNR -10 dB, then extrinsic noise to give an SNR of -30 dB. The
extrinsic noise corresponds to the “noise-only” items that would be provided in
practice. The intrinsic noise is added for evaluation only, to judge whether the
system can remove it (Figure 5). We study two cases where the intrinsic and
extrinsic noise sources are matched or unmatched.
We test our partitioned AE at various settings for the proportion of latents
regularised, and also a standard DAE using the same training data, configured
to use the extrinsic noise as the additive corruption u(·) in the DAE training
process. In all cases we fix λ = 0.75 and train with minibatches of size 16, and

6

An input [1.00519e-06, 0.000775318, 14.3777] [mean 0.0131096]
30
25
20
15
10
5
30
0
25 0
30

latest (iter 2000000) reco_fg [-3.74982e-08, 8.33592e-05, 0.99913] [mean 0.00253828]
latest (iter 2000000) reco_fg [-3.74982e-08, 8.33592e-05, 0.99913] [mean 0.00253828]
100

200

300

400

500

Reconstructed [-3.05402e-05, 0.000793635, 14.3877] [mean 0.012494] (mb err 0.000217023)

20
25
30
15
20
25
10
15
20
5
10
15
0
105 0
05
30 0
0
25 0
30

100

200

300

400

latest (iter 2000000) reco_bg [-5.28898e-05, 0.000168182, 14.3877] [mean 0.00995565]
100

200

300

400

latest (iter 2000000) reco_bg [-5.28898e-05, 0.000168182, 14.3877] [mean 0.00995565]

500
500

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

20
25
15
20
10
15
105
05
0

Figure 4: Example of source separation with the proposed system. Upper: input
spectrogram, 1.5 seconds (birdsong, plus background noise including a notable
loud noise near the end). Middle: reconstructed signal spectrogram. Lower:
reconstructed noise spectrogram.
106 iterations of SGD.
Results evaluated on separately-sampled validation data (Figure 6) show a
number of interesting properties. Firstly, all methods perform best against their
explicit objective in every case: i.e. their best SNR is the one relating to their
objective function (simple reconstruction for the proposed system, reconstruction of the partly-clean input for the DAE). However, the statistic of interest
is reconstruction of the truly-clean spectrogram. In matched conditions (upper
plot), the proposed system strongly outperforms the DAE on this measure. Its
strong performance is stable across a broad range of settings for the proportion of latents regularised (all except the extremes). The poor results at the
setting with regularisation of all latents (100%) confirm that the regularisation
used is strong: if applied to all latents, it causes underfitting. It is not merely
the presence of regularisation that improves the results, but the partitioning
scheme.
In unmatched conditions (lower plot) the strong performance is not sustained. The DAE continues to perform well at its partial-denoising task, but
our proposed DAE fails to generalise across diverse background recordings. It
seems likely that this is due to the small size and depth of the current setup:
a single-layer autoencoder with only 32 convolutional filters has limited ability
to approximate arbitrary functions. The strong performance in matched conditions suggests further study of the method with deeper networks. However,
note that matched conditions are common in applications such as ours, where

7

Evaluate SNR
Clean
signal

Input minibatch

"Signal"
inputs (inc.
some noise)
Added
"intrinsic"
noise

Latent representation

Reconstruction

Denoised
Encode

Decode

Denoised
(should be
silent)

"Noise only"
inputs
Background
noise

Figure 5: For evaluation only (not for application), we construct our “signal+noise” examples from clean signal plus added intrinsic noise, so that we
can evaluate SNR against the truly clean signal which the system never accesses.
the noise-only examples can be taken from the main field recording sessions.

5

Conclusions

We have introduced a partitioned autoencoder that can learn to denoise data
with better fidelity than a standard DAE, in the common practical case where
no noise-free examples are available for training. In matched conditions this
partitioned scheme makes better use of the available data than a DAE. Unlike a
DAE, our partitioned autoencoder learns to represent the signal and the noise
content, and can reconstruct either or both. This may be part of its advantage
over a standard DAE, which does not learn to represent the noise content.
Further work will explore deeper/wider architectures to improve the generality,
and use the representations to support classification and other tasks.

6

Acknowledgments

We gratefully thank Pavel Linhart for recording the chiff chaff bird sounds used
in this study.
DS is supported by EPSRC Fellowship EP/L020505/1.

8

Reconstruction SNRs for autoencoder

25
20
15

SNR (dB)

10
5
0
5
10
15
20
0%
25

Denoising to partly-clean
Denoising to truly-clean
Simple reconstruction
25%

50%
Reconstruction
SNRs75%
for autoencoder100%
Proportion of latents regularised, or DAE

DAE

20
15

SNR (dB)

10
5
0
5
10
15
20

Denoising to partly-clean
Denoising to truly-clean
Simple reconstruction
75%
Proportion of latents regularised, or DAE

DAE

Figure 6: Reconstruction SNRs measured on the validation data, for the partitioned AE or DAE. Intrinsic and extrinsic noise may be matched (upper plot)
or unmatched (lower plot). Experiments are performed separately with three
different noise source recordings, and the error bars show the range across three
experiments. The large error bars are due to individual differences and not
random variation (inspected by eye).
9

References
[1] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol,
“Stacked denoising autoencoders: Learning useful representations in a deep
network with a local denoising criterion,” The Journal of Machine Learning
Research, vol. 11, pp. 3371–3408, 2010.
[2] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
no. 7553, pp. 436–444, 2015.
[3] T. D. Kulkarni, W. Whitney, P. Kohli, and J. B. Tenenbaum, “Deep
convolutional inverse graphics network,” arXiv preprint arXiv:1503.03167,
2015.
[4] B. Cheung, J. A. Livezey, A. K. Bansal, and B. A. Olshausen, “Discovering hidden factors of variation in deep networks,” arXiv preprint
arXiv:1412.6583, 2014.
[5] S. Dieleman and B. Schrauwen, “End-to-end learning for music audio,” in
Proc ICASSP, 2014, pp. 6964–6968.
[6] G. Montavon, G. Orr, and K.-R. Müller, Eds., Neural Networks: Tricks of
the Trade, Springer, 2012.
[7] T. L. Paine, P. Khorrami, W. Han, and T. S. Huang, “An analysis of
unsupervised pre-training in light of recent advances,” arXiv preprint
arXiv:1412.6597, 2014.
[8] R. Memisevic, K. Konda, and D. Krueger, “Zero-bias autoencoders and the
benefits of co-adapting features,” arXiv preprint arXiv:1402.3337, 2014.
[9] M. D. Zeiler, “Adadelta: an adaptive learning rate method,” arXiv preprint
arXiv:1212.5701, 2012.
[10] A. M. Saxe, J. L. McClelland, and S. Ganguli, “Exact solutions to the
nonlinear dynamics of learning in deep linear neural networks,” arXiv
preprint arXiv:1312.6120, 2013.
[11] D. Stowell and M. D. Plumbley, “Segregating event streams and noise with
a Markov renewal process model,” Journal of Machine Learning Research,
vol. 14, pp. 1891–1916, 2013, preprint arXiv:1211.2972.
[12] D. Stowell, S. Muševič, J. Bonada, and M. D. Plumbley, “Improved multiple birdsong tracking with distribution derivative method and Markov
renewal process clustering,” in Proceedings of the International Conference on Audio and Acoustic Signal Processing (ICASSP), 2013, preprint
arXiv:1302.3642.
[13] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. J. Goodfellow, A. Bergeron, N. Bouchard, and Y. Bengio, “Theano: new features and speed
improvements,” in Proceedings of NIPS 2012, 2012.

10

