Lecture Notes on Spectral Graph Methods

arXiv:1608.04845v1 [cs.DS] 17 Aug 2016

Michael W. Mahoney∗

∗
International Computer Science Institute and Department of Statistics, University of California at Berkeley,
Berkeley, CA. E-mail: mmahoney@stat.berkeley.edu.

Abstract
These are lecture notes that are based on the lectures from a class I taught on the topic of
Spectral Graph Methods at UC Berkeley during the Spring 2015 semester.
Spectral graph techniques are remarkably broad, they are widely-applicable and often very
useful, and they also come with a rich underlying theory, some of which provides a very good
guide to practice. Spectral graph theory is often described as the area that studies properties
of graphs by studying properties of eigenvalues and eigenvectors of matrices associated with
the graph. While that is true, especially of “classical” spectral graph theory, many of the
most interesting recent developments in the area go well beyond that and instead involve earlystopped (as well as asymptotic) random walks and diffusions, metric embeddings, optimization
and relaxations, local and locally-biased algorithms, statistical and inferential considerations,
etc. Indeed, one of the things that makes spectral graph methods so interesting and challenging
is that researchers from different areas, e.g., computer scientists, statisticians, machine learners,
and applied mathematicians (not to mention people who actually use these techniques), all come
to the topic from very different perspectives. This leads them to parameterize problems quite
differently, to formulate basic problems as well as variants and extensions of basic problems
in very different ways, and to think that very different things are “obvious” or “natural” or
“trivial.” These lectures will provide an overview of the theory of spectral graph methods,
including many of the more recent developments in the area, with an emphasis on some of these
complementary perspectives, and with an emphasis on those methods that are useful in practice.
I have drawn liberally from the lectures, notes, and papers of others, often without detailed
attribution in each lecture. Here are the sources upon which I drew most heavily, in rough order
of appearance over the semester.
• “Lecture notes,” from Spielman’s Spectral Graph Theory class, Fall 2009 and 2012
• “Survey: Graph clustering,” in Computer Science Review, by Schaeffer
• “Geometry, Flows, and Graph-Partitioning Algorithms,” in CACM, by Arora, Rao, and
Vazirani
• “Lecture Notes on Expansion, Sparsest Cut, and Spectral Graph Theory,” by Trevisan
• “Expander graphs and their applications,” in Bull. Amer. Math. Soc., by Hoory, Linial,
and Wigderson
• “Multicommodity max-flow min-cut theorems and their use in designing approximation
algorithms,” in JACM, by Leighton and Rao
• “Efficient Maximum Flow Algorithms,” in CACM, by Goldberg and Tarjan
• “A Tutorial on Spectral Clustering,” in Statistics and Computing, by von Luxburg
• “A kernel view of the dimensionality reduction of manifolds,” in ICML, by Ham, et al.
• “Laplacian Eigenmaps for dimensionality reduction and data representation,” in Neural
Computation, by Belkin and Niyogi
• “Diffusion maps and coarse-graining: a unified framework for dimensionality reduction,
graph partitioning, and data set parameterization,” in IEEE-PAMI, by Lafon and Lee
• “Transductive learning via spectral graph partitioning,” in ICML, by Joachims
• “Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions,” in ICML, by
Zhu, Ghahramani, and Lafferty
• “Learning with local and global consistency,” in NIPS, by Zhou et al.
• “Random Walks and Electric Networks,” in arXiv, by Doyle and Snell
• “Implementing regularization implicitly via approximate eigenvector computation,” in
ICML, by Mahoney and Orecchia
• “Regularized Laplacian Estimation and Fast Eigenvector Approximation,” in NIPS, by
Perry and Mahoney
• “Spectral Ranking”, in arXiv, by Vigna
• “PageRank beyond the Web,” in SIAM Review, by Gleich
• “The Push Algorithm for Spectral Ranking”, in arXiv, by Boldi and Vigna
• “Local Graph Partitioning using PageRank Vectors,” in FOCS, by Andersen, Chung, and
Lang

3
• “A Local Spectral Method for Graphs: with Applications to Improving Graph Partitions
and Exploring Data Graphs Locally,” in JMLR, by Mahoney, Orecchia, and Vishnoi
• “Anti-differentiating Approximation Algorithms: A case study with Min-cuts, Spectral,
and Flow,” in ICML, by Gleich and Mahoney
• “Think Locally, Act Locally: The Detection of Small, Medium-Sized, and Large Communities in Large Networks,” in PRE, by Jeub, Balachandran, Porter, Mucha, and Mahoney
• “Towards a theoretical foundation for Laplacian-based manifold methods,” in JCSS, by
Belkin and Niyogi
• “Consistency of spectral clustering,” in Annals of Statistics, by von Luxburg, Belkin, and
Bousquet
• “Spectral clustering and the high-dimensional stochastic blockmodel,” in The Annals of
Statistics, by Rohe, Chatterjee, and Yu
• “Regularized Spectral Clustering under the Degree-Corrected Stochastic Blockmodel,” in
NIPS, by Qin and Rohe
• “Effective Resistances, Statistical Leverage, and Applications to Linear Equation Solving,”
in arXiv, by Drineas and Mahoney
• “A fast solver for a class of linear systems,” in CACM, by Koutis, Miller, and Peng
• “Spectral Sparsification of Graphs: Theory and Algorithms,” in CACM, by Batson, Spielman, Srivastava, and Teng
Finally, I should note that these notes are unchanged, relative to the notes that have been
available on my web page since the class completed; but, in response to a number of requests, I
decided to put them all together as a single file and post them on the arXiv. They are still very
rough, and they likely contain typos and errors. Thus, feedback and comments—both in terms
of specific technical issues as well as general scope—are most welcome.
Michael W. Mahoney
August 2016

4

Contents
1 (01/22/2015): Introduction and Overview

11

1.1

Basic motivating background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.2

Types of data and types of problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.3

Examples of graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.4

Some questions to consider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.5

Matrices for graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.6

An overview of some ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.7

Connections with random walks and diffusions . . . . . . . . . . . . . . . . . . . . . 18

1.8

Low-dimensional and non-low-dimensional data . . . . . . . . . . . . . . . . . . . . . 19

1.9

Small world and heavy-tailed examples . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1.10 Outline of class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2 (01/27/2015): Basic Matrix Results (1 of 3)

23

2.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.2

Some basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.3

Two results for Hermitian/symmetric matrices . . . . . . . . . . . . . . . . . . . . . 24

2.4

Consequences of these two results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.5

Some things that were skipped . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.6

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3 (01/29/2015): Basic Matrix Results (2 of 3)

31

3.1

Review and overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.2

Some initial examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.3

Basic ideas behind Perron-Frobenius theory . . . . . . . . . . . . . . . . . . . . . . . 33

3.4

Reducibility and types of connectedness . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.5

Basics of Perron-Frobenius theory

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4 (02/03/2015): Basic Matrix Results (3 of 3)
4.1

39

Review and overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5
4.2

Proof of the Perron-Frobenius theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.3

Positive eigenvalue with positive eigenvector. . . . . . . . . . . . . . . . . . . . . . . 40

4.4

That eigenvalue equals the spectral radius. . . . . . . . . . . . . . . . . . . . . . . . . 41

4.5

An extra claim to make. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.6

Monotonicity of spectral radius. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.7

Algebraic/geometric multiplicities equal one. . . . . . . . . . . . . . . . . . . . . . . 42

4.8

No other non-negative eigenvectors, etc. . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.9

Strict inequality for aperiodic matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.10 Limit for aperiodic matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.11 Additional discussion form periodicity/aperiodic and cyclicity/primitiveness . . . . . 44
4.12 Additional discussion of directness, periodicity, etc. . . . . . . . . . . . . . . . . . . . 46
5 (02/05/2015): Overview of Graph Partitioning

48

5.1

Some general comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.2

A first try with min-cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.2.1

Min-cuts and the Min-cut problem . . . . . . . . . . . . . . . . . . . . . . . . 51

5.2.2

A slight detour: the Max-Flow Problem . . . . . . . . . . . . . . . . . . . . . 52

5.3

Beyond simple min-cut to “better” quotient cut objectives . . . . . . . . . . . . . . . 53

5.4

Overview Graph Partition Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5.5

5.6

5.4.1

Local Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5.4.2

Spectral methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.4.3

Flow-based methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Advanced material and general comments . . . . . . . . . . . . . . . . . . . . . . . . 61
5.5.1

Extensions of the basic spectral/flow ideas . . . . . . . . . . . . . . . . . . . . 61

5.5.2

Additional comments on these methods . . . . . . . . . . . . . . . . . . . . . 62

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

6 (02/10/2015): Spectral Methods for Partitioning Graphs (1 of 2): Introduction
to spectral partitioning and Cheeger’s Inequality
63
6.1

6.2

Other ways to define the Laplacian . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.1.1

As a sum of simpler Laplacians . . . . . . . . . . . . . . . . . . . . . . . . . . 63

6.1.2

In terms of discrete derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Characterizing graph connectivity

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

6
6.2.1

A Perron-Frobenius style result for the Laplacian . . . . . . . . . . . . . . . . 65

6.2.2

Relationship with previous Perron-Frobenius results . . . . . . . . . . . . . . 67

6.3

Statement of the basic Cheeger Inequality . . . . . . . . . . . . . . . . . . . . . . . . 69

6.4

Comments on the basic Cheeger Inequality . . . . . . . . . . . . . . . . . . . . . . . 70

7 (02/12/2015): Spectral Methods for Partitioning Graphs (2 of 2): Proof of
Cheeger’s Inequality
72
7.1

Proof of the easy direction of Cheeger’s Inequality . . . . . . . . . . . . . . . . . . . 72

7.2

Some additional comments

7.3

A more general result for the hard direction . . . . . . . . . . . . . . . . . . . . . . . 74

7.4

Proof of the more general lemma implying the hard direction of Cheeger’s Inequality 76

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

8 (02/17/2015): Expanders, in theory and in practice (1 of 2)

80

8.1

Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

8.2

A first definition of expanders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

8.3

Alternative definition via eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

8.4

Expanders and Non-expanders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
8.4.1

Very sparse expanders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

8.4.2

Some non-expanders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

8.4.3

How large can the spectral gap be? . . . . . . . . . . . . . . . . . . . . . . . . 86

8.5

Why is d fixed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

8.6

Expanders are graphs that are very well-connected . . . . . . . . . . . . . . . . . . . 87

8.7

8.8

8.6.1

Robustness of the largest component to the removal of edges . . . . . . . . . 87

8.6.2

Relatedly, expanders exhibit quasi-randomness . . . . . . . . . . . . . . . . . 88

8.6.3

Some extra comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Expanders are graphs that are sparse versions/approximations of a complete graph . 90
8.7.1

A metric of closeness between two graphs . . . . . . . . . . . . . . . . . . . . 90

8.7.2

Expanders and complete graphs are close in that metric . . . . . . . . . . . . 92

Expanders are graphs on which diffusions and random walks mix rapidly . . . . . . . 93

9 (02/19/2015): Expanders, in theory and in practice (2 of 2)
9.1

96

Introduction to Metric Space Perspective on Expanders . . . . . . . . . . . . . . . . 96
9.1.1

Primal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

7
9.1.2
9.2

Dual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Metric Embedding into ℓ2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

10 (02/24/2015): Flow-based Methods for Partitioning Graphs (1 of 2)

105

10.1 Introduction to flow-based methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
10.2 Some high-level comments on spectral versus flow . . . . . . . . . . . . . . . . . . . . 105
10.3 Flow-based graph partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
10.4 Duality gap properties for flow-based methods . . . . . . . . . . . . . . . . . . . . . . 109
10.5 Algorithmic Applications

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

10.6 Flow Improve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
11 (02/26/2015): Flow-based Methods for Partitioning Graphs (2 of 2)

115

11.1 Review some things about ℓ1 and ℓ2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
11.2 Connection between ℓ1 metrics and cut metrics . . . . . . . . . . . . . . . . . . . . . 116
11.3 Relating this to a graph partitioning objective . . . . . . . . . . . . . . . . . . . . . . 118
11.4 Turning this into an algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
11.5 Summary of where we are . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
12 (03/03/2015): Some Practical Considerations (1 of 4): How spectral clustering
is typically done in practice
123
12.1 Motivation and general approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
12.2 Constructing graphs from data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
12.3 Connections between different Laplacian and random walk matrices

. . . . . . . . . 125

12.4 Using constructed data graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
12.5 Connections with graph cuts and other objectives . . . . . . . . . . . . . . . . . . . . 128
13 (03/05/2015): Some Practical Considerations (2 of 4): Basic perturbation theory and basic dimensionality reduction
132
13.1 Basic perturbation theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
13.2 Linear dimensionality reduction methods . . . . . . . . . . . . . . . . . . . . . . . . . 135
13.2.1 PCA (Principal components analysis) . . . . . . . . . . . . . . . . . . . . . . 135
13.2.2 MDS (Multi-Dimensional Scaling) . . . . . . . . . . . . . . . . . . . . . . . . 136
13.2.3 Comparison of PCA and MDS . . . . . . . . . . . . . . . . . . . . . . . . . . 136
13.2.4 An aside on kernels and SPSD matrices . . . . . . . . . . . . . . . . . . . . . 139

8
14 (03/10/2015): Some Practical Considerations (3 of 4): Non-linear dimension
reduction methods
141
14.1 Some general comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
14.2 ISOMAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
14.3 Local Linear Embedding (LLE)

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

14.4 Laplacian Eigenmaps (LE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
14.5 Interpretation as data-dependent kernels . . . . . . . . . . . . . . . . . . . . . . . . . 145
14.6 Connection to random walks on the graph: more on LE and diffusions . . . . . . . . 146
15 (03/12/2015): Some Practical Considerations (4 of 4): More on diffusions and
semi-supervised graph construction
149
15.1 Introduction to diffusion-based distances in graph construction . . . . . . . . . . . . 149
15.2 More on diffusion-based distances in graph construction . . . . . . . . . . . . . . . . 150
15.3 A simple result connecting random walks to NCUT/conductance . . . . . . . . . . . 151
15.4 Overview of semi-supervised methods for graph construction

. . . . . . . . . . . . . 152

15.5 Three examples of semi-supervised graph construction methods . . . . . . . . . . . . 153
16 (03/17/2015): Modeling graphs with electrical networks

156

16.1 Electrical network approach to graphs . . . . . . . . . . . . . . . . . . . . . . . . . . 156
16.2 A physical model for a graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
16.3 Some properties of resistor networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
16.4 Extensions to infinite graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
17 (03/19/2015): Diffusions and Random Walks as Robust Eigenvectors

163

17.1 Overview of this approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
17.2 Regularization, robustness, and instability of linear optimization . . . . . . . . . . . 165
17.3 Structural characterization of a regularized SDP . . . . . . . . . . . . . . . . . . . . 167
17.4 Deriving different random walks from Theorem 36 . . . . . . . . . . . . . . . . . . . 169
17.5 Interpreting Heat Kernel random walks in terms of stability . . . . . . . . . . . . . . 170
17.6 A statistical interpretation of this implicit regularization result . . . . . . . . . . . . 170
18 (03/31/2015): Local Spectral Methods (1 of 4): Introduction and Overview

175

18.1 Overview of local spectral methods and spectral ranking . . . . . . . . . . . . . . . . 175
18.2 Basics of spectral ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

9
18.3 A brief aside . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
19 (04/02/2015): Local Spectral Methods (2 of 4): Computing spectral ranking
with the push procedure
182
19.1 Background on the method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
19.2 The basic push procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
19.3 More discussion of the basic push procedure . . . . . . . . . . . . . . . . . . . . . . . 185
19.4 A different interpretation of the same process . . . . . . . . . . . . . . . . . . . . . . 185
19.5 Using this to find sets of low-conductance . . . . . . . . . . . . . . . . . . . . . . . . 187
20 (04/07/2015): Local Spectral Methods (3 of 4): An optimization perspective on
local spectral methods
190
20.1 A locally-biased spectral ansatz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
20.2 A geometric notion of correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
20.3 Solution of LocalSpectral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
20.4 Proof of Theorem 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
20.5 Additional comments on the LocalSpectral optimization program . . . . . . . . . . . 197
21 (04/09/2015): Local Spectral Methods (4 of 4): Strongly and weakly locallybiased graph partitioning
198
21.1 Locally-biased graph partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
21.2 Relationship between strongly and weakly local spectral methods . . . . . . . . . . . 201
21.3 Setup for implicit ℓ1 regularization in strongly local spectral methods . . . . . . . . . 201
21.4 Implicit ℓ1 regularization in strongly local spectral methods . . . . . . . . . . . . . . 204
22 (04/14/2015): Some Statistical Inference Issues (1 of 3): Introduction and
Overview
207
22.1 Overview of some statistical inference issues . . . . . . . . . . . . . . . . . . . . . . . 207
22.2 Introduction to manifold issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
22.3 Convergence of Laplacians, setup and background . . . . . . . . . . . . . . . . . . . . 212
22.4 Convergence of Laplacians, main result and discussion . . . . . . . . . . . . . . . . . 214
23 (04/16/2015): Some Statistical Inference Issues (2 of 3): Convergence and consistency questions
217
23.1 Some general discussion on algorithmic versus statistical approaches . . . . . . . . . 217

10
23.2 Some general discussion on similarities and dissimilarities . . . . . . . . . . . . . . . 219
23.3 Some general discussion on embedding data in Hilbert and Banach spaces . . . . . . 221
23.4 Overview of consistency of normalized and unnormalized Laplacian spectral methods 222
23.5 Details of consistency of normalized and unnormalized Laplacian spectral methods . 225
24 (04/21/2015): Some Statistical Inference Issues (3 of 3): Stochastic blockmodels231
24.1 Introduction to stochastic block modeling . . . . . . . . . . . . . . . . . . . . . . . . 231
24.2 Warming up with the simplest SBM . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
24.3 A result for a spectral algorithm for the simplest nontrivial SBM . . . . . . . . . . . 233
24.4 Regularized spectral clustering for SBMs . . . . . . . . . . . . . . . . . . . . . . . . . 237
25 (04/23/2015): Laplacian solvers (1 of 2)

241

25.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
25.2 Basic statement and outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
25.3 A simple slow algorithm that highlights the basic ideas . . . . . . . . . . . . . . . . . 243
26 (04/28/2015): Laplacian solvers (2 of 2)

248

26.1 Review from last time and general comments . . . . . . . . . . . . . . . . . . . . . . 248
26.2 Solving linear equations with direct and iterative methods . . . . . . . . . . . . . . . 250
26.3 Different ways two graphs can be close . . . . . . . . . . . . . . . . . . . . . . . . . . 251
26.4 Sparsified graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
26.5 Back to Laplacian-based linear systems

. . . . . . . . . . . . . . . . . . . . . . . . . 254

Lecture Notes on Spectral Graph Methods

1

11

(01/22/2015): Introduction and Overview

1.1

Basic motivating background

The course will cover several topics in spectral graph methods. By that, I mean that it will cover
not spectral graph theory per se, nor will it cover the application of spectral graph methods per
se. In addition, spectral methods is a more general topic, and graph methods is a more general
topic. Spectral graph theory uses eigenvectors and eigenvalues (and related quantities) of matrices
associated with graphs to say things about those graphs. It is a topic which has been studied
from a wide range of perspectives, e.g., theoretical computer science, scientific computing, machine
learning, statistics, etc., and as such it is a topic which can be viewed from a wide range of
approaches.
The reason for the focus on spectral graph methods is that a wide range of problems are obviously
spectral graph methods, and thus they are useful in practice as well as interesting in theory; but,
in addition, many other methods that are not obviously spectral graph methods really are spectral
graph methods under the hood. We’ll get to what I mean by that, but for now think of running
some procedure that seems to work, and if one were to perform a somewhat rigorous algorithmic or
statistical analysis, it would turn out that that method essentially boiled down to a spectral graph
method. As an example, consider the problem of viral propagation on a social network, which is
usually described in terms of some sort of infectious agent, but which has strong connections with
spectral graph methods.
Our goal will be to understand—by drawing strength from each of the wide range of approaches
that have been brought to bear on these problems—when/why spectral graph methods are useful
in practical machine learning and data analysis applications, and when/why they (or a vanilla
variant of them) are not useful. In the latter case, of course, we’ll be interested in understanding
whether a better understanding of spectral graph methods can lead to the development of improved
algorithmic and statistical techniques—both for large-scale data as well as for small-scale data.
Relatedly, we will be interested in whether other methods can perform better or whether the data
are just “bad” in some sense.

1.2

Types of data and types of problems

Data comes from all sorts of places, and it can be a challenge to find a good way to represent the
data in order to obtain some sort of meaningful insight from the data. Two popular ways to model
data are as matrices and as graphs.
• Matrices often arise when there are n things, each of which is described by m features. In
this case, we have an m × n matrix A, where each column is a data point described by a
bunch of features (or vice versa) and where each row is a vector describing the value of that
feature at each data point. Alternatively, matrices can arise when there are n things and we
have information about the correlations (or other relationships) between them.
• Graphs often arise when there are n things and the pairwise relationships between them are
though to be particularly important. Let’s specify a graph by G = (V, E), where V is the set
of vertices and E is the set of edges, which are pairs of vertices. (Later they can be weighted,

12

M. W. Mahoney
etc., but for now let’s say they are undirected, unweighted, etc.) Examples of data graphs
include the following.
– Discretizations of partial differential equations and other physical operators give rise to
graphs, where the nodes are points in a physical medium and edges correspond to some
sort of physical interaction.
– Social networks and other internet applications give rise to graphs, where the nodes are
individuals and there is an edge between two people if they are friends or have some
other sort of interaction.
– Non-social networks give rise to graphs, where, e.g., devised, routers, or computers are
nodes and where there is an edge between two nodes if they are connected and/or have
traffic between them.
– Graphs arise more generally in machine learning and data analysis applications. For
example, given a bunch of data points, each of which is a feature vector, we could
construct a graph, where the nodes correspond to data points and there is an edge
between two data points if they are close in some sense (or a soft version of this, which
is what rbf kernels do).

In the same way as we can construct graphs from matrices, we can also construct matrices from
graphs. We will see several examples below (e.g., adjacency matrices, Laplacians, low-rank embedding matrices, etc.). Spectral graph methods involve using eigenvectors and eigenvalues of matrices
associated with graphs to do stuff.
In order to do stuff, one runs some sort of algorithmic or statistical methods, but it is good to keep
an eye on the types of problems that might want to be solved. Here are several canonical examples.
• Graph partitioning: finding clusters/communities. Here, the data might be a bunch of data
points (put a picture: sprinkled into a left half and a right half) or it might be a graph (put a
picture: two things connected by an edge). There are a million ways to do this, but one very
popular one boils down to computing an eigenvector of the so-called Laplacian matrix and
using that to partition the data. Why does such a method work? One answer, from TCS,
is that is works since it is a relaxation of a combinatorial optimization problem for which
there are worst-case quality-of-approximation guarantees. Another answer, from statistics
and machine learning, is that is can be used to recover hypothesized clusters, say from a
stochastic blockmodel (where the graph consists of several random graphs put together) or
from a low-dimensional manifold (upon which the data points sit).
• Prediction: e.g., regression and classification. Here, there is a similar picture, and one popular
procedure is to run the same algorithm, compute the same vector and use it to classify the
data. In this case, one can also ask: why does such a method work? One answer that
is commonly given is that if there are meaningful clusters in the data, say drawn from a
manifold, then the boundaries between the clusters correspond to low-density regions; or
relatedly that class labels are smooth in the graph topology, or some notion of distance in Rn .
But, what if the data are from a discrete place? Then, there is the out-of-sample extension
question and a bunch of other issues.
• Centrality and ranking. These are two (different but sometimes conflated) notions from
sociology and social networks having to do with how “important” or “central” is an individual/node and relatedly how to rank individuals/nodes. One way to do this is to choose the

Lecture Notes on Spectral Graph Methods

13

highest degree node, but this is relatively easy to spam and might not be “real” in other ways,
and so there are several related things that go by the name of spectral ranking, eigenvector
centrality, and so on. The basic idea is that a node is important if important nodes thing it
is important. This suggests looking at loops and triangles in a graph, and when this process
is iterated you get random walks and diffusions on the graph. It’s not obvious that this has
very strong connections with the clustering, classification, etc. problems described above, but
it does. Basically, you compute the same eigenvector and use it to rank.
• Encouraging or discouraging “viral propagation.” Here, one is given, say, a social network,
and one is interested in some sort of iterative process, and one wants to understand its
properties. Two examples are the following: there might be a virus or other infectious agent
that goes from node to node making other agents sick; or there might be some sort of “buzz”
about a new movie or new tennis shoes, and this goes from individual to individual. Both
of these are sometimes called viral propagation, but there are important differences, not the
least of which is that in the former people typically want to stop the spread of the virus, while
in the latter people want to encourage the spread of the virus to sell more tennis shoes.

1.3

Examples of graphs

When algorithms are run on data graphs—some of which might be fairly nice but some of which
might not, it can be difficult to know why the algorithm performs as it does. For example, would
it perform that way on every possible graph? Similarly, if we are not in some asymptotic limit or
if worst-case analysis is somewhat too coarse, then what if anything does the method reveal about
the graph? To help address these and related questions, it helps to have several examples of graphs
in mind and to see how algorithms perform on those graphs. Here are several good examples.
• A discretization of nice low-dimensional space, e.g., the integers/lattice in some fixed dimension: Zd , Z2d , and Z3d .
• A star, meaning a central node to which all of the other nodes are attached.
• A binary tree.
• A complete graph or clique.
• A constant-degree expander, which is basically a very sparse graph that has no good partitions. Alternatively, it can be viewed as a sparse version of the complete graph.
• A hypercube on 2n vertices.
• A graph consisting of two complete graphs or two expanders or two copies of Z2d that are
weakly connected by, say, a line graph.
• A lollipop, meaning a complete graph of expander, with a line graph attached, where the
line/stem can have different lengths.
Those are common constructions when thinking about graphs. The following are examples of
constructions that are more common in certain network applications.
• An Erdos-Renyi random graph, Gnp , for p = 3/n or p & log(n)/n

14

M. W. Mahoney
• A “small world” graph, which is basically a ring plus a 3 regular random graph.
• A heavy-tailed random graph, with or without min degree assumption, or one constructed
from a preferential attachment process.

In addition to things that are explicitly graphs, it also helps to have several examples of matrixbased data to have in mind from which graphs can be constructed. These are often constructed
from some sort of nearest-neighbor process. Here are several common examples.
• A nice region of a low-dimensional subspace of Rn or of a nice low-dimensional manifold
embedded in Rn .
• A full-dimensional Gaussian in Rn . Here, most of the mass is on the shell, but what does the
graph corresponding to this “look like”?
• Two low-dimensional Gaussians in Rn . This looks like a dumbbell, with two complete graphs
at the two ends or two copies of Znd at the ends, depending on how parameters are set.

1.4

Some questions to consider

Here are a few questions to consider.
• If the original data are vectors that form a matrix, how sensitive are these methods to the
details of the graph construction? (Answer: in theory, no; in practice, often yes.)
• If the original data are represented by a graph, how sensitive are these methods to a bit of
noise in the graph? (Answer: in theory, no; in practice, often yes.)
• How good a guide is worst case cs and asymptotic statistical theory? (Answer: in theory,
good; in practice, often not, but it depends on what is the reference state, e.g., manifold
versus stochastic blockmodel.)
• What if you are interested in a small part of a very large graph? E.g., you and your 100
closest friends on a social network, as opposed to you and your 109 closest friends on that
social network. Do you get the same results if you run some sort of local algorithm on a small
part of the graph as you do if you run a global algorithm on a subgraph that is cut out?
(Typically no, unless you are very careful.)

1.5

Matrices for graphs

Let G = (V, E, W ) be an undirected, possibly weighted, graph. There are many matrices that one
can associate with a graph. Two of the most basic are the adjacency matrix and the diagonal degree
matrix.
Definition 1. Given G = (V, E, W ), the adjacency matrix A ∈ Rn×n is defined to be

Wij
if (ij) ∈ E
,
Aij =
0
otherwise

Lecture Notes on Spectral Graph Methods

15

and the diagonal degree matrix D ∈ Rn×n is defined to be
 P
if i = j
k Wik
Dij =
.
0
otherwise
Note that for undirected graphs, i.e., when Wij equals 1 or 0 depending on whether or not there is
an edge between nodes i and j, the adjacency matrix specifies which edges are connected and the
diagonal degree matrix gives the degree of ith node at the (ii)th diagonal position.
(Given this setup, it shouldn’t be surprising that most spectral graph methods generalize in nice
ways from unweighted to weighted graphs. Of interest also are things like time-evolving graphs,
directed graphs, etc. In those cases, the situation is more subtle/complex. Typically, methods for
those more complex graphs boil down to methods for simpler undirected, static graphs.)
Much of what we will discuss has strong connections with spectral graph theory, which is an
area that uses eigenvectors and eigenvalues of matrices associated with the graph to understand
properties of the graph. To begin, though, we should note that it shouldn’t be obvious that
eigenstuff should reveal interesting graph properties—after all, graphs by themselves are essentially
combinatorial things and most traditional graph problems and algorithms don’t mention anything
having to do with eigenvectors. In spite of this, we will see that eigenstuff reveals a lot about graphs
that are useful in machine learning and data analysis applications, and we will want to understand
why this is the case and how we can take advantage of it in interesting ways.
Such an approach of using eigenvectors and eigenvalues is most useful when used to understand
a natural operator of natural quadratic form associated with the graphs. Perhaps surprisingly,
adjacency matrices and diagonal degree matrices are not so useful in that sense—but they can be
used to construct other matrices that are more useful in that sense.
One natural and very useful operator to associate with a graph G is the following.
Definition 2. Given G = (V, E, W ), the diffusion operator is
W = D −1 A

(or M = AD −1 , if you multiply from the other side).

This matrix describes the behavior of diffusions and random walks on G. In particular, if x ∈ Rn
is a row vector that gives the probability that a particle is at each vertex of G, and if the particle
then moves to a random neighbor, then pW is the new probability distribution of the particle. If
the graph G is regular, meaning that it is degree-homogeneous, then W is a rescaling of A, but
otherwise it can be very different. Although we won’t go into too much detail right now, note that
applying this operator to a vector can be interpreted as doing one step of a diffusion or random walk
process. In this case, one might want to know what happens if we iteratively apply an operator
like W . We will bet back to this.
One natural and very useful quadratic form to associate with a graph G is the following.
Definition 3. Given G = (V, E, W ), the Laplacian matrix (or combinatorial Laplacian matrix) is
L = D − A.
Although we won’t go into detail, the Laplacian has an interpretation in terms of derivatives. (This
is most common/obvious in continuous applications, where it can be used to measure the smoothness of the Laplacian and/or of some continuous place from which the gradient was constructed—if

16

M. W. Mahoney

it was—in a nice way, which is often not the case.) Given a function or a vector x ∈ Rn , the
Laplacian quadratic form is
X
xT Lx =
(xi − xj )2 .
(ij)∈E

This is a measure of smoothness of the vector/function x—smoothness of x, in some sense, conditioned on the graph structure. (That is, it is a statement about the graph itself, independent of
how it was constructed. This is of interest by itself but also for machine learning and data analysis
application, e.g., since labels associated with the nodes that correspond to a classification function
might be expected to be smooth.)
Alternatively, we can define the normalized Laplacian matrix.
Definition 4. Given G = (V, E, W ), the normalized Laplacian matrix is
L = L = D −1/2 LD −1/2 = I − D −1/2 LD −1/2 .
(Note that I have already started to be a little sloppy, by using the same letter to mean two different
things. I’ll point out as we go where this matters.) As we will see, for degree-homogeneous graphs,
these two Laplacians are essentially the same, but for degree-heterogeneous graphs, they are quite
different. As a general rule, the latter is more appropriate for realistic degree-heterogeneous graphs,
but it is worth keeping the two in mind, since there are strong connections between them and how
they are computed. Similar smoothness, etc. interpretations hold for the normalized Laplacian,
and this is important in many applications.

1.6

An overview of some ideas

Here is a vanilla version of a spectral graph algorithm that will be central to a lot of what we do.
We’ll be more precise and go into a lot more detail later. The algorithm takes as input a graph, as
specified by a Laplacian matrix, L or L.
1. Compute, exactly or approximately, the leading nontrivial eigenvector of L or L.
2. Use that vector to split the nodes of the graph into a left half and a right half.
Those two pieces can be the two clusters, in which case this algorithm is a vanilla version of spectral
graph partitioning; or with some labels that can be used to make predictions for classification or
regression; or we can rank starting from the left and going to the right; or we can use the details of
the approximate eigenvector calculation, e.g., random walks and related diffusion-based methods,
to understand viral diffusion problems. But in all those cases, we are interested in the leading
nontrivial eigenvector of the Laplacian. We’ll have a lot more to say about that later, but for now
think of it just as some vector that in some sense describes important directions in the graph, in
which case what this vanilla spectral algorithm does is putting or “embedding” the nodes of the
graph on this line and cuts the nodes into two pieces, a left half and a right half. (The embedding
has big distortion, in general, for some points at least; the two halves can be very unbalanced, etc.;
but at least in very nice cases, that informal intuition is true, and it is true more generally if the
two halves can be unbalanced, etc. Understanding these issues will be important for what we do.)
(Make a picture on the board.)

Lecture Notes on Spectral Graph Methods

17

We can ask: what is the optimization problem that this algorithm solves? As we will see eventually,
in some sense, what makes a spectral graph algorithm a spectral graph algorithm is the first step,
and so let’s focus on that. Here is a basic spectral optimization problem that this problem solves.

min

xT L G x

s.t. xT DG x = 1
xT DG~1 = 0
x ∈ RV
That is, find the vector that minimizes the quadratic form xT Lx subject to the constraints that x
sits on a (degree-weighted) unit ball and that x is perpendicular (in a degree-weighted norm) to the
“trivial” all-ones vector. The solution to this problem is a vector, and it is the leading nontrivial
eigenvector of L or L.
Importantly, this is not a convex optimization problem; but rather remarkably it can be solved in
low-degree polytime by computing an eigenvector of L. How can it be that this problem is solvable
if it isn’t convex? After all, the usual rule of thumb is that convex things are good and non-convex
things are bad. There are two (related, certainly not inconsistent) answers to this.
• One reason is that this is an eigenvector (or generalized eigenvector) problem. In fact, it
involves computing the leading nontrivial eigenvector of L, and so it is a particularly nice
eigenvalue problem. And computing eigenvectors is a relatively easy thing to do—for example,
with a black box solver, or in this special case with random walks. But more on this later.
• Another reason is that is is secretly convex, in that it is convex in a different place. Importantly, that different place there are better duality properties for this problem, and so it can
be used to understand this problem and its solution better.
Both of these will be important, but let’s start by focusing on the second reason. Consider the
following version of the basic spectral optimization problem.
SDP : min L • X
s.t.

Tr (X) = I0 • X = 1

X  0,


T
=
where
•
stands
for
the
Trace,
or
matrix
inner
product,
operation,
i.e.,
A
•
B
=
Tr
AB
P
ij Aij Bij for matrices A and B. Note that, both here and below, I0 is sometimes the Identity on
the subspace perpendicular to the all-ones vector. This will be made more consistent later. SDP
is a relaxation of the spectral program SPECTRAL from an optimization over unit vectors to an
optimization over distributions over unit vectors, represented by the density matrix X. But, the
optimal values for SPECTRAL and SDP are the same, in the sense that they are given by the second
eigenvector v of L for SPECTRAL and by X = vv T for SDP.
Thus, this is an SDP. While solving the vanilla spectral optimization problem with a black-box
SDP solver is possible, it is not advisable, since one can just can a black-box eigenvalue solver (or

18

M. W. Mahoney

run some non-black-box method that approximates the eigenvalue). Nevertheless, the SDP can be
used to understand spectral graph methods. For example, we can consider the dual of this SDP:
maximize

α

s.t.

LG  αLKn

α∈R

This is a standard dual construction, the only nontrivial thing is we write out explicitly I0 = LKn ,
as the identity matrix on the subspace perpendicular to ~1 is I0 = I − ~1T ~1 = LKn where Kn is the
complete graph on n vertices.
We will get into more detail later what this means, but informally this means that we are in some
sense “embedding” the Laplacian of the graph in the Laplacian of a complete graph. Slightly less
informally, the  is an inequality over graphs (we will define this in more detail later) which says
that the Laplacian quadratic form of one graph is above or below that of another graph (in the
sense of SPSD matrices, if you know what that means). So, in this dual, we want to choose the
largest α such that that inequality is true.

1.7

Connections with random walks and diffusions

A final thing to note is that the vector x∗ that solves these problems has a natural interpretation
in terms of diffusions and random walks. This shouldn’t be surprising, since one of the ways this
vector is to partition a graph into two pieces that captures a qualitative notion of connectivity.
The interpretation is that is you run a random walk—either a vanilla random walk defined by the
matrix W = D −1 A above, meaning that at each step you go to one of your neighbors with equal
probability, or a fancier random walk—then x∗ defines the slowest direction to mixing, i.e., the
direction that is at the pre-asymptotic state before you get to the asymptotic uniform distribution.
So, that spectral graph methods are useful in these and other applications is largely due to two
things.
• Eigenvectors tend to be “global” things, in that they optimize a global objective over the
entire graph.
• Random walks and diffusions optimize almost the same things, but they often do it in a very
different way.
One of the themes of what we will discuss is the connection between random walks and diffusions
and eigenvector-based spectral graph methods on different types of graph-based data. Among
other things, this will help us to address local-global issues, e.g., the global objective that defines
eigenvectors versus the local nature of diffusion updates.
Two things should be noted about diffusions.
• Diffusions are robust/regularized notions of eigenvectors
• The behavior of diffusions is very different on Kn or expander-like metric spaces than it is on
line-like or low-dimensional metric spaces.

Lecture Notes on Spectral Graph Methods

19

An important subtlety is that most data have some sort of degree heterogeneity, and so the extremal
properties of expanders are mitigated since it is constant-degree expanders that are most unlike
low-dimensional metric spaces. (In the limit, you have a star, and there it is trivial why you don’t
have good partitions, but we don’t want to go to that limit.)

1.8

Low-dimensional and non-low-dimensional data

Now, complete graphs are very different than line graphs. So, the vanilla spectral graph algorithm
is also putting the data in a complete graph. To get a bit more intuition as to what is going
on and to how these methods will perform in real applications, consider a different type of graph
known as an expander. I won’t give the exact definition now—we will later—but expanders are
very important, both in theory and in practice. (The former may be obvious, while the latter
may be less obvious.) For now, there are three things you need to know about expanders, either
constant-degree expanders and/or degree-heterogeneous expanders.
• Expanders are extremely sparse graphs that do not have any good clusters/partitions, in a
precise sense that we will define later.
• Expanders are also the metric spaces that are least like low-dimensional spaces, e.g., a line
graph, a two-dimensional grid, etc. That is, if your intuition comes from low-dimensional
places like 1D or 2D places, then expanders are metric spaces that are most different than
that.
• Expanders are sparse versions of the complete graph, in the sense that there are graph inequalities of the form  that relate the Laplacian quadratic forms of expanders and complete
graphs.
So, in a certain precise sense, the vanilla spectral graph method above (as well as other non-vanilla
spectral methods we will get to) put or embed the input data in two extremely different places, a
line as well as a dense expander, i.e., a complete graph.
Now, real data have low-dimensional properties, e.g., sometimes you can visualize them in a twodimensional piece of paper and see something meaningful, and since they often have noise, they
also have expander-like properties. (If that connection isn’t obvious, it soon will be.) We will
see that the properties of spectral graph methods when applied to real data sometimes depend
on one interpretation and sometimes depend on the other interpretation. Indeed, many of the
properties—both the good properties as well as the bugs/features—of spectral graph methods can
be understood in light of this tension between embedding the data in a low-dimensional place and
embedding the data in an expander-like place.
There are some similarities between this—which is a statement about different types of graphs and
metric spaces—and analogous statements about random vectors in Rn , e.g., from a full-dimensional
Gaussian distribution in Rn . Some of these will be explored.

1.9

Small world and heavy-tailed examples

There are several types or classes of generative models that people consider, and different communities tend to adopt one or the other class. Spectral graph methods are applied to all of these,
although they can be applied in somewhat different ways.

20

M. W. Mahoney
• Discretization or random geometric graph of some continuous low-dimensional place, e.g., a
linear low-dimensional space, a low-dimensional curved manifold, etc. In this case, there is a
natural low-dimensional geometry. (Put picture on board.)
• Stochastic blockmodels, where there are several different types of individuals, and each type
interacts with individuals in the same group versus different groups with different probabilities. (Put picture on board, with different connection probabilities.)
• Small-world and heavy-tailed models. These are generative graph models, and they attempt
to capture some local aspect of the data (in one case, that there is some low-dimensional
geometry, and in the other case, that there is big variability in the local neighborhoods of
individuals, as captured by degree or some other simple statistic) and some global aspect to
the data (typically that there is a small diameter).

We will talk about all three of these in due course, but for now let’s say just a bit about the
small-world models and what spectral methods might reveal about them in light of the above
comments. Small-world models start with a one-dimensional or two-dimensional geometry and add
random edges in one of several ways. (Put picture here.) The idea here is that you reproduce
local clustering and small diameters, which is a property that is observed empirically in many
real networks. Importantly, for algorithm and statistical design, we have intuition about lowdimensional geometries; so let’s talk about the second part: random graphs.
Consider Gnp and Gnm , which are the simplest random graph models, and which have an expected
or an exact number of edges, respectively. In particular, start with n isolated vertices/nodes; then:
• For Gnp , insert each of the

n
2



possible edges, independently, each with probability p.

n 
2 ) subsets of m edges, select one, independently at random.
• For Gnm , among all (m

In addition to being of theoretical interest, these models are used in all sorts of places. For example,
they are the building blocks of stochastic block models, in addition to providing the foundation for
common generative network models. (That is, there are heavy-tailed versions of this basic model
as well as other extensions, some of which we will consider, but for now let’s stick with this.) These
vanilla ER graphs are often presented as strawmen, which in some sense they are; but when taken
with a grain of salt they can reveal a lot about data and algorithms and the relationship between
the two.
First, let’s focus on Gnp . There are four regimes of particular interest.
• p < n1 . Here, the graph G is not fully-connected, and it doesn’t even have a giant component,
so it consists of just a bunch of small things.
•

. p . log(n)
n . Here there is a giant component, i.e., set of Ω(n) nodes that are connected,
that has a small O(log(n)) diameter. In addition, random walks mix in O(log2 (n)) steps, and
the graph is locally tree-like.

•

log(n)
n

1
n

. p. Here the graph is fully-connected. In addition, it has a small O(log(n)) diameter
and random walks mix in O(log2 (n)) steps (but for a slightly different reason that we will get
back to later).

Lecture Notes on Spectral Graph Methods
•

21

log(n)
n

≪ p. Here the graph is pretty dense, and methods that are applicable to pretty dense
graphs are appropriate.

If p & log(n)/n, then Gnp and Gnm are “equivalent” in a certain sense. But if they are extremely
sparse, e.g., p = 3/n or p = 10/n, and the corresponding values of m, then they are different.
In particular, if p = 3/n, then the graph is not fully-connected, but we can ask for a random
r-regular graph, where r is some fixed small integer. That is, fix the number of edges to be r, so
we have a total of nr edges which is almost a member of Gnm , and look at a random such graph.
• A random 1-regular graph is a matching.
• A random 2-regular graph is a disjoint union of cycles.
• A random 3-regular graph: it is fully-connected and had a small O(log(n)) diameter; it is
an expander; it contains a perfect matching (a matching, i.e., a set of pairwise non-adjacent
edges that matches all vertices of the graph) and a Hamiltonian cycle (a closed loop that
visits each vertex exactly once).
• A random 4-regular graph is more complicated to analyze
How does this relate to small world models? Well, let’s start with a ring graph (a very simple
version of a low-dimensional lattice, in which each node is connected to neighbors within a distance
k = 1) and add a matching (which is a bunch of random edges in a nice analyzable way). Recall
that a random 3-regular graph has both a Hamiltonian cycle and a perfect matching; well it’s also
the case that the union of an n cycle and a random machine is contiguous to a random 3 regular
random graphs. (This is a type of graph decomposition we won’t go into.)
This is a particular theoretical form to say that small world models have a local geometry but
globally are also expanders in a strong sense of the word. Thus, in particular, when one runs things
like diffusions on them, or relatedly when one runs spectral graph algorithms (which have strong
connections under the hood to diffusions) on them, what one gets will depend sensitively on the
interplay between the line/low-dimensional properties and the noise/expander-like properties.
It is well known that similar results hold for heavy-tailed network models such as PA models or
PLRG models or many real-world networks. There there is degree heterogeneity, and this can give a
lack of measure concentration that is analogous to the extremely sparse Erdos-Renyi graphs, unless
one does things like make minimum degree assumptions. It it less well known that similar things
also hold for various types of constructed graphs. Clearly, this might happen if one constructs
stochastic blockmodels, since then each piece is a random graph and we are interested in the
interactions between different pieces. But what if construct a manifold method, but there is a bit
of noise? This is an empirical question; but noise, if it is noise, can be thought of as a random
process, and in the same way as the low-dimensional geometry of the vanilla small world model is
not too robust to adding noise, similarly geometric manifold-based methods are also not too robust.
In all of this, there algorithm questions, as well as statistical and machine learning questions such as
model selection questions and questions about how to do inference with vector-based of graph-based
data, as well as mathematical questions, as well as questions about how these methods perform in
practice. We will revisit many of these over the course of the semester.

22

M. W. Mahoney

1.10

Outline of class

In light of all that, here is an outline of some representative topics that we will cover.
1. Basics of graph partitioning, including spectral, flow, etc., degree heterogeneity, and other
related objectives.
2. Connections with diffusions and random walks, including connections with resistor network,
diffusion-based distances, expanders, etc.
3. Clustering, prediction, ranking/centrality, and communities, i.e., solving a range of statistics
and data analysis methods with variants of spectral methods.
4. Graph construction and empirical properties, i.e., different ways graphs can be constructed
and empirical aspects of “given” and “constructed” graphs.
5. Machine learning and statistical approaches and uses, e.g., stochastic blockmodels, manifold
methods, regularized Laplacian methods, etc.
6. Computations, e.g., nearly linear time Laplacian solvers and graph algorithms in the language
of linear algebra.

Lecture Notes on Spectral Graph Methods

2

23

(01/27/2015): Basic Matrix Results (1 of 3)

Reading for today.
• “Lecture notes,” from Spielman’s Spectral Graph Theory class, Fall 2009 and 2012

2.1

Introduction

Today and next time, we will start with some basic results about matrices, and in particular the
eigenvalues and eigenvectors of matrices, that will underlie a lot of what we will do in this class. The
context is that eigenvalues and eigenvectors are complex (no pun intended, but true nonetheless)
things and—in general—in many ways not so “nice.” For example, they can change arbitrarily as
the coefficients of the matrix change, they may or may not exist, real matrices may have complex
eigenvectors and eigenvalues, a matrix may or may not have a full set of n eigenvectors, etc. Given
those and related instabilities, it is an initial challenge is to understand what we can determine
from the spectra of a matrix. As it turns out, for many matrices, and in particular many matrices
that underlie spectral graph methods, the situation is much nicer; and, in addition, in some cases
they can be related to even nicer things like random walks and diffusions.
So, let’s start by explaining “why” this is the case. To do so, let’s get some context for how/why
matrices that are useful for spectral graph methods are nicer and also how these nicer matrices sit in
the larger universe of arbitrary matrices. This will involve establishing a few basic linear algebraic
results; then we will use them to form a basis for a lot of the rest of what we will discuss. This is
good to know in general; but it is also good to know for more practical reasons. For example, it
will help clarify when vanilla spectral graph methods can be extended, e.g., to weighted graphs or
directed graphs or time-varying graph or other types of normalizations, etc.

2.2

Some basics

To start, recall that we are interested in the Adjacency matrix of a graph G = (V, E) (or G =
(V, E, W ) if the graph is weighted) and other matrices that are related to the Adjacency matrix.
Recall that the n × n Adjacency matrix is defined to be

Wij
if (ij) ∈ E
Aij =
,
0
otherwise
where Wij = 1, for all (i, j) ∈ E if the graph is unweighted. Later, we will talk about directed
graphs, in which case the Adjacency matrix is not symmetric, but note here it is symmetric. So,
let’s talk about symmetric matrices: a symmetric matrix is a matrix A for which A = AT , i.e., for
which Aij = Aji .
Almost all of what we will talk about will be real-valued matrices. But, for a moment, we will start
with complex-valued matrices. To do so, recall that if x = α + iβ ∈ C is a complex number, then
x̄ = α − iβ ∈ C is the complex conjugate of x. Then, if M ∈ Cm×n is a complex-valued matrix,
i.e., an m × n matrix each entry of which is a complex number, then the conjugate transpose of M ,
which is denoted M ∗ , is the matrix defined as
(M ∗ )ij = M̄ji .

24

M. W. Mahoney

Note that if M happens to be a real-valued m × n matrix, then this is just the transpose.
If x, y ∈ Cn are two complex-valued vectors, then we can define their inner product to be
∗

hx, yi = x y =

n
X

x̄i yi .

i=1

Note that from this we can also get a norm in the usual way, i.e., hx, xi = kxk22 ∈ R. Given all this,
we have the following definition.
Definition 5. If M ∈ Cn×n is a square complex matrix, λ ∈ C is a scalar, and x ∈ Cn \{0} is a
non-zero vector such that
M x = λx
(1)
then λ is an eigenvalue of M and x is the corresponding eigenvector of λ.
Note that when Eqn. (1) is satisfied, then this is equivalent to
(M − λI) x = 0, for x 6= 0,

(2)

where I is an n × n Identity matrix. In particular, this means that we have at least one eigenvalue/eigenvector pair. Since (2) means M − λI is rank deficient, this in turn is equivalent to
det (M − λI) = 0.
Note that this latter expression is a polynomial with λ as the variable. That is, if we fix M , then
the function given by λ → det (M − λI) is a univariate polynomial of degree n in λ. Now, it
is a basic fact that every non-zero, single-variable, degree polynomial of degree n with complex
coefficients has—counted with multiplicity—exactly n roots. (This counting multiplicity thing
might seem pedantic, but it will be important latter, since this will correspond to potentially
degenerate eigenvalues, and we will be interested in how the corresponding eigenvectors behave.)
In particular, any square complex matrix M has n eigenvectors, counting multiplicities, and there
is at least one eigenvalue.
As an aside, someone asked in class if this fact about complex polynomials having n complex roots
is obvious or intuitive. It is sufficiently basic/important to be given the name the fundamental
theorem of algebra, but its proof isn’t immediate or trivial. We can provide some intuition though.
Note that related formulations of this state that every non-constant single-variable polynomial with
complex coefficients has at least one complex root, etc. (e.g., complex roots come in pairs); and that
the field of complex numbers is algebraically closed. In particular, the statements about having
complex roots applies to real-valued polynomials, i.e., since real numbers are complex numbers
polynomials in them have complex roots; but it is false that real-valued polynomials always have
real roots. Equivalently, the real numbers are not algebraically closed. To see this, recall that the
equation x2 − 1 = 0, viewed as an equation over the reals has two real roots, x = ±1; but the
equation x2 + 1 = 0 does not have any real roots. Both of these equations have roots over the
complex plane: the former having the real roots x = ±1, and the latter having imaginary roots
x = ±i.

2.3

Two results for Hermitian/symmetric matrices

Now, let’s define a special class of matrices that we already mentioned.

Lecture Notes on Spectral Graph Methods

25

Definition 6. A matrix M ∈ Cn×n is Hermitian if M = M ∗ . In addition, a matrix M ∈ Rn×n is
symmetric if M = M ∗ = M T .
For complex-valued Hermitian matrices, we can prove the following two lemmas.
Lemma 1. Let M be a Hermitian matrix. Then, all of the eigenvalues of M are real.
Proof: Let M be Hermitian and λ ∈ C and x non-zero be s.t. M x = λx. Then it suffices to show
that λ = λ∗ , since that means that λ ∈ R. To see this, observe that
hM x, xi =
=

XX
i

j

XX
i

M¯ij x¯j xi
Mji xi x¯j

(3)

j

= hx, M xi
where Eqn. (3) follows since M is Hermitian. But we have
hM x, xi = hλx, xi = λ̄ hx, xi = λ̄kxk22
and also that
hx, M xi = hx, λxi = λ hx, xi = λkxk22 .
Thus, λ = λ̄, and the lemma follows.
⋄
Lemma 2. Let M be a Hermitian matrix; and let x and y be eigenvectors corresponding to different
eigenvalues. Then x and y are orthogonal.
Proof: Let M x = λx and M y = λ′ y. Then,
hM x, yi = (M x)∗ y = x∗ M ∗ y = x∗ M y = hx, M yi .
But,
hM x, yi = λ hx, yi
and
hx, M yi = λ′ hx, yi .
Thus

λ − λ′ hx, yi = 0.

Since λ 6= λ′ , by assumption, it follows that hx, yi = 0, from which the lemma follows.

⋄

So, Hermitian and in particular real symmetric matrices have real eigenvalues and the eigenvectors
corresponding to to different eigenvalues are orthogonal. We won’t talk about complex numbers
and complex matrices for the rest of the term. (Actually, with one exception since we need to
establish that the entries of the eigenvectors are not complex-valued.)

26

M. W. Mahoney

2.4

Consequences of these two results

So far, we haven’t said anything about a full set of orthogonal eigenvectors, etc., since, e.g., all
of the eigenvectors could be the same or something funny like that. In fact, we will give a few
counterexamples to show how the niceness results we establish in this class and the next class fail
to hold for general matrices. Far from being pathologies, these examples will point to interesting
ways that spectral methods and/or variants of spectral method ideas do or do not work more
generally (e.g., periodicity, irreducibility etc.)
Now, let’s restrict ourselves to real-valued matrices, in which case Hermitian matrices are just
symmetric matrices. With the exception of some results next time on positive and non-negative
matrices, where we will consider complex-valued things, the rest of the semester will consider realvalued matrices. Today and next time, we are only talking about complex-valued matrices to set the
results that underlie spectral methods in a more general context. So, let’s specialize to real-values
matrices.
First, let’s use the above results to show that we can get a full set of (orthogonalizable) eigenvectors.
This is a strong “niceness” result, for two reasons: (1) there is a full set of eigenvectors; and (2)
that the full set of eigenvectors can be chosen to be orthogonal. Of course, you can always get
a full set of orthogonal vectors for Rn —just work with the canonical vectors or some other set of
vectors like that. But what these results say is that for symmetric matrices we can also get a full
set of orthogonal vectors that in some sense have something to do with the symmetric matrix under
consideration. Clearly, this could be of interest if we want to work with vectors/functions that are
in some sense adapted to the data.
Let’s start with the following result, which says that given several (i.e., at least one) eigenvector,
then we can find another eigenvector that is orthogonal to it/them. Note that the existence of
at least one eigenvector follows from the existence of at least one eigenvalue, which we already
established.
Lemma 3. Let M ∈ Rn×n be a real symmetric matrix, and let x1 , . . . , xk , where 1 ≤ k < n,
be orthogonal eigenvectors of M . Then, there is an eigenvector xk+1 of M that is orthogonal to
x1 , . . . , xk .
Proof: Let V be the (n − k)-dimensional subspace of Rn that contains all vectors orthogonal to
x1 , . . . , xk . Then, we claim that: for all x ∈ V , we have that M x ∈ V . To prove the claim, note
that for all i ∈ [k], we have that
hxi , M xi = xTi M x = (M xi )T x = λi xi x = λi hxi , xi = 0,
where xi is one of the eigenvectors assumed to be given.
Next, let
• B ∈ Rn×(n−k) be a matrix consisting of the vectors b1 , . . . , bn−k that form an orthonormal
basis for V . (This takes advantage of the fact that Rn has a full set of exactly n orthogonal
vectors that span it—that are, of course, not necessarily eigenvectors.)
• B ′ = B T . (If B is any matrix, then B ′ is a matrix such that, for all y ∈ V , we have that B ′ y
is an (n − k)-dimensional vector such that BB ′ y = y. I think we don’t loose any generality
by taking B to be orthogonal.)

Lecture Notes on Spectral Graph Methods

27

• λ be a real eigenvalue of the real symmetric matrix
M ′ = B ′ M B ∈ R(n−k)×(n−k) ,
with y a corresponding real eigenvector of M . I.e., M ′ y = λy.
Then,
B ′ M By = λy,
and so
BB ′ M By = λBy,
from which if follows that
M By = λBy.
The last equation follows from the second-to-last since By ⊥ {x1 , . . . , xk }, from which it follows
that M By ⊥ {x1 , . . . , xk }, by the above claim, and thus BB ′ M By = M By. I.e., this doesn’t
change anything since BB ′ ξ = ξ, for ξ in that space.
So, we can now construct that eigenvector. In particular, we can choose xk+1 = By, and we have
that M xk+1 = λxk+1 , from which the lemma follows.
⋄
Clearly, we can apply the above lemma multiple times. Thus, as an important aside, the following
“spectral theorem” is basically a corollary of the above lemma.
Theorem 1 (Spectral Theorem). Let M ∈ Rn×n be a real symmetric matrix, and let λ1 , . . . , λn
be its real eigenvalues, including multiplicities. Then, there are n orthonormal vectors x1 , . . . , xn ,
with xi ∈ Rn , such that xi is an eigenvector corresponding to λi , i.e., M xi = λi xi .
A few comments about this spectral theorem.
• This theorem and theorems like this are very important and many generalizations and variations of it exist.
• Note the wording: there are n vectors “such that xi is an eigenvector corresponding to λi .”
In particular, there is no claim (yet) about uniqueness, etc. We still have to be careful about
that.
• From this we can derive several other things, some of which we will mention below.
Someone asked in class about the connection with the SVD. The equations M xi = λi xi , for all
λi , can be written as M X = XΛ, or as M = XΛX T , since X is orthogonal. The SVD writes
an arbitrary m × n matrix A a A = U ΣV T , where U and V are orthogonal and Σ is diagonal
and non-negative. So, the SVD is a generalization or variant of this spectral theorem for realvalued square matrices to general m × n matrices. It is not true, however, that the SVD of even a
symmetric matrix gives the above theorem. It is true by the above theorem that you can write a
symmetric matrix as M = XΛX T , where the eigenvectors Λ are real. But they might be negative.
For those matrices, you also have the SVD, but there is no immediate connection. On the other
hand, some matrices have all Λ positive/nonnegative. They are called SPD/SPSD matrices, and
form them the eigenvalue decomposition of the above theorem essentially gives the SVD. (In fact,

28

M. W. Mahoney

this is sometimes how the SVD is proven—take a matrix A and write the eigenvalue decomposition
of the SPSD matrices AAT and AT A.) SPD/SPSD matrices are important, since they are basically
covariance or correlation matrices; and several matrices we will encounter, e.g., Laplacian matrices,
are SPD/SPSD matrices.
We can use the above lemma to provide the following variational characterization of eigenvalues,
which will be very important for us.
Theorem 2 (Variational Characterization of Eigenvalues). Let M ∈ Rn×n be a real symmetric
matrix; let λl ≤ · · · ≤ λn be its real eigenvalues, containing multiplicity and sorted in nondecreasing
order; and let x1 , . . . , xk , for k < n be orthonormal vectors such that M xi = λi xi , for i ∈ [k]. Then
λk+1 =

min

x∈Rn {~0}
x⊥xi ∀i∈[k]

xT M x
,
xT x

and any minimizer of this is an eigenvector of λk+1 .
Proof: First, by repeatedly applying the above lemma, then we get n − k orthogonal eigenvectors
that are also orthogonal to x1 , . . . , xk . Next, we claim that the eigenvalues of this system of n
orthogonal eigenvectors include all eigenvalues of M . The proof is that if there were any other
eigenvalues, then its eigenvector would be orthogonal to the other n eigenvectors, which isn’t
possible, since we already have n orthogonal vectors in Rn .
Call the additional n − k vectors xk+1 , . . . , xn , where xi is an eigenvector of λi . (Note that we are
inconsistent on whether subscripts mean elements of a vectors or different vectors themselves; but
it should be clear from context.) Then, consider the minimization problem
min

x∈Rn {~0}
x⊥xi ∀i∈[k]

xT M x
xT x

The solution x ≡ xk+1 is feasible, and it has cost λk+1 , and so min ≤ λk+1 .
Now, consider any arbitrary feasible solution x, and write it as
x=

n
X

αi xi .

i=k+1

The cost of this solution is
Pn
Pn
2
2
i=k+1 λi αi
i=k+1 αi
Pn
P
≥
λ
k+1
n
2
2 = λk+1 ,
i=k+1 αi
i=k+1 αi

and so min ≥ λk+1 . By combining the above, we have that min = λk+1 .
Note that is x is a minimizer of this expression, i.e., if the cost of x equals λk+1 , then ai = 0 for
all i such that λi > λk+1 , and so x is a linear combination of eigenvectors of λk+1 , and so it itself
is an eigenvector of λk+1 .
⋄
Two special cases of the above theorem are worth mentioning.

Lecture Notes on Spectral Graph Methods
• The leading eigenvector.
λ1 =

xT M x
T
x∈Rn {~0} x x
min

• The next eigenvector.
λ2 =

29

min

x∈Rn {~0},x⊥x1

xT M x
,
xT x

where x1 is a minimizer of the previous expression.

2.5

Some things that were skipped

Spielman and Trevisan give two slightly different versions of the variational characterization and
Courant-Fischer theorem, i.e., a min-max result, which might be of interest to present.
From wikipedia, there is the following discussion of the min-max theorem which is nice.
• Let A ∈ Rn×n be a Hermitian/symmetric matrix, then the Rayleigh quotient RA : Rn {0} →
R is RA (x) = hAx,xi
hx,xi , or equivalently fA (x) = hAx, xi : kxk2 = 1.
• Fact: for Hermitian matrices, the range of the continuous function RA (x) or fA (x) is a
compact subset [a, b] of R. The max b and min a are also the largest and smallest eigenvalue
of A, respectively. The max-min theorem can be viewed as a refinement of this fact.
•

Theorem 3. If A ∈ Rn×n is Hermitian with eigenvalues λ1 ≥ · · · ≥ λk ≥ · · · , then
λk = max{min{RA (x) : x ∈ U, x 6= 0}, dim(U ) = k},
and also
λk = min{max{RA (x) : x ∈ U, x 6= 0}, dim(U ) = n − k + 1}.

• In particular,
for all x ∈ Rn {0}.

λn ≤ RA (x) ≤ λ1 ,

• A simpler formulation for the max and min is
λ1 = max{RA (x) : x 6= 0}

λn = min{RA (x) : x 6= 0}

Another thing that follows from the min-max theorem is the Cauchy Interlacing Theorem. See
Spielman’s 9/16/09 notes and Wikipedia for two different forms of this. This can be used to
control eigenvalues as you make changes to the matrix. It is useful, and we may revisit this later.
And, finally, here is counterexample to these results in general. Lest one thinks that these niceness
results always hold, here is a simple non-symmetric matrix.


0 1
A=
0 0
(This is an example of a nilpotent matrix.)

30

M. W. Mahoney

Definition 7. A nilpotent matrix is a square matrix A such that Ak = 0 for some k ∈ Z+ .
More generally, any triangular matrix with all zeros on the diagonal; but it could also be a dense
matrix.)
For this matrix A, we can define RA (x) as with the Rayleigh quotient. Then,
• The only eigenvalue of A equals 0.
• The maximum value of RA (x) is equal to 21 , which is larger that 0.
So, in particular, the Rayleigh quotient doesn’t say much about the spectrum.

2.6

Summary

Today we showed that any symmetric matrix (e.g., adjacency matrix A of an undirected graph,
Laplacian matrix, but more generally) is nice in that it has a full set of n real eigenvalues and a
full set of n orthonormal eigenvectors.
Next time, we will ask what those eigenvectors look like, since spectral methods make crucial use
of that. To do so, we will consider a different class of matrices, namely positive or nonnegative (not
PSD or SPSD, but element-wise positive or nonnegative) and we will look at the extremal, i.e., top
or bottom, eigenvectors.

Lecture Notes on Spectral Graph Methods

3

31

(01/29/2015): Basic Matrix Results (2 of 3)

Reading for today.
• Same as last class.

3.1

Review and overview

Last time, we considered symmetric matrices, and we showed that is M is an n × n real-valued
matrix, then the following hold.
• There are n eigenvalues, counting multiplicity, that are all real.
• The eigenvectors corresponding to different eigenvalues are orthogonal.
• Given k orthogonal eigenvectors, we can construct one more that is orthogonal to those k,
and thus we can iterate this process to get a full set of n orthogonal eigenvectors
• This spectral theorem leads to a variational characterization of eigenvalues/eigenvectors and
other useful characterizations.
These results say that symmetric matrices have several “nice” properties, and we will see that
spectral methods will use these extensively.
Today, we will consider a different class of matrices and establish a different type of “niceness”
result, which will also be used extensively by spectral methods. In particular, we want to say
something about how eigenvectors, and in particular the extremal eigenvectors, e.g., the largest
one or few or the smallest one of few “look like.” The reason is that spectral methods—both
vanilla and non-vanilla variants—will rely crucially on this; thus, understanding when and why this
is true will be helpful to see how spectral methods sit with respect to other types of methods, to
understand when they can be generalized, or not, and so on.
The class of matrices we will consider are positive matrices as well as related non-negative matrices.
By positive/non-negative, we mean that this holds element-wise. Matrices of this form could be,
e.g., the symmetric adjacency matrix of an undirected graph, but they could also be the nonsymmetric adjacency matrix of a directed graph. (In the latter case, of course, it is not a symmetric
matrix, and so the results of the last class don’t apply directly.) In addition, the undirected/directed
graphs could be weighted, assuming in both cases that weights are non-negative. In addition, it
could apply more generally to any positive/non-negative matrix (although, in fact, we will be able
to take a positive/non-negative matrix and interpret it as the adjacency matrix of a graph). The
main theory that is used to make statements in this context and that we will discuss today and
next time is something called Perron-Frobenius theory.

3.2

Some initial examples

Perron-Frobenius theory deals with positive/non-negative vectors and matrices, i.e., vectors and
matrices that are entry-wise positive/nonnegative. Before proceeding with the main results of
Perron-Frobenius theory, let us see a few examples of why it might be of interest and when it
doesn’t hold.

32

M. W. Mahoney

Example. Non-symmetric and not non-negative matrix. Let’s start with the following matrix,
which is neither positive/non-negative nor symmetric.


0 −1
A=
.
2 3
The characteristic polynomial of this matrix is
−λ −1
2 3−λ
= −λ(3 − λ) + 2

det (A − λI) =

= λ2 − 3λ + 2

= (λ − 1) (λ − 2) ,

from which if follows that the eigenvalues are 1 and 2. Plugging in λ = 1, we get x1 + x2 = 0, and
so the eigenvector corresponding to λ = 1 is


1
1
xλ=1 = √
.
2 −1
Plugging in λ = 2, we get 2x1 + x2 = 0, and so the eigenvector corresponding to λ = 2 is


1
1
.
xλ=1 = √
5 −2
So, this matrix has two eigenvalues and two eigenvectors, but they are not orthogonal, which is ok,
since A is not symmetric.
Example. Defective matrix. Consider the following matrix, which is an example of a “defective”
matrix.


1 1
A=
.
0 1
The characteristic polynomial is

det (A − λI) =

1−λ
1
0
1−λ

= (1 − λ)2 ,

and so 1 is a double root. If we plug this in, then we get the system of equations


  
0 1
x1
0
=
,
0 0
x2
0
meaning that x2 = 0 and x1 is arbitrary. (BTW, note that the matrix that appears in that system
of equations is a nilpotent matrix. See below. From the last class, this has a value of the Rayleigh
quotient that is not in the closed interval defined by the min to max eigenvalue.) Thus, there is
only one linearly independent eigenvector corresponding to the double eigenvalue λ = 1 and it is
 
1
xλ=1 =
.
0
Example. Nilpotent matrix. Consider the following matrix,


0 1
A=
.
0 0

Lecture Notes on Spectral Graph Methods

33

The only eigenvalue of this equals zero. The eigenvector is the same as in the above example. But
this matrix has the property that if you raise it to some finite power then it equals the all-zeros
matrix.
Example. Identity. The problem above with having only one linearly independent eigenvector is
not due to the multiplicity in eigenvalues. For example, consider the following identity matrix,


1 0
A=
.
0 1
which has characteristic polynomial λ2 −1 = 0, and so which has λ = 1 as a repeated root. Although
it has a repeated root, it has two linearly independent eigenvectors. For example,
 
 
1
0
x1 =
x2 =
,
0
1
or, alternatively,
1
x1 = √
2



1
1



1
x2 = √
2



−1
1



.

This distinction as to whether there are multiple eigenvectors associated with a degenerate eigenvalue is an important distinction, and so we introduce the following definitions.
Definition 8. Given a matrix A, for an eigenvalue λi
• it’s algebraic multiplicity, denoted µA (λi ), is the multiplicity of λ as a root of the characteristic polynomial; and
• it’s geometric multiplicity, denoted γA (λi ) is the maximum number of linearly independent
eigenvectors associated with it.
Here are some facts (and terminology) concerning the relationship between the algebraic multiplicity
and the geometric multiplicity of an eigenvalue.
• 1 ≤ γA (λi ) ≤ µA (λi ).
• If µA (λi ) = 1, then λi is a simple eigenvalue.
• If γA (λi ) = µA (λi ), then λi is a semi-simple eigenvalue.
• If γA (λi ) < µA (λi ), for some i, then the matrix A is defective. Defective matrices are more
complicated since you need things like Jordan forms, and so they are messier.
P
• If i γA (λi ) = n, then A has n linearly independent eigenvectors. In this case, A is diagonalizable. I.e., we can write AQ = QΛ, and so Q−1 AQ = Λ. And conversely.

3.3

Basic ideas behind Perron-Frobenius theory

The basic idea of Perron-Frobenius theory is that if you have a matrix A with all positive entries
(think of it as the adjacency matrix of a general, i.e., possibly directed, graph) then it is “nice” in
several ways:

34

M. W. Mahoney
• there is one simple real eigenvalue of A that has magnitude larger than all other eigenvalues;
• the eigenvector associated with this eigenvalue has all positive entires;
• if you increase/decrease the magnitude of the entries of A, then that maximum eigenvalue
increases/decreases; and
• a few other related properties.

These results generalize to non-negative matrices (and slightly more generally, but that is of less
interest in general). There are a few gotchas that you have to watch out for, and those typically
have an intuitive meaning. So, it will be important to understand not only how to establish the
above statements, but also what the gotchas mean and how to avoid them.
These are quite strong claims, and they are certainly false in general, even for non-negative matrices,
without those additional assumptions. About that, note that every nonnegative matrix is the limit
of positive matrices, and so there exists an eigenvector with nonnegative components. Clearly,
the corresponding eigenvalue is nonnegative and greater or equal in absolute value. Consider the
following examples.
Example. Symmetric matrix. Consider the following matrix.


0 1
A=
.
1 0
This is a non-negative matrix, and there is an eigenvalue equal to 1. However, there exist other
eigenvalues of the same absolute value (and not strictly less) as this maximal one. The eigenvalues
are −1 and 1, both of which have absolute value 1.
Example. Non-symmetric matrix. Consider the following matrix.


0 1
.
A=
0 0
This is a matrix in which the maximum eigenvalue is not simple.
  The only root of the characteristic
1
, is not strictly positive.
polynomial is 0, and the corresponding eigenvector, i.e.,
0
These two counter examples contain the basic ideas underlying the two main gotchas that must be
dealt with when generalizing Perron-Frobenius theory to only non-negative matrices.
(As an aside, it is worth wondering what is unusual
generalized. One is

0 1 0
 0 0 1
A=
 0 0 0
0 0 0

about that latter matrix and how it can be

0
0 
,
1 
0

and there are others. These simple examples might seem trivial, but they contain several key ideas
we will see later.)
One point of these examples is that the requirement that the entries of the matrix A be strictly
positive is important for Perron-Frobenius theory to hold. If instead we only have non-negativity,
we need further assumption on A which we will see below (and in the special case of matrices
associated with graphs, the reducibility property of the matrix is equivalent to the connectedness
of the graph).

Lecture Notes on Spectral Graph Methods

3.4

35

Reducibility and types of connectedness

We get a non-trivial generalization of Peron-Frobenius theory from all-positive matrices to nonnegative matrices, if we work with the class of irreducible matrices. (We will get an even cleaner
statement if we work with the class of irreducible aperiodic matrices. We will start with the former
first, and then we will bet to the latter.)
We start with the following definition, which applied to an n × n matrix A. For those readers
familiar with Markov chains and related topics, there is an obvious interpretation we will get to,
but for now we just provide the linear algebraic definition.
Definition 9. A matrix A ∈ Rn×n is reducible if there exist a permutation matrix P such that


A11 A12
T
,
C = P AP =
0 A22
with A11 ∈ Rr×r and A22 ∈ R(n−r)×(n−r) , where 0 < r < n. (Note that the off-diagonal matrices, 0
and A12 , will in general be rectangular.) A matrix A ∈ Rn×n is irreducible if it is not reducible.
As an aside, here is another definition that you may come across and that we may point to later.
Definition 10. A nonnegative matrix A ∈ Rn×n is irreducible if ∀i, j ∈ [n]2 , ∃t ∈ N : Atij > 0.
And it is primitive if ∃t ∈ N, ∀i, j ∈ [n]2 : Atij > 0.
This is less intuitive, but I’m mentioning it since these are algebraic and linear algebraic ideas,
and we haven’t yet connected it with random walks. But later we will understand this in terms of
things like lazy random walks (which is more intuitive for most people than the gcd definition of
aperiodicity/primitiveness).
Fact: If A, a non-negative square matrix, is nilpotent (i.e., s.t. Ak = 0, for some k ∈ Z+ ,
then it is reducible. Proof: By contradiction, suppose A is irreducible, and nilpotent. Let k be
k−1
> 0 for some i, j,
the smallest k such that Ak = 0. Then we know Ak−1 6= 0. Suppose Aij
since A irreducible, we now there exist t ≥ 1 such that Atji > 0. Note all powers of A are nonk−1+t
k−1 t
k−1 t
negative, then Aii
= Ai,·
A·,i ≥ Aij
Aji > 0 which gives a contradiction, since we have
′
′
k
k
A = 0 ⇒ A = 0 ∀k ≥ k, while k − 1 + t ≥ k, but Ak−1+t 6= 0.
⋄
We start with a lemma that, when viewed the right way, i.e., in a way that is formal but not
intuitive, is trivial to prove.
Lemma 4. Let A ∈ Rn×n be a non-negative square matrix. If A is primitive, then A is reducible.
Proof: ∃∀ → ∀∃

⋄

It can be shown that the converse is false. But we can establish a sort of converse in the following
lemma. (It is a sort of converse since A and I + A are related, and in particular in our applications
to spectral graph theory the latter will essentially have an interpretation in terms of a lazy random
walk associated with the former.)
Lemma 5. Let A ∈ Rn×n be a non-negative square matrix. If A is irreducible, then I + A is
primitive.

36

M. W. Mahoney

Proof: Write out the binomial expansion
(I + A)n =

n  
X
n
k=0

k

Ak .

This has all positive entries since A is irreducible, i.e., it eventually has all positive entries if k is
large enough.
⋄
Note that a positive matrix may be viewed as the adjacency matrix of a weighted complete graph.
Let’s be more precise about directed and undirected graphs.
Definition 11. A directed graph G(A) associated with an n × n nonnegative matrix A consists of
n nodes/vertices P1 , . . . , Pn , where an edge leads from Pi to Pj iff Aij 6= 0.
Since directed graphs are directed, the connectivity properties are a little more subtle than for
undirected graphs. Here, we need the following. We will probably at least mention other variants
later.
Definition 12. A directed graph G is strongly connected if ∀ ordered pairs (Pi , Pj ) of vertices of
G, ∃ a path, i.e., a sequence of edges, (Pi , Pl1 ), (Pl1 , Pl2 ), . . . , (Plr−1 , Pj ), which leads from Pi to Pj .
The length of the path is r.
Fact: The graph G(Ak ) of a nonnegative matrix A consists of all paths of G(A) of length k (i.e.
there is an edge from i to j in G(Ak ) iff there is a path of length k from i to j in G).
Keep this fact in mind since different variants of spectral methods involve weighting paths of
different lengths in different ways.
Here is a theorem that connects the linear algebraic idea of irreducibility with the graph theoretic
idea of connectedness. Like many things that tie together notions from two different areas, it can
seem trivial when it is presented in such a way that it looks obvious; but it really is connecting two
quite different ideas. We will see more of this later.
Theorem 4. An n × n matrix A is irreducible iff the corresponding directed graph G(A) is strongly
connected.
Proof: Let A be an irreducible matrix. Assume, for contradiction, that G(A) is not strongly
connected. Then, there exists an ordered pair of nodes, call them (Pi , Pj ), s.t. there does not exist
a connection from Pi to Pj . In this case, let S1 be the set of nodes connected to Pi , and let S2
be the remainder of the nodes. Note that there is no connection between any nodes Pℓ ∈ S2 and
any node Pq ∈ S1 , since otherwise we sould have Pℓ ∈ S1 . And note that both sets are nonempty,
since Pj ∈ S1 and Pi ∈ S2 . Let r = |S1 | and n − r = |S2 |. Consider a permutation transformation
C = P AP T that reorders the nodes of G(A) such that
(
P1 , P2 , · · · , Pr ∈ S1
Pr+1 , Pr+2 , · · · , Pn ∈ S2
That is
Ckℓ = 0

(
k = r + 1, r + 2, . . . , n
∀
ℓ = 1, 2, . . . , r.

Lecture Notes on Spectral Graph Methods

37

But this is a contradiction, since A is irreducible.
Conversely, assume that G(A) is strongly connected, and assume for contradiction that A is not
irreducible. Reverse the order of the above argument, and we arrive at the conclusion that G(A)
is not strongly connected, which is a contradiction.
⋄
We conclude by noting that, informally, there are two types of irreducibility. To see this, recall
that in the definition of reducibility/irreducibility, we have the following matrix:
C = P AP

T

=



A11 A12
0 A22



.

In one type, A12 6= 0: in this case, we can go from the first set to the second set and get stuck in
some sort of sink. (We haven’t made that precise, in terms of random walk interpretations, but
there is some sort of interaction between the two groups.) In the other type, A12 = 0: in this
case, there are two parts that don’t talk with each other, and so essentially there are two separate
graphs/matrices.

3.5

Basics of Perron-Frobenius theory

Let’s start with the following definition. (Note here that we are using subscripts to refer to elements
of a vector, which is inconsistent with what we did in the last class.)
Definition 13. A vector x ∈ Rn is positive (resp, non-negative) if all of the entries of the vector
are positive (resp, non-negative), i.e., if xi > 0 for all i ∈ [n] (resp if xi ≥ 0 for all i ∈ [n]).
A similar definition holds for m×n matrices. Note that this is not the same as SPD/SPSD matrices.
Let’s also provide the following definition.
Definition 14. Let λ1 , . . . , λn be the (real or complex) eigenvalues of a matrix A ∈ Cn×n . Then
the spectral radius ρA = ρ(A) = maxi (|λi |)
Here is a basic statement of the Perron-Frobenius theorem.
Theorem 5 (Perron-Frobenius). Let A ∈ Rn×n be an irreducible non-negative matrix. Then,
1. A has a positive real eigenvalue equal to its spectral radium.
2. That eigenvalue ρA has algebraic and geometric multiplicity equal to one.
3. The one eigenvector x associated with the eigenvalue ρA has all positive entries.
4. ρA increases when any entry of A increases.
5. There is no other non-negative eigenvector of A different than x.
6. If, in addition, A is primitive, then each other eigenvalue λ of A satisfies |λ| < ρA .

38

M. W. Mahoney

Before giving the proof, which we will do next class, let’s first start with some ideas that will suggest
how to do the proof.
Let P = (I + A)n . Since P is positive, it is true that for every non-negative and non-null vector
v, that we have that P v > 0 element-wise. Relatedly, if v ≤ w element-wise, and v 6= w, then
P v < P w.
Let
Q = {x ∈ Rn s.t. x ≥ 0, x 6= 0}
be the nonnegative orthant, excluding the origin. In addition, let
C = {x ∈ Rn s.t. x ≥ 0, ||x|| = 1} ,
where || · || is any vector norm. Clearly, C is compact, i.e., closed and bounded.
Then, for all z ∈ Q, we can define the following function: let
f (z) = max {s ∈ R : sz ≤ Az} =

min

1≤i≤n,zi 6=0

(Az)i
zi

Here are facts about f .
• f (rz) = f (z), for all r > 0.
• If Az = λz, i.e., if (λ, z) is an eigenpair, then f (z) = λ.
• If sz ≤ Az, then sP z ≤ P Az = AP z, where the latter follows since A and P clearly commute.
So,
f (z) ≤ f (P z).
In addition, if z is not an eigenvector of A, then sz 6= Az, for all s; and sP z < AP z. From
the second expression for f (z) above, we have that in this case that
f (z) < f (P z),
i.e., an inequality in general but a strict inequality if not an eigenvector.
This suggests an idea for the proof: look for a positive vector that maximizes the function f ; show
it is an eigenvector we want in the theorem; and show that it established the properties stated in
the theorem.

Lecture Notes on Spectral Graph Methods

4

39

(02/03/2015): Basic Matrix Results (3 of 3)

Reading for today.
• Same as last class.

4.1

Review and overview

Recall the basic statement of the Perron-Frobenius theorem from last class.
Theorem 6 (Perron-Frobenius). Let A ∈ Rn×n be an irreducible non-negative matrix. Then,
1. A has a positive real eigenvalue λmax ; which is equal to the spectral radius; and λmax has an
associated eigenvector x with all positive entries.
2. If 0 ≤ B ≤ A, with B 6= A, then every eigenvalue σ of B satisfies |σ| < λmax = ρA . (Note
that B does not need to be irreducible.) In particular, B can be obtained from A by zeroing
out entries; and also all of the diagonal minors A(i) obtained from A by deleting the ith
row/column have eigenvalues with absolute value strictly less than λmax = ρA . Informally,
this says: ρA increases when any entry of A increases.
3. That eigenvalue ρA has algebraic and geometric multiplicity equal to one.
4. If y ≥ 0, y 6= 0 is a vector and µ is a number such that Ay ≤ µy, then y > 0 and µ ≥ λmax ;
with µ = λmax iff y is a multiple of x. Informally, this says: there is no other non-negative
eigenvector of A different than x.
5. If, in addition, A is primitive/aperiodic, then each other eigenvalue λ of A satisfies |λ| < ρA .
6. If, in addition, A is primitive/aperiodic, then
lim

t→∞



t
1
A = xy T ,
ρA

where x and y are positive eigenvectors of A and AT with eigenvalue ρA , i.e., Ax = ρA x and
AT y = ρA y (i.e., y T A = ρA y T ), normalized such that xT y = 1.
Today, we will do three things: (1) we will prove this theorem; (2) we will also discuss periodicity/aperiodicity issues; (3) we will also briefly discuss the first connectivity/non-connectivity result
for Adjacency and Laplacian matrices of graphs that will use the ideas we have developed in the
last few classes.
Before proceeding, one note: an interpretation of a matrix B generated from A by zeroing out an
entry or an entire row/column is that you can remove an edge from a graph or you can remove
a node and all of the associated edges from a graph. (The monotonicity provided by that part of
this theorem will be important for making claims about how the spectral radius behaves when such
changes are made to a graph.) This obviously holds true for Adjacency matrices, and a similar
statement also holds true for Laplacian matrices.

40

M. W. Mahoney

4.2

Proof of the Perron-Frobenius theorem

We start with some general notation and definitions; then we prove each part of the theorem in turn.
Recall from last time that we let P = (I + A)n and thus P is positive. Thus, for every nonnegative and non-null vector v, then we have that P v > 0 element-wise; and (equivalently) if v ≤ w
element-wise, and v 6= w, then we have that P v < P w. Recall also that we defined
Q = {x ∈ Rn s.t. x ≥ 0, x 6= 0}

C = {x ∈ Rn s.t. x ≥ 0, ||x|| = 1} ,
where || · || is any vector norm. Note in particular that this means that C is compact, i.e., closed
and bounded. Recall also that, for all z ∈ Q, we defined the following function: let
f (z) = max {s ∈ R : sz ≤ Az} =

min

1≤i≤n,zi 6=0

(Az)i
zi

Finally, recall several facts about the function f .
• f (rz) = f (z), for all r > 0.
• If Az = λz, i.e., if (λ, z) is an eigenpair, then f (z) = λ.
• In general, f (z) ≤ f (P z); and if z is not an eigenvector of A, then f (z) < f (P z). (The reason
for the former is that if sz ≤ Az, then sP z ≤ P Az = AP z. The reason for the latter is that
in this case sz 6= Az, for all s, and sP z < AP z, and by considering the second expression for
f (z) above.)
We will prove the theorem in several steps.

4.3

Positive eigenvalue with positive eigenvector.

Here, we will show that there is a positive eigenvalue λ∗ and that the associated eigenvector x∗ is
a positive vector.
To do so, consider P (C), the image of C under the action of the operator P . This is a compact
set, and all vectors in P (C) are positive. By the second expression in definition of f (·) above, we
have that f is continuous of P (C). Thus, f achieves its maximum value of P (C), i.e., there exists
a vector x ∈ P (C) such that
f (x) = sup f (P z).
z∈C

Since f (z) ≤ f (P z), the vector x realizes the maximum value fmax of f on Q. So,
fmax = f (x) ≤ f (P x) ≤ fmax .
Thus, from the third property of f above, x is an eigenvector of A with eigenvalue fmax . Since
x ∈ P (C), then x is a positive vector; and since Ax > 0 and Ax = fmax x, it follows that fmax > 0.
(Note that this result shows that fmax = λ∗ is achieved on an eigenvector x = x∗ , but it doesn’t
show yet that it is equal to the spectral radius.)

Lecture Notes on Spectral Graph Methods

4.4

41

That eigenvalue equals the spectral radius.

Here, we will show that fmax = ρA , i.e., fmax equals the spectral radius.
To do so, let z ∈ Cn be an eigenvector of A with eigenvalue λ ∈ C; and let |z| be a vector, each
entry of which equals |zi |. Then, |z| ∈ Q.
Pn
We claim that |λ||z| ≤ A|z|.
To
establish
the
claim,
rewrite
it
as
|λ||z
|
≤
i
k=1 Aik |zk |. Then,
P
since Az = λz, i.e., λzi = nk=1 Aik zk , and since Aik ≥ 0, we have that
|λ||z| =

from which the claim follows.

n
X
k=1

Aik zk ≤

n
X
k=1

Aik |zk |,

i
Thus, by the definition of f (i.e., since f (z) = min (Az)
(z)i , we have that |λ| ≤ f (|z|). Hence,
|λ| ≤ fmax , and thus ρA ≤ fmax (where ρA is the spectral radius). Conversely, from the above, i.e.,
since fmax is an eigenvalue it must be ≤ the maximum eigenvalue, we have that fmax ≤ ρA . Thus,
fmax = ρA .

4.5

An extra claim to make.

We would like to establish the following result:
f (z) = fmax ⇒ (Az = fmax z and z > 0) .
To establish this result, observe that above it is shown that: if f (z) = fmax , then f (z) = f (P z).
Thus, z is an eigenvector of A for eigenvalue fmax . It follows that P z = λz, i.e., that z is also an
eigenvector of P . Since P is positive, we have that P z > 0, and so z is positive.

4.6

Monotonicity of spectral radius.

Here, we would like to show that 0 ≤ B ≤ A and B 6= A implies that ρB < ρA . (Recall that B
need not be irreducible, but A is.)
To do so, suppose that Bz = λz, with z ∈ Cn and with λ ∈ C. Then,
|λ||z| ≤ B|z| ≤ A|z|,
from which it follows that
and thus ρB ≤ ρA .

|λ| ≤ fA (|z|) ≤ ρA ,

Next, assume for contradiction that |λ| = ρA . Then from the above claim (in Section 4.5), we have
that fA (z) = ρA . Thus from above it follows that |z| is an eigenvector of A for the eigenvalue ρA
and also that z is positive. Hence, B|z| = A|z|, with z > 0; but this is impossible unless A = B.
Remark. Replacing the ith row/column of A by zeros gives a non-negative matrix A(i) such that
0 ≤ A(i) ≤ A. Moreover, A(i) 6= A, since the irreducibility of A precludes the possibility that all
entries in a row are equal to zero. Thus, for all matrices A(i) that are obtained by eliminating the
ith row/column of A, the eigenvalues of A(i) < ρ.

42

M. W. Mahoney

4.7

Algebraic/geometric multiplicities equal one.

Here, we will show that the algebraic and geometric multiplicity of λmax equal 1. Recall that the
geometric multiplicity is less than or equal to the algebraic multiplicity, and that both are at least
equal to one, so it suffices to prove this for the algebraic multiplicity.
Before proceeding, also define the following: given a square matrix A:
• Let A(i) be the matrix obtained by eliminating the ith row/column. In particular, this is a
smaller matrix, with one dimension less along each column/row.
• Let Ai be the matrix obtained by zeroing out the ith row/column. In particular, this is a
matrix of the same size, with all the entries in one full row/column zeroed out.
To establish this result, here is a lemma that we will use; its proof (which we won’t provide) boils
down to expanding det (Λ − A) along the ith row.
Lemma 6. Let A be a square matrix, and let Λ be a diagonal matrix of the same size with λ1 , . . . , λn
(as variables) along the diagonal. Then,

∂
det (Λ − A) = det Λ(i) − A(i) ,
∂λi

where the subscript (i) means the matrix obtained by eliminating the ith row/column from each
matrix.
Next, set λi = λ and apply the chain rule from calculus to get
n

X

d
det λI − A(i) .
det (λI − A) =
dλ
i=1

Finally, note that

det (λI − Ai ) = λdet λI − A(i) .


But by what we just proved (in the Remark at the end of last page), we have that det ρA I − A(i) >
0. Thus, the derivative of the characteristic polynomial of A is nonzero at ρA , and so the algegraic
multiplicity equals 1.

4.8

No other non-negative eigenvectors, etc.

Here, we will prove the claim about other non-negative vectors, including that there are no other
non-negative eigenvectors.
To start, we claim that: 0 ≤ B ≤ A ⇒ fmax (B) ≤ fmax (A). (This is related to but a little different
than the similar result we had above.) To establish the claim, note that if z ∈ Q is s.t. sz ≤ Bz,
then sz ≤ Az (since Bz ≤ Az), and so fB (z) ≤ fA (z), for all z.
We can apply that claim to AT , from which it follows that AT has a positive eigenvalue, call it η.
So, there exists a row vector, w > 0 s.t. wT A = ηwT . Recall that x > 0 is an eigenvector of A with
maximum eigenvalue λmax . Thus,
wT Ax = ηwT x = λmax wT x,

Lecture Notes on Spectral Graph Methods

43

and thus η = λmax (since wT x > 0).
Next, suppose that y ∈ Q and Ay ≤ µy. Then,
λmax wT y = wT Ay ≤ µwT y,
from which it follows that λmax ≤ µ. (This is since all components of w are positive and some
components of y is positive, and so wT y > 0).
In particular, if Ay = µy, then µ = λmax .
Further, if y ∈ Q and Ay ≤ µy, then µ ≥ 0 and y > 0. (This is since 0 < P y = (I + A)n−1 y ≤
(1 + µ)n−1 y.)
This proves the first two parts of the result; now, let’s prove the last part of the result.
If µ = λmax , then wT (Ay−λmax y) = 0. But, Ay−λmax y ≤ 0. So, given this, from wT (Ay − λmax y) =
0, it follows that Ay = λmax y. Since y must be an eigenvector with eigenvalue λmax , the last result
(i.e., that y is a scalar multiple of x) follows since λmax has multiplicity 1.
To establish the converse direction march through these steps in the other direction.

4.9

Strict inequality for aperiodic matrices

Here, we would like to establish the result that the eigenvalue we have been talking about is strictly
larger in magnitude than the other eigenvalues, under the aperiodicity assumption.
To do so, recall that the tth powers of the eigenvalues of A are the eigenvalues of At . So, if we want
to show that there does not exist eigenvalues of a primitive matrix with absolute value = ρA , other
than ρA , then it suffices to prove this for a positive matrix A.
Let A be a positive matrix, and suppose that Az = λz, with z ∈ Cn , λ ∈ C, and |λ| = ρA , in which
case the goal is to show λ < ρA .
(We will do this by showing that any eigenvector with eigenvalue equal in magnitude to ρA is the
top eigenvalue.) (I.e., we will show that such a z equals |z| and thus there is no other one with ρA .)
Then,
ρA |z| = |Az| ≤ A|z|,
from which it follows that
ρA ≤ f (|z|) ≤ ρA ,
which implies that f (|z|) = ρA . From a result above, this implies that |z| is an eigenvector of A
with eigenvalue ρA . Moreover, |Az| = A|z|. In particular,
n
X
i=1

A1i zi =

n
X
i=1

A1i |zi |.

Since all of the entries of A are positive, this implies that there exists a number u ∈ C (with |u| = 1)
s.t. for all i ∈ [n], we have that zi = u|zi |. Hence, z and |z| are collinear eigenvectors of A. So, the
corresponding eigenvalues of λ and ρ are equal, as required.

44

M. W. Mahoney

4.10

Limit for aperiodic matrices

Here, we would like to establish the limiting result.
To do so, note that AT has the same spectrum (including multiplicities) as A; and in particular
the spectral radius of AT equals ρA .
Moreover, since AT is irreducible (a consequence of being primitive), we can apply the PerronFrobenius theorem toPit to get yA = ρA y. Here y is determined up to a scalar multiple, and so let’s
choose it s.t. xT y = ni=1 xi yi = 1.

Next, observe that we can decompose the n-dimensional vector space Rn into two parts,
Rn = R ⊕ N,

where both R and N are invariant under the action of A. To do this, define the rank-one matrix
H = xy T , and:
• let R be the image space of H; and
• let N be the null space of H.
Note that H is a projection matrix (in particular, H 2 = H), and thus I − H is also a projection
matrix, and the image space of I − H is N . Also,
AH = Axy T = ρA xy T = xρA y T = xy T A = HA.
So, we have a direct sum decomposition of the space Rn into R ⊕ N , and this decomposition is
invariant under the action of A.
Given this, observe that the restriction of A to N has all of its eigenvalues strictly less that ρA
in absolute value, while the restriction of A to the one-dimensional space R is simply a multiplication/scaling by ρA . So, if P is defined to be P = ρ1A A, then the restriction of P to N has its
eigenvalues < 1 in absolute value. This decomposition is also invariant under all positive integral
k
powers of P . So, the restriction
 of P
 to N tends to zero as k → ∞, while the restriction of P to
R is the identity. So, limt→∞

4.11

1
ρA A

t

= H = xy T .

Additional discussion form periodicity/aperiodic and cyclicity/primitiveness

Let’s switch gears and discuss the periodicity/aperiodic and cyclicity/primitiveness issues.
(This is an algebraic characterization, and it holds for general non-negative matrices. I think that
most people find this less intuitive that the characterization in terms of connected components, but
it’s worth at least knowing about it.)
Start with the following definition.
Definition 15. The cyclicity of an irreducible non-negative matrix A is the g.c.d. (greatest common
denominator) of the length of the cycles in the associated graph.

Lecture Notes on Spectral Graph Methods

45

Let’s let Nij be a positive subset of the integers s.t.
{t ∈ N s.t. (At )ij > 0},
that is, it is the values of t ∈ N s.t. the matrix At ’s (i, j) entry is positive (i.e. exists a path from
i to j of length t) . Then, to define γ to be the cyclicity of A, first define γi = gcd (Nii ), and
then clearly γ = gcd ({γi s.t. i ∈ V }). Note that each Nii is closed under addition, and so it is a
semi-group.
Here is a lemma from number theory (that we won’t prove).
Lemma 7. A set N of positive integers that is closed under addition contains all but a finite number
of multiples of its g.c.d.
From this it follows that ∀i ∈ [n], γi = γ.
The following theorem (which we state but won’t prove) provides several related conditions for an
irreducible matrix to be primitive.
Theorem 7. Let A be an irreducible matrix. Then, the following are equivalent.
1. The matrix A is primitive.
2. All of the eigenvalues of A different from its spectral radius ρA satisfy |λ| < ρA .
t

3. The sequence of matrices ρ1A A converges to a positive matrix.
4. There exists an i ∈ [n] s.t., γi = 1.

5. The cyclicity of A equals 1.
For completeness, note that sometimes one comes across the following definition.
Definition 16. Let A be an irreducible non-negative square matrix. The period of A is the g.c.d.
of all natural numbers m s.t. (Am )ii > 0 for some i. Equivalently, the g.c.d. of the lengths of closed
directed paths of the directed graph GA associated with A.
Fact. All of the statements of the Perron-Frobenius theorem for positive matrices remain true for
irreducible aperiodic matrices. In addition, all of those statements generalize to periodic matrices.
The the main difference in this generalization is that for periodic matrices the “top” eigenvalue
isn’t “top” any more, in the sense that there are other eigenvalues with equal absolute value that
are different: they equal the pth roots of unity, where p is the periodicity.
Here is an example of a generalization.
Theorem 8. Let A be an irreducible non-negative n × n matrix, with period equal to h and spectral
radius equal to ρA = r. Then,
1. r > 0, and it is an eigenvalue of A.
2. r is a simple eigenvalue, and both its left and right eigenspace are one-dimensional.

46

M. W. Mahoney
3. A has left/right eigenvectors v/w with eigenvalue r, each of which has all positive entries.
4. A has exactly h complex eigenvalues with absolute value = r; and each is a simple root of the
characteristic polynomial and equals the r · hth root of unity.
5. If h > 0, then there exists a permutation matrix P s.t.

0 A12
0

0
A
23


..
..
P AP T = 
.
.

 0
0 Ah−1,h
Ah1 0
0

4.12






.



(4)

Additional discussion of directness, periodicity, etc.

Today, we have been describing Perron-Frobenius theory for non-negative matrices. There are a lot
of connections with graphs, but the theory can be developed algebraically and linear-algebraically,
i.e., without any mention of graphs. (We saw a hint of this with the g.c.d. definitions.) In particular,
Theorem 8 is a statement about matrices, and it’s fair to ask what this might say about graphs we
will encounter. So, before concluding, let’s look at it and in particular at Eqn. (4) and ask what
that might say about graphs—and in particular undirected graphs—we will consider.
To do so, recall that the Adjacency Matrix of an undirected graph is symmetric; and, informally,
there are several different ways (up to permutations, etc.) it can “look like.” In particular:
• It can look like this:

A=



A11 A12
AT12 A22



,

(5)

where let’s assume that all-zeros blocks are represented as 0 and so each Aij is not all-zeros.
This corresponds to a vanilla graph you would probably write down if you were asked to write
down a graph.
• It can look like this:

A=



A11 0
0 A22



,

(6)

in which case the corresponding graph is not connected.
• It can even look like this:

A=



0 A12
A21 0



,

(7)

which has the interpretation of having two sets of nodes, each of which has edges to only the
other set, and which will correspond to a bipartite graph.
• Of course, it could be a line-like graph, which would look like a tridiagonal banded matrix,
which is harder for me to draw in latex, or it can look like all sorts of other things.
• But it cannot look like this:

A=



A11 A12
0 A22



,

(8)

Lecture Notes on Spectral Graph Methods
and it cannot look like this:
A=



0 A12
0 0

47


,

(9)

where recall we are assuming that each Aij is not all-zeros. In both of these cases, these
matrices are not symmetric.
In light of today’s results and looking forward, it’s worth commenting for a moment on the relationship between Eqns. (4) and Eqns (5) through (9).
Here are a few things to note.
• One might think from Eqns. (4) that periodicity means that that the graph is directed and
so if we work with undirected graphs we can ignore it. That’s true if the periodicity is 3 or
more, but note that the matrix of Eqn (7) is periodic with period equal to 2. In particular,
Eqn (7) is of the form of Eqn. (4) if the period h = 2. (It’s eigenvalues are real, which they
need to be since the matrix is symmetric, since the complex “2th roots of unity,” which equal
±1, are both real.)
• You can think of Eqn. (6) as a special case of Eqn. (8), with the A12 block equal to 0, but it
is not so helpful to do so, since its behavior is very different than for an irreducible matrix
with A12 6= 0.
• For directed graphs, e.g., the graph that would correspond to Eqn. (8) (or Eqn. (9)), there
is very little spectral theory. It is of interest in practice since edges are often directed.
But, most spectral graph methods for directed graphs basically come up—either explicitly
or implicitly—with some sort of symmetrized version of the directed graph and then apply
undirected spectral graph methods to that symmetrized graph. (Time permitting, we’ll see
an example of this at some point this semester.)
• You can think of Eqn. (8) as corresponding to a “bow tie” picture (that I drew on the board
and that is a popular model for the directed web graph and other directed graphs). Although
this is directed, it can be made irreducible by adding a rank-one update of the form 11T to
the adjacency matrix. E.g., A → A + ǫ11T . This has a very natural interpretation in terms
of random walkers, it is the basis for a lot of so-called “spectral ranking” methods, and it is
a very popular way to deal with directed (and undirected) graphs. In addition, for reasons
we will point out later, we can get spectral methods to work in a very natural way in this
particular case, even if the initial graph is undirected.

48

M. W. Mahoney

5

(02/05/2015): Overview of Graph Partitioning

Reading for today.
• “Survey: Graph clustering,” in Computer Science Review, by Schaeffer
• “Geometry, Flows, and Graph-Partitioning Algorithms,” in CACM, by Arora, Rao, and Vazirani
The problem of graph partitioning or graph clustering refers to a general class of problems that
deals with the following task: given a graph G = (V, E), group the vertices of a graph into groups or
clusters or communities. (One might be interested in cases where this graph is weighted, directed,
etc., but for now let’s consider non-directed, possibly weighted, graphs. Dealing with weighted
graphs is straightforward, but extensions to directed graphs are more problematic.) The graphs
might be given or constructed, and there may or may not be extra information on the nodes/edges
that are available, but insofar as the black box algorithm that actually does the graph partitioning
is concerned, all there is is the information in the graph, i.e., the nodes and edges or weighted edges.
Thus, the graph partitioning algorithm takes into account the node and edge properties, and thus
it typically relies on some sort of “edge counting” metric to optimize. Typically, the goal is to
group nodes in such a manner that nodes within a cluster are more similar to each other than to
nodes in different clusters, e.g., more and/or better edges within clusters and relatively few edges
between clusters.

5.1

Some general comments

Two immediate questions arise.
• A first question is to settle on an objective that captures this bicriteria. There are several
ways to quantify this bicriteria which we will describe, but each tries to cut a data graph into
2 or more “good” or “nice” pieces.
• A second question to address is how to compute the optimal solution to that objective. In
some cases, it is “easy,” e.g., it is computable in low-degree polynomial time, while in other
cases it is “hard,” e.g., it is intractable in the sense that the corresponding decision problem
is NP-hard or NP-complete.
In the case of an intractable objective, people are often interested in computing some sort of
approximate solution to optimize the objective that has been decided upon. Alternatively, people
may run a procedure without a well-defined objective stated and decided upon beforehand, and in
some cases this procedure returns answers that are useful. Moreover, the procedures often bear some
sort of resemblance to the steps of algorithms that solve well-defined objectives exactly. Clearly,
there is potential interest in understanding the relationship between these two complementary
approaches: this will help people who run procedures know what they are optimizing; this can feed
back and help to develop statistically-principled and more-scalable procedures; and so on.
Here, we will focus on several different methods (i.e., classes of algorithms, e.g., “spectral graph
algorithms” as well as other classes of methods) that are very widespread in practice and that can
be analyzed to prove strong bounds on the quality of the partitions found. The methods are the
following.

Lecture Notes on Spectral Graph Methods

49

1. Spectral-based methods. This could include either global or local methods, both of which
come with some sort of Cheeger Inequality.
2. Flow-based methods. These have connections with the min-cut/max-flow theorem, and they
can be viewed in terms of embeddings via their LP formulation, and here too there is a local
improvement version.
In addition, we will also probably consider methods that combine spectral and flow in various ways.
Note that most or all of the theoretically-principles methods people use have steps that boil down
to one of these. Of course, we will also make connections with methods such as local improvement
heuristics that are less theoretically-principled but that are often important in practice.
Before doing that, we should point out something that has been implicit in the discussion so
far. That is, while computer scientists (and in particular TCS) often draw a strong distinction
between problems and algorithms, researchers in other areas (in particular machine learning and
data analysis as well as quantitatively-inclined people in nearly every other applied area) often do
not. For the latter people, one might run some sort of procedure that solves something insofar
as, e.g., it finds clusters that are useful by a downstream metric. As you can imagine, there is a
proliferation of such methods. One of the questions we will address is when we can understand
those procedures in terms of the above theoretically-principled methods. In many cases, we can;
and that can help to understand when/why these algorithms work and when/why they don’t.
Also, while we will mostly focus on a particular objective (called expansion or conductance) that
probably is the combinatorial objective that most closely captures the bicriteria of being wellconnected intra-cluster and not well-connected inter-cluster, we will probably talk about some other
related methods. For example, finding dense subgraphs, and finding so-called good-modularity
partitions. Those are also of widespread interest; they will illustrate other ways that spectral
methods can be used; and understanding the relationship between those objectives and expansion/conductance is important.
Before proceeding, a word of caution: For a given objective quantifying how “good” is a partition,
it is not the case that all graphs have good partitions—but all graph partitioning algorithms (as
will other algorithms) will return some answer, i.e., they will give you some output clustering. In
particular, there is a class of graphs called expanders that do not have good clusters with respect
to the so-called expansion/conductance objective function. (Many real data graphs have strong
expander-like properties.)
In this case, i.e., when there are no good clusters, the simple answer is just to say don’t do clustering.
Of course, it can sometimes in practice be difficult to tell if you are in that case. (For example,
with a thousand graphs and a thousand methods—that may or may not be related but that have
different knobs and so are at least minorly-different—you are bound to find things look like clusters,
and controlling false discovery, etc., is tricky in general but in particular for graph-based data.)
Alternatively, especially in practice, you might have a graph that has both expander-like properties
and non-expander-like properties, e.g., in different parts of the graph. A toy example of this could
be given by the lollipop graph. In that case, it might be good to know how algorithms behave on
different classes of graphs and/or different parts of the graph.
Question (raised by this): Can we certify that there are no good clusters in a graph? Or certify
the nonexistence of hypothesized things more generally? We will get back to this later.
Let’s go back to finding an objective we want to consider.

50

M. W. Mahoney

As a general rule of thumb, when most people talk about clusters or communities (for some reason,
in network and especially in social graph applications clusters are often called communities—they
may have a different downstream, e.g., sociological motivation, but operationally they are typically
found with some sort of graph clustering algorithm) “desirable” or “good” clusters tend to have
the following properties:
1. Internally (intra) - well connected with other members of the cluster. Minimally, this means
that it should be connected—but it is a challenge to guarantee this in a statistically and
algorithmically meaningful manner. More generally, this might mean that it is “morally
connected”—e.g., that there are several paths between vertices in intra-clusters and that
these paths should be internal to the cluster. (Note: this takes advantage of the fact that
we can classify edges incident to v ∈ C as internal (connected to other members of C) and
external (connected to C̄).
2. Externally (inter) - relatively poor connections between members of a cluster and members
of a different cluster. For example, this might mean that there are very few edges with one
endpoint in one cluster and the other endpoint in the other cluster.
Note that this implies that we can classify edges, i.e., pairwise connections, incident to a vertex
v ∈ C into edges that are internal (connected to other members of C) and edges that are external
(connected to members of C̄). This technically is well-defined; and, informally, it makes sense, since
if we are modeling the data as a graph, then we are saying that things and pairwise relationships
between things are of primary importance.
So, we want a relatively dense or well-connected (very informally, those two notions are similar,
but they are often different when one focuses on a particular quantification of the informal notion)
induced subgraph with relatively few inter-connections between pieces. Here are extreme cases to
consider:
• Connected component, i.e., the “entire graph,” if the graph is connected, or one connected
component if the graph is not connected.
• Clique or maximal clique, i.e., complete subgraph or a maximal complete subgraph, i.e.,
subgraph in which no other vertices can be added without loss of the clique property.
But how do we quantify this more generally?

5.2

A first try with min-cuts

Here we will describe an objective that has been used to partition graphs. Although it is widely-used
for certain applications, it will have certain aspects that are undesirable for many other applications.
In particular, we cover it for a few reasons: first, as a starter objective before we get to a better
objective; second, since the dual is related to a non-spectral way to partition graphs; and third,
although it doesn’t take into account the bi-criteria we have outlined, understanding it will be a
basis for a lot of the stuff later.

Lecture Notes on Spectral Graph Methods
5.2.1

51

Min-cuts and the Min-cut problem

We start with the following definition.
Definition 17. Let G = (V, E) be a graph. A cut C = (S, T ) is a partition of the vertex set V
of G. An s-t-cut C = (S, T ) of G = (V, E) is a cut C s.t. s ∈ S and t ∈ T , where s, t ∈ V are
pre-specified source and sink vertices/nodes. A cut set is {(u, v) ∈ E : u ∈ S, v ∈ T }, i.e., the edges
with one endpoint on each side of the cut.
The above definition applies to both directed and undirected graphs. Notice in the directed case,
the cut set contains the edges from node in S to nodes in T , but not those from T to S.
Given this set-up the min-cut problem is: find the “smallest” cut, i.e., find the cut with the
“smallest” cut set, i.e. the smallest boundary (or sum of edge weights, more generally). That is:
Definition 18. The capacity of an s-t-cut is c(S, T ) =
Problem is to solve

P

(u,v)∈(S,S̄) cuv .

In this case, the Min-Cut

min c(S, S̄).

s∈S,t∈S̄

That is, the problem is to find the “smallest” cut, where by smallest we mean the cut with the
smallest total edge capacity across it, i.e., with the smallest “boundary.”
Things to note about this formalization:
1. Good: Solvable in low-degree polynomial time by a polynomial time algorithm. (As we will
see, min-cut = max-flow is related.)
2. Bad: Often get very unbalanced cut. (This is not necessarily a problem, as maybe there are
no good cuts, but for this formalization, this happens even when it is known that there are
good large cuts. This objective tends to nibble off small things, even when there are bigger
partitions of interest.) This is problematic for several reasons:
• In theory. Cut algorithms are used as a sub-routine in divide and conquer algorithm,
and if we keep nibbling off small pieces then the recursion depth is very deep; alternatively, control over inference is often obtained by drawing strength over a bunch of data
that are well-separated from other data, and so if that bunch is very small then the
inference control is weak.
• In practice. Often, we want to “interpret” the clusters or partitions, and it is not nice
if the sets returned are uninteresting or trivial. Alternatively, one might want to do
bucket testing or something related, and when the clusters are very small, it might not
be worth the time.
(As a forward-looking pointer, we will see that an ‘improvement” of the idea of cut and
min-cut may also get very imbalanced partitions, but it does so for a more subtle/non-trivial
reason. So, this is a bug or a feature, but since the reason is somewhat trivial people typically
view this as a bug associated with the choice of this particular objective in many applications.)

52

M. W. Mahoney

5.2.2

A slight detour: the Max-Flow Problem

Here is a slight detour (w.r.t. spectral methods per se), but it is one that we will get back to, and
it is related to our first try objective.
Here is a seemingly-different problem called the Max-Flow problem.
Definition 19. The capacity of an edge e ∈ E is a mapping c : E → R+ , denoted ce or cuv (which
will be a constraint on the maximum amount of flow we allow on that edge).
Definition 20. A flow in a directed graph is a mapping f : E → R, denoted fe or fuv s.t.:
• fuv ≤ cuv , ∀(u, v) ∈ E (Capacity constraint.)
P
P
•
∀u ∈ V \{s, t} (Conservation of flow, except at source and
v:(u,v)∈E fuv =
v:(v,u)∈E fvu ,
sink.)
• fe ≥ 0

∀e ∈ E (obey directions)

A flow in a undirected graph is a mapping f : E → R, denoted fe . We arbitrarily assign directions
to each edge, say e = (u, v), and when we write f(v,u) , it is just a notation for −f(u,v)
• |fuv | ≤ cuv , ∀(u, v) ∈ E (Capacity constraint.)
P
•
∀u ∈ V \{s, t} (Conservation of flow, except at source and sink.)
v:(u,v)∈E fuv = 0,
P
Definition 21. The value of the flow |f | = v∈V |fsv |, where s is the source. (This is the amount of
flow flowing out of s. It is easy to see that as all the nodes other than s, t obey the flow conservation
constraint, the flow out of s is the same as the flow into t. This is the amount of flow flowing from
s to t.) In this case, the Max-Flow Problem is
max |f |.
Note: what we have just defined is really the “single commodity flow problem” since there exists
only 1 commodity that we are routing and thus only 1 source/sink pair s and t that we are routing
from/to. (We will soon see an important generalization of this to something called multi-commodity
flow, and this will be very related to a non-spectral method for graph partitioning.)
Here is an important result that we won’t prove.
Theorem 9 (Max-Flow-Min-Cut Theorem). The max value of an s − t flow is equal to the min
capacity of an s − t cut.
Here, we state the Max-Flow problem and the Min-Cut problem, in terms of the primal and dual
optimization problems.
Primal: (Max-Flow):
max
s.t.

|f |

fuv ≤ Cuv , (uv) ∈ E
X
X
fvu −
fuv ≤ 0,

v:(vu)∈E

fuv ≥ 0

v:(uv)∈E

u∈V

Lecture Notes on Spectral Graph Methods

53

Dual: (Min-Cut):
min

X

cij dij

(i,j)∈E

s.t.

dij − pi + pj ≥ 0, (ij) ∈ E

ps = 1, pt = 0,
pi ≥ 0, i ∈ V

dij ≥ 0, ij ∈ E
There are two ideas here that are important that we will revisit.
• Weak duality: for any instance and any feasible flows and cuts, max f low ≤ min cut.
• Strong duality: for any instance, ∃ feasible flow and feasible cut s.t. the objective functions
are equal, i.e., s.t. max f low = min cut.
We are not going to go into these details here—for people who have seen it, it is just to set the
context, and for people who haven’t seen it, it is to give an important fyi. But we will note the
following.
• Weak duality generalizes to many settings, and in particular to multi-commodity flow; but
strong duality does not. The next question is: does there exist a cut s.t. equality is achieved.
• We can get an approximate version of strong duality, i.e., an approximate Min-Cut-Max-Flow
theorem in that multi-commodity case. That we can get such a bound will have numerous
algorithmic implications, in particular for graph partitioning.
• We can translate this (in particular, the all-pairs multi-commodity flow problem) into 2-way
graph partitioning problems (this should not be immediately obvious, but we will cover it
later) and get nontrivial approximation guarantees.
About the last point: for flows/cuts we have introduced special source and sink nodes, s and t, but
when we apply it back to graph partitioning there won’t be any special source/sink nodes, basically
since we will relate it to the all-pairs multi-commodity flow problem, i.e., where we consider all n2
possible source-sink pairs.

5.3

Beyond simple min-cut to “better” quotient cut objectives

The way we described what is a “good” clustering above was in terms of an intra-connectivity versus
intra-connectivity bi-criterion. So, let’s revisit and push on that. A related thing or a different way
(that gives the same result in many cases, but sometimes does not) is to say that a bi-criterion is:
• We want a good “cut value”—not too many crossing edges—where cut value is E(S, S̄) or a
weighted version of that. I.e., what we just considered with min-cut.
• We want good “balance” properties—i.e., both sides of the cut should be roughly the same
size—so both S, S̄ are the same size or approximately the same size.

54

M. W. Mahoney

There are several ways to impose a balance condition. Some are richer or more fruitful (in theory
and/or in practice) than others. Here are several. First, we can add “hard” or explicit balance
conditions:
• Graph bisection—find a min cut s.t. |S| = |S̄| = n/2, i.e., ask for exactly 50-50 balance.
• β-balanced cut—find a min cut s.t |S| = βn, |S̄| = (1 − β)n, i.e., give a bit of wiggle room
and ask for exactly (or more generally no worse than), say, a 70-30 balance.
Second, there are also “soft” or implicit balance conditions, where there is a penalty and separated
nodes “pay” for edges in the cut. (Actually, these are not “implicit” in the way we will use the
word later; here it is more like “hoped for, and in certain intuitive cases it is true.” And they are
not quite soft, in that they can still lead to imbalance; but when they do it is for much more subtle
and interesting reasons.) Most of these are usually formalized as quotient-cut-style objectives:
• Expansion:
• Sparsity:

E(S,S̄)
|S|
n

E(S,S̄)
|S||S̄|

• Conductance:

or

(def this as :h(S) ) (a.k.a. q-cut)

(def this as :sp(S) ) (a.k.a. approximate-expansion)

E(S,S̄)

Vol(S)

• Normalized-cut:

E(S,S̄)
min{|S|,|S̄|}

or

E(S,S̄)
min(Vol(|S|),Vol(|S̄|))

(a.k.a. Normalized cut)

n

E(S,S̄)

Vol(|S|)·Vol(|S̄|)

(a.k.a. approximate conductance)

Here, E(S, S̄) is the number of edges between S and S̄, or morePgenerally for a weighted graph
the sum of the edge weights between S and S̄, and Vol(S) =
ij∈E deg(Vi ). In addition, the
denominator in all four cases correspond to different volume notions: the first two are based on the
number of nodes in S, and the last two are based on the number of edges in S (i.e., the sum of the
degrees of the nodes in S.)
Before proceeding, it is worth asking what if we had taken a difference rather than a ratio, e.g.,
SA − V OL, rather than SA/V OL. At the high level we are discussing now, that would do the same
thing—but, quantitatively, using an additive objective will generally give very different results than
using a ratio objective, in particular when one is interested in fairly small clusters.
(As an FYI, the first two, i.e., expansion and sparsity, are typically used in the theory algorithms
algorithms, since they tend to highlight the essential points; while the latter two, i.e., conductance
and normalized cuts, are more often used in data analysis, machine learning, and other applications,
since issues of normalization are dealt with better.)
Here are several things to note:
• Expansion provides a slightly stronger bias toward being well-balanced than sparsity (i.e.
between a 10 − 90 cut and a 40 − 60 cut, the advantage in the denominator for the more
balanced 40 − 60 cut in expansion is 4 : 1, while it is 2400 : 900 < 4 : 1 in sparsity) (and there
might be some cases where this is important. That is, the product variants have a “factor of
2” weaker preference for balance than the min variants. Similarly for normalized cuts versus
conductance.

Lecture Notes on Spectral Graph Methods

55

• That being said, that difference is swamped by the following. Expansion and sparsity are the
“same” (in the following sense:)
min h(S) ≈ min sp(S)
Similarly for normalized cuts versus conductance.
• Somewhat more precisely, although the expansion of any particular set isn’t in general close
to the sparsity of that set, The expansion problem and the sparsity problem are equivalent
in the following sense:
It is clear that
argmin Φ′ (G) = argmin nΦ′ (G) = argmin
S

S

S

C(S, S̄)
n
min{|S|, |S̄|} max{|S|, |S̄|}

n
As 1 < max{|S|,|
≤ 2 , the min partition we find by optimizing sparsity will also give, off by
S̄|}
a multiplicative factor of 2, the optimal expansion. As we will see this is small compared to
O(log n) approximation from flow or the quadratic factor with Cheeger, and so is not worth
worrying about from an optimization perspective. Thus, we will be mostly cavalier about
going back and forth.

• Of course, the sets achieving the optimal may be very different. An analogous thing was seen
in vector spaces—the optimal may rotate by 90 degrees, but for many things you only need
that the Rayleigh quotient is approximately optimal. Here, however, the situation is worse.
Asking for the certificates achieving the optimum is a more difficult thing—in the vector space
case, this means making awkward “gap” assumptions, and in the graph case it means making
strong and awkward combinatorial statements.
• Expansion 6= Conductance, in general, except for regular graphs. (Similarly, Sparsity 6=
Normalized Cuts, in general, except for regular graphs.) The latter is in general preferable
for heterogeneous graphs, i.e., very irregular graphs. The reason is that there are closer
connections with random walks and we get tighter versions of Cheeger’s inequality if we take
the weights into account.
• Quotient cuts capture exactly the surface area to volume bicriteria that we wanted. (As a
forward pointer, a question is the following: what does this mean if the data come from a low
dimensional space versus a high dimensional space; or if the data are more or less expanderlike; and what is the relationship between the original data being low or high dimensional
versus the graph being expander-like or not?
• For “space-like” graphs, these two bicriteria as “synergistic,” in that they work together; for
expanders, they are “uncoupled,” in that the best cuts don’t depend on size, as they are all
bad; and for many “real-world” heavy-tailed informatics graphs, they are “anti-correlated,”
in that better balance means worse cuts.
• An obvious question: are there other notions, e.g., of “volume” that might be useful and that
will lead to similar results we can show about this? (In some cases, the answer to this is yes:
we may revisit this later.) Moreover, one might want to choose a different reweighting for
statistical or robustness reasons.

56

M. W. Mahoney

(We will get back to the issues raised by that second-to-last point later when we discuss “local”
partitioning methods. We simply note that “space-like” graphs include, e.g., Z 2 or random geometric graphs or “nice” planar graphs or graphs that “live” on the earth. More generally, there is
a trade-off and we might get very imbalanced clusters or even disconnected clusters. For example,
for the G(n, p) random graph model if p ≥ log n2 /n then we have an expander, while for extremely
sparse random graphs, i.e., p < log n/n, then due to lack of concentration we can have deep small
cuts but be expander-like at larger size scales.)

5.4

Overview Graph Partition Algorithms

Here, we will briefly describe the “lay of the land” when it comes to graph partitioning algorithms—
in the next few classes, we will go into a lot more details about these methods. There are three
basic ideas you need to know for graph partitioning, in that nearly all methods can be understood
in terms of some combination of these methods.
• Local Improvement (and multi-resolution).
• Spectral Methods.
• Flow-based Methods.
As we will see, in addition to being of interest in clustering data graphs, graph partitioning is
a nice test-case since it has been very well-studied in theory and in practice and there exists a
large number of very different algorithms, the respective strengths and weaknesses of which are
well-known for dealing with it.
5.4.1

Local Improvement

Local improvement methods refer to a class of methods that take an input partition and do moreor-less naive steps to get a better partition:
• 70s Kernighan-Lin.
• 80s Fiduccia-Mattheyses—FM and KL start with a partition and improve the cuts by flipping
nodes back and forth. Local minima can be a big problem for these methods. But they can
be useful as a post-processing step—can give a big difference in practice. FM is better than
KL since it runs in linear time, and it is still commonly used, often in packages.
• 90s Chaco, Metis, etc. In particular, METIS algorithm from Karypis and Kumar, works very
well in practice, especially on low dimensional graphs.
The methods of the 90s used the idea of local improvement, coupled with the basically linear
algebraic idea of Multiresolution get algorithms that are designed to work well on space-like graphs
and that can perform very well in practice. The basic idea is:
• Contract edges to get a smaller graph.
• Cut the resulting graph.

Lecture Notes on Spectral Graph Methods

57

• Unfold back up to the original graph.
Informally, the basic idea is that if there is some sort of geometry, say the graph being partitioned is
the road network of the US, i.e., that lives on a two-dimensional surface, then we can “coarse grain”
over the geometry, to get effective nodes and edges, and then partition over the coarsely-defined
graph. The algorithm will, of course, work for any graph, and one of the difficulties people have
when applying algorithms as a black box is that the coarse graining follows rules that can behave
in funny ways when applied to a graph that doesn’t have the underlying geometry.
Here are several things to note:
• These methods grew out of scientific computing and parallel processing, so they tend to work
on “space-like” graphs, where there are nice homogeneity properties—even if the matrices
aren’t low-rank, they might be diagonal plus low-rank off-diagonal blocks for physical reasons,
or whatever.
• The idea used previously to speed up convergence of iterative methods.
• Multiresolution allows globally coherent solutions, so it avoids some of the local minima
problems.
• 90s: Karger showed that one can compute min-cut by randomly contracting edges, and so
multiresolution may not be just changing the resolution at which one views the graph, but it
may be taking advantage of this property also.
An important point is that local improvement (and even multiresolution methods) can easily get
stuck in local optima. Thus, they are of limited interest by themselves. But they can be very
useful to “clean up” or “improve” the output of other methods, e.g., spectral methods, that in a
principled way lead to a good solution, but where the solution can be improved a bit by doing some
sort or moderately-greedy local improvement.
5.4.2

Spectral methods

Spectral methods refer to a class of methods that, at root, are a relaxation or rounding method
derived from an NP-hard QIP and that involves eigenvector computations. In this case that we are
discussing, it is the QIP formulation of the graph bisection problem that is relaxed. Here is a bit
of incomplete history.
• Donath and Hoffman (ca. 72,73) introduced the idea of using the leading eigenvector of the
Adjacency Matrix AG as a heuristic to find good partitions.
• Fiedler (ca. 73) associated the second smallest eigenvalue of the Laplacian LG with graph
connectivity and suggested splitting the graph by the value along the associated eigenvector.
• Barnes and Hoffman (82,83) and Bopanna (87) used LP, SDP, and convex programming
methods to look at the leading nontrivial eigenvector.
• Cheeger (ca. 70) established connections with isoperimetric relationships on continuous manifolds, establishing what is now known ad Cheeger’s Inequality.

58

M. W. Mahoney
• 80s: saw performance guarantees from Alon-Milman, Jerrum-Sinclair, etc., connecting λ2 to
expanders and rapidly mixing Markov chains.
• 80s: saw improvements to approximate eigenvector computation, e.g., Lanczos methods,
which made computing eigenvectors more practical and easier.
• 80s/90s: saw algorithms to find separators in certain classes of graphs, e.g., planar graphs,
bounds on degree, genus, etc.
• Early 90s: saw lots of empirical work showing that spectral partitioning works for “real”
graphs such as those arising in scientific computing applications
• Spielman and Teng (96) showed that “spectral partitioning works” on bounded degree planar
graphs and well-shaped meshed, i.e., in the application where it is usually applied.
• Guattery and Miller (95, 97) showed “spectral partitioning doesn’t work” on certain classes of
graphs, e.g., the cockroach graph, in the sense that there are graphs for which the quadratic
factor is achieved. That particular result holds for vanilla spectral, but similar constructions
hold for non-vanilla spectral partitioning methods.
• Leighton and Rao (87, 98) established a bound on the duality gap for multi-commodity flow
problems, and used multi-commodity flow methods to get an O(log n) approximation to the
graph partitioning problem.
• LLR (95) considered the geometry of graphs and algorithmic applications, and interpreted LR
as embedding G in a metric space, making the connection with the O(log n) approximation
guarantee via Bourgain’s embedding lemma.
• 90s: saw lots of work in TCS on LP/SDP relaxations of IPs and randomized rounding to get
{±1} solutions from fractional solutions.
• Chung (97) focused on the normalized Laplacian for degree irregular graphs and the associated
metric of conductance.
• Shi and Malik (99) used normalized cuts for computer vision applications, which is essentially
a version of conductance.
• Early 00s: saw lots of work in ML inventing and reinventing and reinterpreting spectral
partitioning methods, including relating it to other problems like semi-supervised learning
and prediction (with, e.g., boundaries between classes being given by low-density regions).
• Early 00s: saw lots of work in ML on manifold learning, etc., where one constructs a graph
and recovers an hypothesized manifold; constructs graphs for semi-supervised learning applications; and where the diffusion/resistance coordinates are better or more useful/robust than
geodesic distances.
√
• ARV (05) got an SDP-based embedding to get an O( log n) approximation, which combined
ideas from spectral and flow; and there was related follow-up work.
• 00s: saw local/locally-biased spectral methods and improvements to flow improve methods.
• 00s: saw lots of spectral-like methods like viral diffusions with social/complex networks.

Lecture Notes on Spectral Graph Methods

59

For the moment and for simplicity, say that we are working with unweighted graphs. The graph
partitioning QIP is:
min
s.t.

xT Lx
xT 1 = 0
xi ∈ {−1, +1}

and the spectral relaxation is:
min
s.t.

xT Lx
xT 1 = 0
xi ∈ R,

xT x = n

That is, we relax x from being in {−1, 1}, which is a discrete/combinatorial constraint, to being a
real continuous number that is 1 “on average.” (One could relax in other ways—e.g., we could relax
to say that it’s magnitude is equal to 1, but that it sits on a higher-dimensional sphere. We will
see an example of this later. Or other things, like relaxing to a metric.) This spectral relaxation is
not obviously a nice problem, e.g., it is not even convex; but it can be shown that the solution to
this relaxation can be computed as the second smallest eigenvector of L, the Fiedler vector, so we
can use an eigensolver to get the eigenvector.
Given that vector, we then have to perform a rounding to get an actual cut. That is, we need to
take the real-valued/fractional solution obtained from the continuous relaxation and round it back
to {−1, +1}. There are different ways to do that.
So, here is the basic spectral partitioning method.
• Compute an eigenvector of the above program.
• Cut according to some rules, e.g., do a hyperplane rounding, or perform some other more
complex rounding rule.
• Post process with a local improvements method.
The hyperplane rounding is the easiest to analyze, and we will do it here, but not surprisingly
factors of 2 can matter in practice; and so—when spectral is an appropriate thing to do—other
rounding rules often do better in practice. (But that is a “tweak” on the larger question of spectral
versus flow approximations.) In particular, we can do local improvements here to make the output
slightly better in practice. Also, there is the issue of what exactly is a rounding, e.g., if one performs
a sophisticated flow-based rounding then one may obtain a better objective function but a worse
cut value. Hyperplane rounding involves:
• Choose a split point x̂ along the vector x
• Partition nodes into 2 sets: {xi < x̂} and {xi > x̂}
By Vanilla spectral, we refer to spectral with hyperplane rounding of the Fiedler vector embedding.
Given this setup of spectral-based partitioning, what can go “wrong” with this approach.

60

M. W. Mahoney
• We can choose the wrong direction for the cut:
– Example—Guattery and Miller construct an example that is “quadratically bad” by
taking advantage of the confusion that spectral has between “long paths” and “deep
cuts.”
– Random walk interpretation—long paths
√ can also cause slow mixing since the expected
progress of a t-step random walk is O( t).
• The hyperplane rounding can hide good cuts:
– In practice, it is often better to post-process with FM to improve the solution, especially
if want good cuts, i.e., cuts with good objective function value.

An important point to emphasize is that, although both of these examples of “wrong” mean that
the task one is trying to accomplish might not work, i.e., one might not find the best partition,
sometimes that is not all bad. For example, the fact that spectral methods “strip off” long stringy
pieces might be ok if, e.g., one obtains partitions that are “nice” in other ways. That is, the direction
chosen by spectral partitioning might be nice or regularized relative to the optimal direction. We
will see examples of this, and in fact it is often for this reason that spectral performs well in practice.
Similarly, the rounding step can also potentially give an implicit regularization, compared to more
sophisticated rounding methods, and we will return to discuss this.
5.4.3

Flow-based methods

There is another class of methods that uses very different ideas to partition graphs. Although this
will not be our main focus, since they are not spectral methods, we will spend a few classes on it,
and it will be good to know about them since in many ways they provide a strong contrast with
spectral methods.
This class of flow-based methods uses the “all pairs” multicommodity flow procedure to reveal
bottlenecks in the graph. Intuitively, flow should be “perpendicular” to the cut (i.e. in the sense
of complementary slackness for LPs, and similar relationship between primal/dual variables to
dual/primal constraints in general). The idea is to route a large number of commodities simultaneously between random pairs of nodes and then choose the cut with the most edges congested—the
idea being that a bottleneck in the flow computation corresponds to a good cut.
Recall that the single commodity max-flow-min-cut procedure has zero duality gap, but that is not
the case for multi-commodity problem. On the other hand, the k-multicommodity has O(log k)
duality gap—this result is due to LR and LLR, and it says that there is an approximate min-flowmax-cut. Also, it implies an O(log n) gap for the all pairs problem.
The following is an important point to note.
Claim 1. The O(log n) is tight on expanders.
For flow, there are connections to embedding and linear programming, so as we will see, we can
think of the algorithm as being:
• Relax flow to LP, and solve the LP.

Lecture Notes on Spectral Graph Methods

61

• Embed solution in the ℓ1 metric space.
• Round solution to {0, 1}.

5.5

Advanced material and general comments

We will conclude with a brief discussion of these results in a broader context. Some of these issues
we may return to later.

5.5.1

Extensions of the basic spectral/flow ideas

Given the basic setup of spectral and flow methods, both of which come with strong theory, here
are some extensions of the basic ideas.
• Huge graphs. Here want to do computations depending on the size of the sets and not the
size of the graph, i.e., we don’t even want to touch all the nodes in the graph, and we want
to return a cut that is nearby an input seed set of nodes. This includes “local” spectral
methods—that take advantage of diffusion to approximate eigenvectors and get Cheeger-like
guarantees.
• Improvement Methods. Here we want to “improve” an input partition—there are both spectral and flow versions.
• Combining Spectral and Flow.
– ARV solves an SDP, which takes time like O(n4.5 ) or so; but we can do it faster (e.g.,
on graphs with ≈ 105 nodes) using ideas related to approximate multiplicative weights.
– There are strong connections here to online learning—roughly since we can view “worst
case” analysis as a “game” between a cut player and a matching player.
– Similarly, there are strong connections to boosting, which suggest that these combinations might have interesting statistical properties.
A final word to reemphasize: at least as important for what we will be doing as understanding
when these methods work is understanding when these methods “fail”—that is, when they achieve
their worst case quality-of-approximation guarantees:
• Spectral methods “fail” on graphs with “long stringy” pieces, like that constructed by Guattery and Miller.
• Flow-based
methods “fail” on expander graphs (and, more generally, on graphs where most
n
of the 2 pairs but most pairs are far log n apart).

Importantly, a lot of real data have “stringy” pieces, as well as expander-like parts; and so it is not
hard to see artifacts of spectral and flow based approximation algorithms when they are run on
real data.

62

M. W. Mahoney

5.5.2

Additional comments on these methods

Here are some other comments on spectral versus flow.
• The SVD gives good “global” but not good “local” guarantees. For example, it provides
global reconstruction error, and going to the low-dimensional space might help to speed up
all sorts of algorithms; but any pair of distances might be changed a lot in the low-dimensional
space, since the distance constraints are only satisfied on average. This should be contrasted
with flow-based embedding methods and all sorts of other embedding methods that are used
in TCS and related areas, where one obtains very strong local or pairwise guarantees. There
are two important (but not immediately obvious) consequences of this.
– The lack of local guarantees makes it hard to exploit these embeddings algorithmically
(in worst-case), whereas the pair-wise guarantees provided by other types of embeddings
means that you can get worst-case bounds and show that the solution to the subproblem
approximates in worst case the solution to the original problem.
– That being said, the global guarantee means that one obtains results that are more robust
to noise and not very sensitive to a few “bad” distances, which explains why spectral
methods are more popular in many machine learning and data analysis applications.
– That local guarantees hold for all pair-wise interactions to get worst-case bounds in
non-spectral embeddings essentially means that we are “overfitting” or “most sensitive
to” data points that are most far apart. This is counter to a common design principle,
e.g., exploited by Gaussian rbf kernels and other NN methods, that the most reliable
information in the data is given by nearby points rather than far away points.

5.6

References

1. Schaeffer, ”Graph Clustering”, Computer Science Review 1(1): 27-64, 2007
2. Kernighan, B. W.; Lin, Shen (1970). ”An efficient heuristic procedure for partitioning
graphs”. Bell Systems Technical Journal 49: 291-307.
3. CM Fiduccia, RM Mattheyses. ”A Linear-Time Heuristic for Improving Network Partitions”.
Design Automation Conference.
4. G Karypis, V Kumar (1999). ”A Fast and High Quality Multilevel Scheme for Partitioning
Irregular Graphs”. Siam Journal on Scientific Computing.

Lecture Notes on Spectral Graph Methods

6

63

(02/10/2015): Spectral Methods for Partitioning Graphs (1 of
2): Introduction to spectral partitioning and Cheeger’s Inequality

Reading for today.
• “Lecture Notes on Expansion, Sparsest Cut, and Spectral Graph Theory,” by Trevisan
Today and next time, we will cover what is known as spectral graph partitioning, and in particular
we will discuss and prove Cheeger’s Inequality. This result is central to all of spectral graph theory
as well as a wide range of other related spectral graph methods. (For example, the isoperimetric
“capacity control” that it provides underlies a lot of classification, etc. methods in machine learning that are not explicitly formulated as partitioning problem.) Cheeger’s Inequality relates the
quality of the cluster found with spectral graph partitioning to the best possible (but intractable to
compute) cluster, formulated in terms of the combinatorial objectives of expansion/conductance.
Before describing it, we will cover a few things to relate what we have done in the last few classes
with how similar results are sometimes presented elsewhere.

6.1

Other ways to define the Laplacian

Recall that L = D − A is the graph Laplacian, or we could work with the normalized Laplacian, in
which case L = I − D −1/2 AD −1/2 . While these definition might not make it obvious, the Laplacian
actually has several very intuitive properties (that could alternatively be used as definitions). Here,
we go over two of these.
6.1.1

As a sum of simpler Laplacians

Again, let’s consider d-regular graphs. (Much of the theory is easier for this case, and expanders
are more extremal in this case; but the theory goes through to degree-heterogeneous graphs, and
this will be more natural in many applications, and so we will get back to this later.)
Recall the definition of the Adjacency Matrix of an unweighted graph G = (V, E):

1
if (ij) ∈ E
,
Aij =
0
otherwise
In this case, we can define the Laplacian as L = D − A or the normalized Laplacian as L = I − d1 A.
Here is an alternate definition for the Laplacian L = D − A. Let G12 be a graph on two vertices
with one edge between those two vertices, and define


1 −1
LG12 =
−1 1
Then, given a graph on n vertices with just one edge between vertex u and v, we can define L to
be the all-zeros matrix, except for the intersection between the uth and v th column and row, where
we define that intersection to be


1 −1
LGuv =
.
−1 1

64

M. W. Mahoney

Then, for a general graph G = (V, E), one we can define
X
LG =
LGuv .
(u,v)∈E

This provides a simpler way to think about the Laplacian and in particular changes in the Laplacian,
e.g.,
Pwhen one adds or removes edges. In addition, note also that this generalizes in a natural way
to (u,v)∈E wuv LGuv if the graph G = (V, E, W ) is weighted.
Fact. This is identical to the definition L = D − A. It is simple to prove this.

From this characterization, several things follow easily. For example,
X
xT Lx =
wuv (xu − xv )2 ,
(u,v)∈E

from which it follows that if v is an eigenvector of L with eigenvalue λ, then v T Lv = λv T v ≥ 0.
This means that every eigenvalue is nonnegative, i.e., L is SPSD.
6.1.2

In terms of discrete derivatives

Here are some notes that I didn’t cover in class that relate the Laplacian matrix to a discrete notion
of a derivative.
In classical vector analysis, the Laplace operator is a differential operator given by the divergence
of the gradient of a function in Euclidean space. It is denoted:
∇ · ∇ or ∇2 or △
In the cartesian coordinate system, it takes the form:


∂
∂
∇=
,··· ,
,
∂x1
∂xn
and so
△f =

n
X
∂2f
i=1

∂x2i

.

This expression arises in the analysis of differential equations of many physical phenomena, e.g.,
electromagnetic/gravitational potentials, diffusion equations for heat/fluid flow, wave propagation,
quantum mechanics, etc.
The discrete Laplacian is defined in an analogous manner. To do so, somewhat more pedantically,
let’s introduce a discrete analogue of the gradient and divergence operators in graphs.
Given an undirected graph G = (V, E) (which for simplicity we take as unweighted), fix an arbitrary
orientation of the edges. Then, let K ∈ RV ×E be the edge-incidence matrix of G, defined as

if edge e exits vertex u
 +1
Kue =
−1
if edge e enters vertex u .

0
otherwise
Then,

Lecture Notes on Spectral Graph Methods

65

• define the gradient as follows: let f : V → R be a function on vertices, viewed as a row vector
indexed by V ; then K maps f → f K, a vector indexed by E, measures the change of f along
edges of the graph; and if e is an edge from u to v, then (f K)e = fu − fv .
• define the divergence as follows: let g : E → R be a function on edges, viewed as a column vector indexed by E; then K maps g → Kg, a vector indexed by V ; if we think
of
P g as describing
P flow, then its divergence at vertex is the net outbound flow: (Kg)v =
g
−
e enters v gv
e exits v e
• define the Laplacian as follows: it should map f to KK T f , where f : V → R. So, L = LG =
KK T is the discrete Laplacian.

Note that it is easy to show that
Luv


 −1
=
deg(u)

0

if (u, v) ∈ E
if u = v
,
otherwise

which is in agreement with the previous definition. Note also that
X
f Lf T = f KK T f = kf Kk22 =
(fu − fv )2 ,
(u,v)∈E

which we will later interpret as a smoothness condition for functions on the vertices of the graph.

6.2

Characterizing graph connectivity

Here, we provide a characterization in terms of eigenvalues of the Laplacian of whether or not a
graph is connected. Cheeger’s Inequality may be viewed as a “soft” version of this result.
6.2.1

A Perron-Frobenius style result for the Laplacian

What does the Laplacian tell us about the graph? A lot of things. Here is a start. This is a
Perron-Frobenius style result for the Laplacian.
Theorem 10. Let G be a d-regular undirected graph, let L = I − 1d A be the normalized Laplacian;
and let λ1 ≤ λ2 ≤ · · · ≤ λn be the real eigenvalues, including multiplicity. Then:
1. λ1 = 0, and the associated eigenvector x1 =

~
√1
n

2. λ2 ≤ 2.

=



√1 , . . . , √1
n
n


.

3. λk = 0 iff G has at least k connected components. (In particular, λ2 > 0 iff G is connected.)
4. λn = 2 iff at least one connected component is bipartite.
Proof: Note that if x ∈ Rn , then xT Lx =
λ1 =

1
d

P

(u,v)∈E

(xu − xv )2 and also

xT Lx
≥ 0.
x∈Rn {0} xT x
min

66

M. W. Mahoney

Take ~1 = (1, . . . , 1), in which case ~1T L~1 = 0, and so 0 is the smallest eigenvalue, and ~1 is one of
the eigenvectors in the eigenspace of this eigenvalue. This proves part 1.
We also have the following formulation of λk by Courant-Fischer:
λk =

min

S⊆Rn
dim(S)=k

xT Ax
max
T
x∈S{~0} x x

P

(xu − xv )2
P
d u x2u

(u,v)∈E

P
So, if λk = 0, then ∃ a k-dimensional subspace S such that ∀x ∈ S, we have (u,v)∈E (xu − xv )2 = 0.
But this means that ∀x ∈ S, we have xu = xv ∀ edges (u, v) ∈ E with positive weight, and so
xu = xv , for any u, v in the same connected component. This means that x ∈ S is constant
within each connected component of G. So, k = dim(S) ≤ Ξ, where Ξ is the number of connected
components.
Conversely, if G has ≥ k connected components, then we can let S be the space of vectors that are
constant
on each component; and this S has dimension ≥ k. Furthermore, ∀x ∈ S, we have that
P
2
xT Ax
(u,v)∈E (xu − xv ) = 0. Thus maxx∈Sk {~0} xT x = 0 for any dimension k subspace Sk of the S
we choose. Then it is clear from Courant-Fischer λk = 0 as any Sk provides an upperbound.
Finally, to study λn = 2, note that
λn =

xT Lx
T
x∈Rn {~0} x x
max

This follows by using the variational characterization of the eigenvalues of −L and noting that −λn
is the smallest eigenvalue of −L. Then, observe that ∀x ∈ Rn , we have that
xT Lx
2− T =
x x

P

(xu + xv )2
P 2
≥ 0,
d u xu

(u,v)∈E

from which it follows that λn ≤ 2 (also λk ≤ 2 for all k = 2, . . . , n).
P
2
In addition, if λn = 2, then ∃x 6= 0 s.t.
(u,v)∈E (xu + xv ) = 0. This means that xu = −xv , for
all edges (u, v) ∈ E.
Let v be a vertex s.t. xv = a 6= 0. Define sets
A = {v : xv = a}

B = {v : xv = −a}

R = {v : xv 6= ±a}.

Then, the set A∪ B is disconnected from the rest of the graph, since otherwise anP
edge with an endpoint in A ∪ B and the other endpoint in R would give a positive contribution to ij Aij (xi + xj )2 .
Also, every edge incident on A has other endpoint in B, and vice versa. So A ∪ B is a bipartite
connected component (or a collection of connected components) of G, with bipartition A, B.
⋄
(Here is an aside. That proof was from Trevisan; Spielman has a somewhat easier proof, but it is
only for two components. I need to decide how much I want to emphasize the possibility of using k
eigenvectors for soft partitioning—I’m leaning toward it, since several students asked about it—and

Lecture Notes on Spectral Graph Methods

67

if I do I should probably go with the version of here that mentions k components.) As an FYI,
here is Spielman’s proof of λ2 = 0 iff G is disconnected; or, equivalently, that
λ2 > 0 ⇔ G is connected.
Start with proving the first direction: if G is disconnected, then λ2 = 0. If G is disconnected, then
G is the union of (at least) 2 graphs, call then G1 and G2 . Then, we can renumber the vertces so
that we can write the Laplacian of G as


L G1
0
LG =
0
L G2



 
1
0
So, LG has at least 2 orthogonal eigenvectors with eigenvalue 0, i.e.,
and
, where the
0
1
two vectors are given with the same renumbering as in the Laplacians. Conversely,
if G is connected
P
and x is an eigenvector such that LG x = 0x, then, LG x = 0, and xT LG x = (ij)∈E (xi − xj )2 = 0.
So, for all (u, v) connected by an edge, we have that xu = xu . Apply this iteratively, from which it
follows that x is a constant vector, i.e., xu = xv , forall u, v. So, the eigenspace of eigenvalue 0 has
dimension 1. This is the end of the aside.)
6.2.2

Relationship with previous Perron-Frobenius results

Theorem 10 is an important result, and it has several important extensions and variations. In
particular, the “λ2 > 0 iff G is connected” result is a “hard” connectivity statement. We will be
interested in how this result can be extended to a “soft” connectivity, e.g., “λ2 is far from 0 iff
the graph is well-connected,” and the associated Cheeger Inequality. That will come soon enough.
First, however, we will describe how this result relates to the previous things we discussed in the last
several weeks, e.g., to the Perron-Frobenius result which was formulated in terms of non-negative
matrices.
To do so, here is a similar result, formulated slightly differently.
Lemma 8. Let AG be the Adjacency Matrix of a d regular graph, and recall that it has n real
eigenvalues α1 ≥ · · · ≥ αn and n associated orthogonal eigenvectors vi s.t. Avi = λi vi . Then,
• α1 = d, with v1 =
• αn ≥ −d.

√1
n

=



√1 , . . . , √1
n
n


.

• The graph is connected iff α1 > α2 .
• The graph is bipartite iff α1 = −αn , i.e., if αn = −d.
Lemma 8 has two changes, relative to Theorem 10.
• The first is that it is a statement about the Adjacency Matrix, rather than the Laplacian.
• The second is that it is stated in terms of a “scale,” i.e., the eigenvalues depend on d.

68

M. W. Mahoney

When we are dealing with degree-regular graphs, then A is trivially related to L = D − A (we will
see this below) and also trivially related to L = I − 1d A (since this just rescales the previous L by
1/d). We could have removed the scale from Lemma 8 by multiplying the Adjacency Matrix by
1/d (in which case, e.g., the eigenvalues would be in [−1, 1], rather than [−d, d]), but it is more
common to remove the scale from the Laplacian. Indeed, if we had worked with L = D − A, then
we would have had the scale there too; we will see that below.
(When we are dealing with degree-heterogeneous graphs, the situation is more complicated. The
reason is basically since the eigenvectors of the Adjacency matrix and unnormalized Laplacian don’t
have to be related to the diagonal degree matrix D, which defined the weighted norm which relates
the normalized and unnormalized Laplacian. In the degree-heterogeneous case, working with the
normalized Laplacian will be more natural due to connections with random walks. That can be
interpreted as working with an unnormalized Laplacian, with an appropriate degree-weighted norm,
but then the trivial connection with the eigen-information of the Adjacency matrix is lost. We will
revisit this below too.)
In the above, A ∈ Rn×n is the Adjacency Matrix of an undirected graph G = (V, E). This will
provide the most direct connection with the Perro-Frobenius results we talked about last week.
Here are a few questions about the Adjacency Matrix.
• Question: Is it symmetric? Answer: Yes, so there are real eigenvalues and a full set of
orthonormal eigenvectors.
• Question: Is it positive? Answer: No, unless it is a complete graph. In the weighted case, it
could be positive, if there were all the edges but they had different weights; but in general it
is not positive, since some edges might be missing.
• Question: Is it nonnegative? Answer: Yes.
• Question: Is it irreducible? Answer: If no, i.e., if it is reducible, then


A11 A12
A=
0 A22
must also have A12 = 0 by symmetry, meaning that the graph is disconnected, in which case
we should think of it as two graphs. So, if the graph is connected then it is irreducible.
• Question: Is is aperiodic? Answer: If no, then since it must be symmetric, and so it must
look like


0 A12
A=
,
AT12 0
meaning that it is period equal to 2, and so the “second” large eigenvalue, i.e., the one on
the complex circle equal to a root of unity, is real and equal to −1.
How do we know that the trivial eigenvector is uniform? Well, we know that there is only one
all-positive eigenvector. Let’s try the all-ones vector ~1. In this case, we get

~

which means that α1 = d and v1 = √1n
and the graph is bipartite if α1 = −αn .

A~1 = d~1,


= √1n , . . . , √1n . So, the graph is connected if α1 > α2 ,

Lecture Notes on Spectral Graph Methods

69

For the Laplacian L = D − A, there exists a close relationship between the spectrum of A and
L. (Recall, we are still considering the d-regular case.) To see this, let d = α1 ≥ . . . αn be
the eigenvalues of A with associated orthonormal eigenvectors v1 , . . . , vn . (We know they are
orthonormal, since A is symmetric.) In addition, let 0 ≤ λ1 ≤ · · · ≤ λn be the eigenvalues of L.
(We know they are all real and in fact all positive from the above alternative definition.) Then,
αi = d − λi
and
AG vi = (dI − LG ) vi = (d − λi ) vi .

So, L “inherits” eigen-stuff from A. So, even though L isn’t positive or non-negative, we get PerronFrobenius style results for it, in addition to the results we get for it since it is a symmetric matrix.
In addition, if L → D −1/2 LD −1/2 , then the eigenvalues of L become in [0, 2], and so on. This can
be viewed as changing variables y ← D −1/2 x, and then defining Laplacian (above) and the Rayleigh
quotient in the degree-weighted dot product. (So, many of the results we will discuss today and
next time go through to degree-heterogeneous graphs, for this reason. But some of the results, in
particular the result having to do with expanders being least like low-dimensional Euclidean space,
do not.)

6.3

Statement of the basic Cheeger Inequality

We know the λ2 captures a “hard” notion of connectivity, since the above result in Theorem 10
states that λ2 = 0 ⇔ G is disconnected. Can we get a “soft” version of this?
To do so, let’s go back to d-regular graphs, and recall the definition.

Definition 22. Let G = (V, E) be a d-regular graph, and let S, S̄ be a cut, i.e., a partition of the
vertex set. Then,
• the sparsity of S is: σ (S) =

E (S,S̄ )
d
|S|·|S̄|
|V |

• the edge expansion of S is: φ (S) =

E (S,S̄ )
d|S|

This definition holds for sets of nodes S ⊂ V , and we can extend them to hold for the graph G.
Definition 23. Let G = (V, E) be a d-regular graph. Then,
• the sparsity of G is: σ (G) = minS⊂V :S6=0,S6=V σ (S).
• the edge expansion of G is: φ (G) = minS⊂V :|S|≤ |V | φ (S).
2

For d-regular graphs, the graph partitioning problem is to find the sparsity or edge expansion of G.
Note that this means finding a number, i.e., the value of the objective function at the optimum,
but people often want to find the corresponding set of nodes, and algorithms can do that, but the
“quality of approximation” is that number.
Fact. For all d regular graphs G, and for all S ⊂ V s.t. |S| ≤
1
σ (S) ≤ φ (S) ≤ σ (S) .
2

|V |
2 ,

we have that

70

M. W. Mahoney


Thus, since σ (S) = σ S̄ , we have that

1
σ (G) ≤ φ (G) ≤ σ (G) .
2

BTW, this is what we mean when we say that these two objectives are “equivalent” or “almost
equivalent,” since that factor of 2 “doesn’t matter.” By this we mean:
• If one is interested in theory, then this factor of 2 is well below the guidance that theory can
provide. That is, this objective is intractable to compute exactly, and the only approximation
algorithms give quadratic or logarithmic (or square root of log) approximations. If they could
provide 1 ± ǫ approximations, then this would matter, but they can’t and they are much
coarser than this factor of 2.
• If one is interested in practice, then we can often do much better than this factor-of-2 improvement with various local improvement heuristics.
• In many cases, people actually write one and optimize the other.
• Typically in theory one is most interested in the number, i.e., the value of the objective, and
so we are ok by the above comment. On the other hand, typically in practice, one is interested
in using that vector to do things, e.g., make statements that the two clusters are close; but
that requires stronger assumptions to say nontrivial about the actual cluster.
Given all that, here is the basic statement of Cheeger’s inequality.
Theorem 11 (Cheeger’s Inequality). Recall that
xT Lx
T
x:x⊥~1 x:x⊥~1 x x

λ2 = min max
where L = I − d1 A. Then,

6.4

p
λ2
≤ φ(G) ≤ 2λ2 .
2

Comments on the basic Cheeger Inequality

Here are some notes about the basic Cheeter Inequality.
• This result “sandwiches” λ2 and φ close to each other on both sides. Clearly, from this result
it immediatly follows that
φ(G)2
≤ λ2 ≤ 2φ(G).
2
• Later, we will see that φ(G) is large, i.e., is bounded away from 0, if the graph is wellconnected. In addition, other related things, e.g., that random walks will mix rapidly, will
also hold. So, this result says that λ2 is large if the graph is well-connected and small if the
graph is not well-connected. So, it is a soft version of the hard connectivity statement that
we had before.

Lecture Notes on Spectral Graph Methods

71

• The inequality λ22 ≤ φ(G) is sometimes known as the “easy direction” of Cheeger’s Inequality.
The reason is that the proof is more straightforward and boils down to showing one of two
related things: that you can present a test vector, which is roughly the indicator vector for a
set of interest, and since λ2 is a min of a Rayleigh quotient, then it is lower than the Rayleigh
quotient of the test vector; or that the Rayleigh quotient is a relaxation of the sparsest cut
problem, i.e., it is minimizing the same objective over a larger set.
√
• The inequality φ(G) ≤ 2λ2 is sometimes known as the “hard direction” of Cheeger’s Inequality. The reason is that the proof is constructive and is basically a vanilla spectral partitioning
algorithm. Again, there are two related proofs for the “hard” direction of Cheeger. One way
uses a notion of inequalities over graphs; the other way can be formulated as a randomized
rounding argument.
• Before dismissing the easy direction, note that it gives a polynomial-time certificate that a
graph is expander-like, i.e., that ∀ cuts (and there are 2n of them to check) at least a certain
number of edges cross that cut. (So the fact that is holds is actually pretty strong—we have
a polynomial-time computable certificate of having no sparse cuts, which you can imagine is
of interest since the naive way to check is to check everything.)
Before proceeding, a question came up in the class about whether the upper or lower bound was
more interesting or useful in applications. It really depend on on what you want.
• For example, if you are in a networking application where you are routing bits and you are
interested in making sure that your network is well-connected, then you are most interested
in the easy direction, since that gives you a quick-to-compute certificate that the graph is
well-connected and that your bits won’t get stuck in a bottleneck.
• Alternatively, if you want to run a divide and conquer algorithm or you want to do some sort
of statistical inference, both of which might require showing that you have clusters in your
graph that are well-separated from the rest of the data, then you might be more interested in
the hard direction which provides an upper bound on the intractable-to-compute expansion
and so is a certificate that there are some well-separated clusters.

72

M. W. Mahoney

7

(02/12/2015): Spectral Methods for Partitioning Graphs (2 of
2): Proof of Cheeger’s Inequality

Reading for today.
• Same as last class.
Here, we will prove the easy direction and the hard direction of Cheeger’s Inequality. Recall that
what we want to show is that
p
λ2
≤ φ(G) ≤ 2λ2 .
2

7.1

Proof of the easy direction of Cheeger’s Inequality

For the easy direction, recall that what we want to prove is that
λ2 ≤ σ(G) ≤ 2φ(G).
To do this, we will show that the Rayleigh quotient is a relaxation of the sparsest cut problem.
Let’s start by restating the sparsest cut problem:
σ(G) =

E S, S̄

min

d
|V | |S|

S⊂V :S6=0,S6=V

=

min

d
x∈{0,1}n {~0,~1} n

=

min

d
x∈{0,1}n {~0,~1} n



· |S̄|

P

{u,v}∈E

P

{u,v}∈E

|xu − xv |

P

{u,v}∈V ×V

P

{u,v}∈V ×V

|xu − xv |

|xu − xv |2

|xu − xv |2

,

(10)

where the last equality follows since xu and xv are Boolean values, which means that |xu − xv | is
also a Boolean value.
Next, recall that
λ2 =

|xu − xv |2
P 2
.
d v xv

min

P

min

P

x∈Rn {~0},x⊥~1

{u,v}∈E

Given that, we claim the following.
Claim 2.
λ2 =

d
x∈Rn Span{~1} n

Proof: Note that
X
u,v

|xu − xv |2 =

X

{u,v} |xu

x2u +

uv

= 2n

X
uv

X
v

|xu − xv |2

{u,v}∈E

P

x2v

− xv |2

x2v − 2

−2

X
v

X

(11)

.

(12)

xu xv

uv

xv

!2

.

Lecture Notes on Spectral Graph Methods
Note that for all x ∈ Rn {~0} s.t. x ⊥ ~1, we have that
X

x2v =

v

=

73
P

v

xv = 0, so

1 X
|xu − xv |2
2n u,v
1 X
|xu − xv |2 ,
n
{u,v}

where the first sum is over unordered pairs u, v of nodes, and where the second sum of over ordered
pairs {u, v} (i.e. we double count (u, v) and (v, u) in first sum, but not in second sum). So,
P
P
2
2
{u,v}∈E |xu − xv |
{u,v}∈E |xu − xv |
P 2
=
min
.
min
P
d
2
d v xv
x∈Rn {0},x⊥~1 n
x∈Rn {~0},x⊥~1
{u,v} |xu − xv |
Next, we need to remove the part along the all-ones vector, since the claim doesn’t have that.

To do so, let’s choose an x∗ that maximizes Eqn. (12). Observe the following. If we shift every
coordinate of that vector x∗ by the same constant, then we obtain another optimal solution, since
the shift will cancel in all the expressions in the numerator and denominator. (This works for any
shift, and we will choose a particular shift to get what we want.)
P
So, we can define x′ s.t. x′v = xv − n1 u xu , and note that the entries of x′ sum to zero. Thus
x′ ⊥ ~1. Note we need x 6∈ Span(~1) to have x′ 6= ~0 So,
min

P

d
x∈Rn {0},x⊥~1 n

This establishes the claim.

{u,v}∈E

P

|xu − xv |2

{u,v} |xu

− xv |2

=

min

P

d
x∈Rn Span{~1} n

{u,v}∈E

P

|xu − xv |2

{u,v} |xu

− xv |2

.

⋄
So, from Eqn. (10) and Eqn. (12), it follows that λ is a continuous relaxation of σ(G), and so
λ2 ≤ σ(G), from which the easy direction of Cheeger’s Inequality follows.

7.2

Some additional comments

Here are some things to note.
• There is nothing required or forced on us about the use of this relaxation, and in fact there
are other relaxations. We will get to them later. Some of them lead to traditional algorithms,
and one of them provides the basis for flow-based graph partitioning algorithms.
• Informally, this relaxation says that we can replace x ∈ {0, 1}n or x ∈ {−1, 1}n constraint
with the constraint that x satisfies this “on average.” By that, we mean that x in the relaxed
problem is on the unit ball, but any particular value of x might get distorted a lot, relative to
its “original” {0, 1} or {−1, 1} value. In particular, note that this is a very “global” constraint.
As we will see, that has some good features, e.g., many of the well-known good statistical
properties; but, as we will see, it has the consequence that any particular local pairwise metric
information gets distorted, and thus it doesn’t lead to the usual worst-case bounds that are
given only in terms of n the size of the graph (that are popular in TCS).

74

M. W. Mahoney
• While providing the “easy” direction, this lemma gives a quick low-degree polynomial time
(whatever time it takes to compute an exact or approximate leading nonrtivial eigenvector)
certificate that a given graph is expander-like, in the sense that for all cuts, at least a certain
number of edges cross it.
• There has been a lot of work in recent years using approaches like this one; I don’t know the
exact history in terms of who did it first, but it was explained by Trevisan very cleanly in
course notes he has had, and this and the proof of the other direction is taken from that. In
particular, he describes the randomized rounding method for the other direction. Spielman
has slightly different proofs. These proofs here are a combination of results from them.
• We could have proven this “easy direction” by just providing a test vector. E.g., a test vector
that is related to an indicator vector or a partition. We went with this approach to highlight
similarities and differences with flow-based methods in a week or two.
• The other reason
√ to describe λ2 as a relaxation of σ(G) is that the proof of the other direction
that φ(G) ≤ 2λ2 can be structured as a randomized rounding algorithm, i.e., given a realvalued solution to Eqn. (12), find a similarly good solution to Eqn. (10). This is what we will
do next time.

7.3

A more general result for the hard direction

For the hard direction, recall that what we want to prove is that
p
φ(G) ≤ 2λ2 .

Here, we will state—and then we will prove—a more general result. For the proof, we will use the
randomized rounding method. The proof of this result is algorithmic/constructive, and it can be
seen as an analysis for the following algorithm.
VanillaSpectralPartitioning. Given as input a graph G = (V, E), a vector x ∈ Rn ,
1. Sort the vertices of V in non-decreasing order of values of entries of x, i.e., let V = {v1 , · · · , vn },
where xv1 ≤ · · · ≤ xvn .
2. Let i ∈ [n − 1] be s.t.
max{φ ({v1 , · · · , vi }) , φ ({vi+1 , · · · , vn })},
is minimal.
3. Output S = {v1 , . . . , vi } and S̄ = {vi+1 , . . . vn }.
This is called a “sweep cut,” since it involves sweeping over the input vector and looking at n
(rather than 2n partitions) to find a good partition.
We have formulated this algorithm as taking as input a graph G and any vector x. You might
be more familiar with the version that takes as input a graph G that first compute the leading
nontrivial eigenvector and then performs a sweep cut. We have formulated it the way we did for
two reasons.

Lecture Notes on Spectral Graph Methods

75

• We will want to separate out the spectral partitioning question from the question about how to
compute the leading eigenvector or some approximation to it. For example, say that we don’t
run an iteration “forever,” i.e., to the asymptotic state to get an “exact” answer to machine
precision. Then we have a vector that only approximates the leading nontrivial eigenvector.
Can we still use that vector and get nontrivial results? There are several interesting results
here, and we will get back to this.
• We will want to separate out the issue of global eigenvector to something about the structure
of the relaxation. We will see that we can use this result to get local and locally-biased
partitions, using both optimization and random walk based idea. In particular, we will use
this to generalize to locally-biased spectral methods.
So, establishing the following lemma is sufficient for what we want.
Lemma 9. Let G = (V, E) be a d-regular graph, and let x ∈ Rn be s.t. x ⊥ ~1. Define
R(x) =

P

|xu − xv |2
P
d v x2v

{u,v}∈E

and let S be the side with less than |V |/2 nodes of the output cut of VanillaSpectralPartitioning. Then,
p
φ(S) ≤ 2R(x)
Before proving this lemma, here are several things to note.

• If we apply this lemma to √
a vector x that is an eigenvector of λ2 , then R(x) = λ2 , and so we
have that φ(G) ≤ φ(S) ≤ 2λ2 , i.e., we have the difficult direction of Cheeger’s Inequality.
• On the other hand, any vector whose Rayleigh quotient is close to that of λ2 also gives a good
solution. This “rotational ambiguity” is the usual thing with eigenvectors, and it is different
than any approximation of the relatation to the original expansion IP. We get “goodness”
results for such a broad class of vectors to sweep over since we are measuring goodness rather
modestly: only in terms of objective function value. Clearly, the actual clusters might change
a lot and in general will be very different if we sweep over two vectors that are orthogonal to
each other.
• This result actually holds for vectors x more generally, i.e., vectors that have nothing to
do with the leading eigenvector/eigenvalue, as we will see below with locally-biased spectral
methods, where we will use it to get upper bounds on locally-biased variants of Cheeger’s
Inequality.
√
• Inpthis case, in “eigenvector time,” we have found a set S with expansion s.t. φ(S) ≤ λ2 ≤
2 φ(G).

• This is not a constant-factor approximation, or any nontrivial approximation factor in terms
of n; and it is incomparable with other methods (e.g., flow-based methods) that do provide
such an approximation factor. It is, however, nontrivial in terms of an important structural
parameter of the graph. Moreover, it is efficient and useful in many machine learning and
data analysis applications.

76

M. W. Mahoney
• The above algorithm can be implemented in roughly O (|V | log |V | + |E|) time, assuming
arithmetic operations and comparisons take constant time. This is since once we have computed
E ({v1 , . . . , vi }, {vi+1 , . . . , vn }) ,
it only takes O(degree(vi+1 )) time to compute
E ({v1 , . . . , vi+1 }, {vi+2 , . . . , vn }) .
• As a theoretical point, there exists nearly linear time algorithm to find a vector x such that
R(x) ≈ λ2 , and so by coupling
this algorithm with the above algorithm we can find a cut

p
with expansion O
φ(G) in nearly-linear time. Not surprisingly, there is a lot of work on
providing good implementations for these nearly linear time algorithms. We will return to
some of these later.
• This quadratic factor is “tight,” in that there are graphs that are that bad; we will get to
these (rings or Guattery-Miller cockroach, depending on exactly how you ask this question)
graphs below.

7.4

Proof of the more general lemma implying the hard direction of Cheeger’s
Inequality

Note that λ2 is a relaxation of σ(G) and the lemma provides a rounding algorithtm for real vectors
that are a solution of the relaxation. So, we will view this in terms of a method from TCS known as
randomized rounding. This is a useful thing to know, and other methods, e.g., flow-based methods
that we will discuss soon, can also be analyzed in a similar manner.
For those who don’t know, here is the one-minute summary of randomized rounding.
• It is a method for designing and analyzing the quality of approximation algorithms.
• The idea is to use the probabilistic method to convert the optimal solution of a relaxation
of a problem into an approximately optimal solution of the original problem. (The probabilistic method is a method from combinatorics to prove the existence of objects. It involves
randomly choosing objects from some specified class in some manner, i.e., according to some
probability distribution, and showing that the objects can be found with probability > 0,
which implies that the object exists. Note that it is an existential/non-constructive and not
algorithmic/constructive method.)
• The usual approach to use randomized rounding is the following.
– Formulate a problem as an integer program or integer linear program (IP/ILP).
– Compute the optimal fractional solution x to the LP relaxation of this IP.
– Round the fractional solution x of the LP to an integral solution x′ of the IP.
• Clearly, if the objective is a min, then cost(x) ≤ cost(x′ ). The goal is to show that cost(x′ ) is
not much more that cost(x).

Lecture Notes on Spectral Graph Methods

77

• Generally, the method involves showing that, given any fractional solution x of the LP, w.p.
> 0 the randomized rounding procedure produces an integral solution x′ that approximated
x to some factor.
• Then, to be computationally efficient, one must show that x′ ≈ x w.h.p. (in which case the
algorithm can stay randomized) or one must use a method like the method of conditional
probabilities (to derandomize it).
Let’s simplify notation: let V = {1, . . . , n}; and so x1 ≤ x2 ≤ · · · xn . In this case, the goal is to
show that there exists i ∈ [n] w.t.
φ ({1, . . . , i}) ≤

p

2R(x) and

φ ({i + 1, . . . , n}) ≤

p

2R(x).

We will prove the lemma by showing that there exists a distribution D over sets S of the form
{1, . . . , i} s.t.

p
ES∼D E(S, S̄)

(13)
≤ 2R(x).
ES∼D d min{|S|, |S̄|}

Before establishing this, note that Eqn. (13) does not imply the lemma. Why? In general, it is the

case that E X
6= E{X}
Y
E{Y } , but it suffices to establish something similar.
Fact. For random variables X and Y over the sample space, even though E
case that


X
E {X}
P
≤
> 0,
Y
E {Y }

X
Y

6=

E{X}
E{Y } ,

it is the

provided that Y > 0 over the entire sample space.
But, by linearity of expectation, from Eqn. (13) it follows that
h
i
p
ES∼D E(S, S̄) − d 2R(x) min{|S|, |S̄|} ≤ 0.
So, there exists a set S in the sample space s.t.
p
E(S, S̄) − d 2R(x) min{|S|, |S̄|} ≤ 0.

That is, for S and S̄, at least on of which has size ≤ n2 ,
φ(S) ≤
from which the lemma will follow.

p
2R(x),

So, because of this, it will suffice to establish Eqn. (13). So, let’s do that.
Assume, WLOG, that x⌈ n2 ⌉ = 0, i.e., the median of the entires of x equals 0; and x21 + x2n = 1. This
is WLOG since, if x ⊥ ~1, then adding a fixed constant c to all entries of x can only decrease the

78

M. W. Mahoney

Rayleigh quotient:
|(xu + c) − (xv + c)|2
P
d v (xv + c)2
P
2
{(u,v)}∈E |xu − xv |
P 2
P
=
d
x − 2dc v xv + nc2
P v v
2
{(u,v)}∈E |xu − xv |
P
=
d v x2v + nc2
≤ R(x).

R (x + (c, . . . , c)) =

P

{(u,v)}∈E

Also, multiplying all entries by fixed constant does not change the value of R(x), nor does it change
the property that x1 ≤ · · · ≤ xn .
We have made these choices since they will allow us to define a distribution D over sets S s.t.
X

ES∼D min |S|, |S̄| =
x2i
(14)
i

Define a distribution D over sets {1, . . . , i}, 1 ≤ i ≤ n − 1, as the outcome of the following
probabilistic process.
1. Choose a t ∈ [x1 , xn ] ∈ R with probability density function equal to f (t) = 2|t|, i.e., for
x1 ≤ a ≤ b ≤ xn , let
 2
Z b
|a − b2 |
if a and b have the same sign
2|t|dt =
P [a ≤ t ≤ b] =
,
2 + b2
a
if a and b have different signs
a
2. Let S = {u : xi ≤ t}
From this definition
• The probability that an element i ≤ n2 belongs to the smaller of the sets S, S̄ equals the
probability of i ∈ S and |S| ≤ |S̄|, which equals the probability that the threshold t is in the
range [xi , 0], which equals x2i .
• The probability that an element i > n2 belongs to the smaller of the sets S, S̄ equals the
probability of i ∈ S̄ and |S| ≥ |S̄|, which equals the probability that the threshold t is in the
range [0, xi ], which equals x2i .
So, Eqn. (14) follows from linearity of expectation.
Next, we want to estimate the expected number of edges between S and S̄, i.e.,
X




E E S, S̄ =
P edge (i, j) is cut by (S, S̄) .
(i,j)∈E

To estimate this, note that the event that the edge (i, j) is cut by the partition (S, S̄) happens
when t falls in between xi and xj . So,

Lecture Notes on Spectral Graph Methods

79

• if xi and xj have the same sign, then


P edge (i, j) is cut by (S, S̄) = |x2i − x2j |

• if xi and xj have the different signs, then


P edge (i, j) is cut by (S, S̄) = x2i + x2j

The following expression is an upper bound that covers both cases:


P edge (i, j) is cut by (S, S̄) ≤ |xi − xj | · (|xi | + |xj |) .

Plugging into the expressions for the expected number of cut edges, and applying the CauchySchwatrz inequality gives
X

EE S, S̄ ≤
|xi − xj | (|xi | + |xj |)
(i,j)∈E

≤

Finally, to deal with the expression
X

(ij)∈E

s X

(i,j)∈E

P

(ij)∈E

(xi − xj )

2

s X

(i,j)∈E

(|xi | + |xj |)2

(|xi | + |xj |)2 , recall that (a + b)2 ≤ 2a2 + 2b2 . Thus,

(|xi | + |xj |)2 ≤

X

(ij)∈E

2x2i + 2x2j = 2d

X

x2i .

i

Putting all of the pieces together, we have that
q P
qP


2
(x
−
x
)
2d i x2i
p
i
j
E E S, S̄
(ij)∈E

≤
P 2
= 2R(x),
d i xi
E d min{|S|, |S̄|}
from which the result follows.

80

M. W. Mahoney

8

(02/17/2015): Expanders, in theory and in practice (1 of 2)

Reading for today.
• “Expander graphs and their applications,” in Bull. Amer. Math. Soc., by Hoory, Linial, and
Wigderson

8.1

Introduction and Overview

Expander graphs, also called expanders, are remarkable structures that are widely-used in TCS and
discrete mathematics. They have a wide range of applications:
• They reduce the need for randomness and are useful for derandomizing randomized algorithms—
so, if random bits are a valuable resource and thus you want to derandomized some of the
randomized algorithms we discussed before, then this is a good place to start.
• They can be used to find good error-correcting codes that are efficiently encodable and
decodable—roughly the reason is that they spread things out.
• They can be used to provide a new proof of the so-called PCP theorem, which provides a new
characterization of the complexity class NP, and applications to the hardness of approximate
computation.
• They are a useful concept in data analysis applications, since expanders look random, or are
empirically quasi-random, and it is often the case that the data, especially when viewed at
large, look pretty noisy.
For such useful things, it is somewhat surprising that (although they are very well-known in computer science and TCS in particular due to their algorithmic and complexity connections) expanders
are almost unknown outside computer science. This is unfortunate since:
• The world is just a bigger place when you know about expanders.
• Expanders have a number of initially counterintuitive properties, like they are very sparse
and very well-connected, that are typical of a lot of data and thus that are good to have an
intuition about.
• They are “extremal” in many ways, so they are a good limiting case if you want to see how
far you can push your ideas/algorithms to work.
• Expanders are the structures that are “most unlike” low-dimensional spaces—so if you don’t
know about them then your understanding of the mathematical structures that can be used
to describe data, as well as of possible ways that data can look will be rather limited, e.g.,
you might think that curved low-dimensional spaces are good ideas.
Related to the comment about expanders having extremal properties, if you know how your algorithm behaves on, say, expanders, hypercubes (which are similar and different in interesting ways),
trees (which we won’t get to as much, but will mention), and low-dimensional spaces, they you
probably have a pretty good idea of how it will behave on your data. That is very different than

Lecture Notes on Spectral Graph Methods

81

knowing how it will behave in any one of those places, which doesn’t give you much insight into how
it will behave more generally; this extremal property is used mostly by TCS people for algorithm
development, but it can be invaluable for understanding how/when your algorithm works and when
it doesn’t on your non-worst-case data.
We will talk about expander graphs. One issue is that we can define expanders both for degreehomogeneous graphs as well as for degree-heterogeneous graphs; and, although many of the basic
ideas are similar in the two cases, there are some important differences between the two cases. After
defining them (which can be done via expansion/conductance or the leading nontrivial eigenvalue
of the combinatorial/normalized Laplacian), we will focus on the following aspects of expanders
and expander-like graphs.
• Expanders are graphs that are very well-connected.
• Expanders are graphs that are sparse versions/approximations of a complete graph.
• Expanders are graphs on which diffusions and random walks mix rapidly.
• Expanders are the metric spaces that are least like low-dimensional Euclidean spaces.
Along the way, we might have a chance to mention a few other things, e.g.: how big λ2 could
be with Ramanujan graphs and Wigner’s semicircle result; trivial ways with dmax to extend the
Cheeger Inequality to degree-heterogeneous graphs, as well as non-trivial ways with the normalized
Laplacian; pseudorandom graphs, converses, and the Expander Mixing Lemma; and maybe others.
Before beginning with some definitions, we should note that we can’t draw a meaningful/interpretable
picture of an expander, which is unfortunate since people like to visualize things. The reason for that
is that there are no good “cuts” in an expander—relatedly, they embed poorly in low-dimensional
spaces, which is what you are doing when you visualize on a two-dimensional piece of paper. The
remedy for this is to compute all sorts of other things to try to get a non-visual intuition about
how they behave.

8.2

A first definition of expanders

Let’s start by working with d-regular graphs—we’ll relax this regularity assumption later. But many
of the most extremal properties of expanders hold for degree-regular graphs, so we will consider
them first.
Definition 24. A graph G = (V, E) is d-regular if all vertices have the same degree d, i.e., each
vertex is incident to exactly d edges.
Also, it will be useful to have the following notion of the set of edges between two sets S and T (or
from S to T ), both of which are subsets of the vertex set (which may or may not be the complement
of each other).
Definition 25. For S, T ⊂ V , denote
E(S, T ) = {(u, v) ∈ E| u ∈ S, v ∈ T }.
Given this notation, we can define the expansion of a graph. (This is slightly different from other
definitions I have given.)

82

M. W. Mahoney

Definition 26. The expansion or edge expansion ratio of a graph G is
E(S, S̄)
|S|

h(G) = min n
S:|S|≤ 2

Note that this is slightly different (just in terms of the scaling) than the edge expansion of G which
we defined before as:

E S, S̄
.
φ (G) =
min
|V |
d|S|
S⊂V :|S|≤
2

We’ll use this today, since I’ll be following a proof from HLW, and they use this, and following
their notation should make it easier. There should be no surprises, except just be aware that there
is a factor of d difference from what you might expect.
(As an aside, recall that there are a number of extensions of this basic idea to measure other or
more fine versions of this how well connected is a graph:
• Different notions of boundary—e.g., vertex expansion.
• Consider size-resolved minimum—in Markov chains and how good communities are as a
function of size.
• Different denominators, which measure different notions of the “size” of a set S:
S̄)
—this is equivalent to expansion in a certain sense that
– Sparsity or cut ratio: min E(S,
|S|·|S̄|
we will get to.

– Conductance or NCut—this is identical for d-regular graphs but is more useful in practice
and gives tighter bounds in theory if there is degree heterogeneity.
We won’t deal with these immediately, but we will get back to some later. This ends the aside.)
In either case above, the expansion is a measure to quantify how well-connected is the graph. Given
this, informally we call a d-regular graph G an expander if h(G) ≥ ǫ where ǫ is a constant. More
precisely, let’s define an expander:
Definition 27. A graph G is a (d, ǫ)-expander if it is d-regular and h(G) ≥ ǫ, where ǫ is a constant,
independent of n.
Alternatively, sometimes expansion is defined in terms of a sequence of graphs:
Definition 28. A sequence of d-regular graphs {Gi }i∈Z + is a family of expander graphs if ∃ǫ > 0
s.t. h(Gi ) ≥ ǫ, ∀i.
If we have done the normalization correctly, then h(G) ∈ [0, d] and φ(G) ∈ [0, 1], where large means
more expander-like and small means that there are good partitions. So, think of the constant ǫ as
d/10 (and it would be 1/10, if we used φ(G) normalization). Of course, there is a theory/practice
issue here, e.g., sometimes you are given a single graph and sometimes it can be hard to tell a
moderately large constant from a factor of log(n); we will return to these issues later.

Lecture Notes on Spectral Graph Methods

8.3

83

Alternative definition via eigenvalues

Although expanders can be a little tricky and counterintuitive, there are a number of ways to deal
with them. One of those ways, but certainly not the only way, is to compute eigenvectors and
eigenvalues associated with matrices related to the graph. For example, if we compute the second
eigenvalue of the Laplacian, then we have Cheeger’s Inequality, which says that if the graph G is
an expander, then we have a (non-tight, due to the quadratic approximation) bound on the second
eigenvalue, and vice versa. That is, one way to test if a graph is an expander is to compute that
eigenvalue and check.
, which is the Fiedler value or second smallest eigenvalue
Of central interest to a lot of things is λLAP
2
of the Laplacian. Two things to note:
• If we work with Adjacency matrices rather than Laplacians, then we are interested in how
far λADJ
is from d.
2
• We often normalized things so as to interpret them in terms of a random walk, in which case
the top eigenvalue = 1 with the top eigenvector being the probability distribution. In that
case, we are interested in how far λ2 is from 1.
(Since I’m drawing notes from several different places, we’ll be a little inconsistent on what the
notation means, but we should be consistent within each class or section of class.)
Here is Cheeger’s Inequality, stated in terms of h(G) above.
• If 0 = λ1 ≤ λ2 ≤ · · · ≤ λn are the eigenvalues of the Laplacian (not normalized, i.e. D − A)
of a d-regular graph G, then:
p
λ2
≤ h(G) ≤ 2dλ2
2
√
The d in the upper bound is due to our scaling.
Alternatively, here is Cheeger’s Inequality, stated in terms of h(G) for an Adjacency Matrix.
• If d = µ1 ≥ µ2 ≥ . . . ≥ µn are the eigenvalues of the Adjacency Matrix A(G) of d-regular
graph G, then:
p
d − µ2
≤ h(G) ≤ 2d(d − µ2 )
2
Therefore, the expansion of the graph is related to its spectral gap (d − µ2 ). Thus, we can define
a graph to be an expander if µ2 ≤ d − ǫ or λ2 ≥ ǫ where λ2 is the second eigenvalue of the matrix
L(G) = D − A(G) where D is the diagonal degree matrix. Slightly more formally, here is the
alternate definition of expanders:
Definition 29. A sequence of d-regular graphs {Gn }n ∈ N is a family of expander graphs if
|λADJ
| ≤ d − ǫ, i.e. if all the eigenvalues of A are bounded away from d
i
Remark. The last requirement can be written as λLAP
≥ c, ∀n, i.e., that all the eigenvalues of
2
the Laplacian bounded below and away from c > 0.
In terms of the edge expansion φ(G) we defined last time, this definition would become the following.

84

M. W. Mahoney

Definition 30. A family of constant-degree expanders is a family of graphs {Gn }n∈N s.t. each
graph in Gn is d-regular graph on n vertices and such that there exists an absolute constant φ∗ ,
independent of n, s.t. φ(Gn ) ≥ φ∗ , for all n.

8.4

Expanders and Non-expanders

A clique or a complete graph is an expander, if we relax the requirement that the d-regular graph
is also an
have a fixed d, independent of n. Moreover, Gn,p (the random graph), for p & log(n)
n
expander, with d growing only weakly with n. (We may show that later.) Of greatest interest—at
least for theoretical considerations—is the case that d is a constant independent of n.

8.4.1

Very sparse expanders

In this case, the idea of an expander, i.e., an extremely sparse and extremely well-connected graph
is nice; but do they exist? It wasn’t obvious until someone proved it, but the answer is YES. In
fact, a typical d-regular graph is an expander with high probability under certain random graph
models. Here is a theorem that we will not prove.
Theorem 12. Fix d ∈ Z+ ≥ 3. Then, a randomly chosen d-regular graph is an expander w.h.p.
Remark. Clearly, the above theorem is false if d = 1 (in which case we get a bunch of edges) or if
d = 2 (in which case we get a bunch of cycles); but it holds even for d = 3.
Remark. The point of comparison for this should be if d & log(n)
n . In this case, “measure
concentration” in the asymptotic regime, and so it is plausible (and can be proved to be true)
that the graph has no good partitions. To understand this, recall that one common random graph
model is the Erdos-Renyi Gn,p model, where there are n vertices and edges are chosen to exist with
probability p. (We will probably describe this ER model as well as some of its basic properties later;
at a minimum, we will revisit it when we talk about stochastic blockmodels.) The related Gn,m
model is another common model where graphs with n vertices and m edges are chosen uniformly
at random. An important fact is that if we set p such that there are on average m edges, then
Gn,m is very similar (in strong senses of the word) to Gn,p —if p ≥ log n/n. (That is the basis for
the oft-made observation that Gn,m and Gn,p are “the same.”) However, for the above definition
of expanders, we require in addition that d is a constant. Importantly, in that regime, the graphs
are sparse enough that measure hasn’t concentrated, and they are not the same. In particular, if
p = 3/n, Gn,p usually generates a graph that is not connected (and there are other properties that
we might return to later). However, (by the above theorem) Gn,m with corresponding parameters
usually yields a connected graph with very high expansion. We can think of randomized expander
construction as a version of Gn,m , further constrained to d-regular graphs.
Remark. There are explicit deterministic constructions for expanders—they have algorithmic
applications. That is an FYI, but for what we will be doing that won’t matter much. Moreover,
later we will see that the basic idea is still useful even when we aren’t satisfying the basic definition
of expanders given above, e.g., when there is degree heterogeneity, when a graph has good small
but no good large cuts, etc.

Lecture Notes on Spectral Graph Methods
8.4.2

85

Some non-expanders

It might not be clear how big is big and how small is small—in particular, how big can h (or λ)
be. Relatedly, how “connected” can a graph be? To answer this, let’s consider a few graphs.
• Path graph. (For a path graph, µ1 = Θ(1/n2 ). If we remove 1 edge, then we can cut the
graph into two 50-50 pieces.
√
√
√
√
• Two-dimensional n × n grid. (For a n × n grid, µ1 = Θ(1/n).) Here, you can’t
disconnect the graph by removing 1 edge, and the removal of a constant number of edges can
√
only remove a constant number of vertices from the graph. But, it is possible to remove n
of the edge, i.e., an O( √1n ) fraction of the total, and split the graph into two 50-50 pieces.
• For a 3D grid, µ1 = Θ(1/n2/3 ).
• A k-dimensional hypercube is still better connected. But it is possible to remove a very small
1
fraction of the edges (the edges of a dimension cut, which are k1 = log(n)
fraction of the total)
and split half the vertices from the other half.
• For a binary tree, e.g., a complete binary tree on n vertices, µ1 = Θ(1/n).
• For a Kn − Kn dumbbell, (two expanders or complete graphs joined by an edge) µ1 = Θ(1/n).
• For a ring on n vertices, µ1 = Θ(1/n).
• Clique. Here, to remove a p fraction of the vertices from the rest, you must remove ≥ p(1 − p)
fraction of the edges. That is, it is very well connected. (While can take a complete graph to
be the “gold standard” for connectivity, it does, however, have the problem that it is dense;
thus, we will be interested in sparse versions of a complete that are similarly well-connected.)
• For an expander, µ1 = Θ(1).
Remark. A basic question to ask is whether, say, µ1 ∼ Θ(poly(1/n)) is “good” or “bad,” say,
in some applied sense? The answer is that it can be either: it can be bad, if you are interested
in connectivity, e.g., a network where nodes are communication devices or computers and edges
correspond to an available link; or it can be good, either for algorithmic reasons if e.g. you are
interested in divide and conquer algorithms, or for statistical reasons since this can be used to
quantify conditional independence and inference.
Remark. Recall the quadratic relationship between d − λ2 and h. If d − λ2 is Θ(1), then that
is not much difference (a topic which will return to later), but if it is Θ(1/n) or Θ(1/n2 ) then
it makes a big difference. A consequence of this is that by TCS standards, spectral partitioning
does a reasonably-good job partitioning expanders (basically since the quadratic of a constant is
a constant), while everyone else would wonder why it makes sense to partition expanders; while
by TCS standards, spectral partitioning does not do well in general, since it has a worst-case
approximation factor that depends on n, while everyone else would say that it does pretty well on
their data sets.

86

M. W. Mahoney

8.4.3

How large can the spectral gap be?

A question of interest is: how large can the spectral gap be? The answer here depends on the
relationship between n, the number of nodes in the graph and d, the degree of each node (assumed
to be the same for now.) In particular, the answer is different if d is fixed as n grows or if d grows
with n as n grows. As as extreme example of the latter case, consider the complete graph Kn on
n vertices, in which case d = n − 1. The adjacency matrix of Kn is AKn = J − I, where J is the
all-ones matrix, and where I = In is the diagonal identity matrix. The spectrum of the adjacency
matrix of Kn is {n − 1, −1, . . . , −1}, and λ = 1. More interesting for us here is the case that d is
fixed and n is large, in which case n ≫ d, in which case we have the following theorem (which is
due to Alon and Boppana).
Theorem 13 (Alon-Boppana). Denoting λ = max(|µ2 |, |µn |), we have, for every d-regular graph:
√
λ ≥ 2 d − 1 − on (1)
√
So, the eigengap d − µ2 is not larger than d − 2 d − 1. For those familiar with Wigner’s semicircle
law, note the similar form.
The next question is: How tight is this? In fact, it is pretty close to tight in the following sense:
there exists constructions
of graphs, called Ramanujan graphs, where the second eigenvalue of L(G)
√
is λ1 (G) = d − 2 d − 1, and so the tightness is achieved. Note also that this is of the same scale as
Wigner’s semicircle law; the precise statements are somewhat different, but the connection should
not be surprising.

8.5

Why is d fixed?

A question that arises is why is d fixed in the definition, since there is often degree variability in
practice. Basically that is since it makes things harder, and so it is significant that expanders exist
even then. Moreover, for certain theoretical issues that is important. But, in practice the idea of
an expander is still useful, and so we go into that here.
We can define expanders: i.t.o. boundary expansion; or i.t.o. λ2 . The intuition is that it is
well-connected and then get lots of nice properties:
• Well-connected, so random walks converge fast.
• Quasi-random, meaning that it is empirically random (although in a fairly weak sense).
Here are several things to note:
• Most theorems in graph theory go through to weighted graphs, if you are willing to have factors
max
like w
wmin —that is a problem if there is very significant degree heterogeneity or heterogeneity
in weights, as is common. So in that case many of those results are less interesting.
• In many applications the data are extremely sparse, like a constant number of edges on
average (although there may be a big variance).
• There are several realms of d, since it might not be obvious what is big and what is small:

Lecture Notes on Spectral Graph Methods

87

– d = n: complete (or nearly complete) graph.
– d = Ω(polylog(n)): still dense, certainly in a theoretical sense, as this is basically the
asymptotic region.
– d = Θ(polylog(n)): still sufficiently dense that measure concentrates, i.e., enough concentration for applications; Harr measure is uniform, and there are no “outliers”
– d = Θ(1): In this regime things are very sparse, Gnm =
6 Gnp , so you have a situation
where the graph has a giant component but isn’t fully connected; so 3-regular random
graphs are different than Gnp with p = n3 .
You should think in terms of d = Θ(polylog(n)) at most, although often can’t tell O(log n)
versus a big constant, and comparing trivial statistics can hide what you want.
• The main properties we will show will generalize to degree variability. In particular:
– High expansion → high conductance.

– Random walks converge to “uniform” distribution → random walks converge to a distribution that is uniform over the edges, meaning proportional to the degree of a node.
– Expander Mixing Property → Discrepancy and Empirical Quasi-randomness
So, for theoretical applications, we need d = Θ(1); but for data applications, think i.t.o. a graph
being expander-like, i.e., think of some of the things we are discussing as being relevant for the
properties of that data graph, if: (1) it has good conductance properties; and (2) it is empirically
quasi-random. This happens when data are extremely sparse and pretty noisy, both of which they
often are.

8.6

Expanders are graphs that are very well-connected

Here, we will describe several results that quantify the idea that expanders are graphs that are very
well-connected.
8.6.1

Robustness of the largest component to the removal of edges

Here is an example of a lemma characterizing how constant-degree graphs with constant expansion
are very sparse graphs with extremely good connectivity properties. In
 words, what the following
k
lemma says is that the removal of k edges cannot cause more that O d vertices to be disconnected
from the rest. (Note that it is always possible to disconnect kd vertices after removing k edges, so
the connectivity of an expander is the best possible.)
Lemma 10. Let G = (V, E) be a regular graph with expansion φ. Then, after an ǫ < φ fraction of
ǫ
the edges are adversarially removed, the graph has a connected component that has at least 1 − 2φ
fraction of the vertices.
Proof: Let d be the degree of G. Let E ′ ⊆ E be an arbitrary subset of ≤ ǫ|E| = ǫd |V2 | edges. Let
C1 , . . . , Cm be the connected components of the graph (V, EE ′ ), ordered s.t.
|C1 | ≥ |C2 | ≥ · · · |Cm |.

88

M. W. Mahoney

In this case, we want to prove that


2ǫ
|C1 | ≥ |V | 1 −
φ
To do this, note that
|E ′ | ≥
So, if |C1 | ≤

|V |
2 ,

1X
1X
E (Ci , Cj ) =
E (Ci , V Ci ) .
2
2
ij

i

then
|E ′ | ≥

1X
1
dφ|Ci | = dφ|V |,
2
2
i

which is a contradiction if φ > ǫ. On the other hand, if |C1 | ≥
S = C2 ∪ . . . ∪ Cm . Then, we have

|V |
2 ,

then let’s define S to be

|E ′ | ≥ E(C1 , S) ≥ dφ|S|,
which implies that
|S| ≤

and so |C1 ≥ 1 −
8.6.2

ǫ
2φ



ǫ
|V |,
2φ

|V |, from which the lemma follows.

⋄

Relatedly, expanders exhibit quasi-randomness

In addition to being well-connected in the above sense (and other senses), expanders also “look
random” in certain senses.
One direction For example, here I will discuss connections with something I will call “empirical
quasi-randomness.” It is a particular notion of things looking random that will be useful for what
we will discuss. Basically, it says that the number of edges between any two subsets of nodes is
very close to the expected value, which is what you would see in a random graph. Somewhat more
precisely, it says that when λ below is small, then the graph has the following quasi-randomness
property: for every two disjoint sets of vertices, S and T , the number of edges between S and T is
close to nd |S| · |T |, i.e., what we would expect a random graph with the same average degree d to
have. (Of course, this could also hide other structures of potential interest, as we will discuss later,
but it is a reasonable notion of “looking random” in the large scale.) Here, I will do it in terms of
expansion—we can generalize it and do it with conductance and discrepancy, and we may do that
later.
We will start with the following theorem, called the “Expander Mixing Lemma,” which shows that
if the spectral gap is large, then the number of edges between two subsets of the graph vertices
can be approximated by the same number for a random graph, i.e., what would be expected on
average, so it looks empirically random. Note that nd |S| · |T | is the average value of the number of
p
edges between the two sets of nodes in a random graph; also, note that λ |S| · |T | is an “additive”
scale factor, which might be very large, e.g., too large for the following lemma to give an interesting
bound, in particular when one of S or T is very small.

Lecture Notes on Spectral Graph Methods

89

Theorem 14 (Expander Mixing Lemma). Let G = (V, E) be a d-regular graph, with |V | = n
and λ = max(|µ2 |, |µn |), where µi is the i-th largest eigenvalue of the (non-normalized) Adjacency
Matrix. Then, for all S, T ⊆ V , we have the following:
|E(S, T )| −

p
d
|S| · |T | ≤ λ |S| · |T |.
n

Proof. Define χS and χT to be the characteristic vectors of S and T . Then, if {vj }nj=1 are orthonormal eigenvectors of AG , with v1 = √1n (1, . . . , 1), then we can write the expansion of χS and χT in
P
P
terms of those eigenvalues as: χS = i αi vi and χT = j βj vj . Thus,
|E(S, T )| = χTS AχT
X

=

αi vi

i

=

!



A

j



βj vj 


!
X
X
αi vi 
µj βj vj 
i

=

X

X

j

µi αi βi

since the vi ’s are orthonormal.

i

Thus,
X

|E(S, T )| =

µi αi βi

= µ1 α1 β1 +

X

µi αi βi

i≥2

= d

|S|.|T | X
+
µi αi βi ,
n
i≥1

−
→

where the last inequality is because, α1 = hχS , √1n i =
Hence,
d
|E(S, T )| − |S| · |T |
n

=

n
X

|S|
√
n

and (similarly) β1 =

µi αi βi

i=2

≤

X
i≥2

≤ λ

|µi αi βi |

X
i≥1

|αi ||βi |

≤ λ||α||2 ||β||2 = λ||χS ||2 ||χT ||2 = λ

Other direction

|T |
√
,
n

There is also a partial converse to this result:

p

|S| · |T |

and µ1 = d.

90

M. W. Mahoney

Theorem 15 (Bilu and Linial). Let G be a d-regular graph, and suppose that
E(S, T ) −

p
d
|S| · |T | ≤ ρ |S| · |T |
n

holds ∀ disjoint S,T and for some ρ > 0. Then

 
d
λ ≤ O ρ 1 + log( )
ρ
8.6.3

Some extra comments

We have been describing these results in terms of regular and unweighted graphs for simplicity,
especially of analysis since the statements of the theorems don’t change much under generalization.
Important to note: these results can be generalized to weighted graphs with irregular number
of edges per nodes using discrepancy. Informally, think of these characterizations as intuitively
defining what the interesting properties of an expander are for real data, or what an expander is
more generally, or what it means for a data set to look expander-like.
Although we won’t worry too much about those issues, it is important to note that for certain,
mostly algorithmic and theoretical applications, the fact that d = Θ(1), etc. are very important.

8.7

Expanders are graphs that are sparse versions/approximations of a complete graph

To quantify the idea that constant-degree expanders are sparse approximations to the complete
graph, we need two steps:
1. first, a way to say that two graphs are close; and
2. second, a way to show that, with respect to that closeness measure, expanders and the
complete graph are close.
8.7.1

A metric of closeness between two graphs

For the first step, we will view a graph as a Laplacian and vice versa, and we will consider the
partial order over PSD matrices. In particular, recall that for a symmetric matrix A, we can write
A0
to mean that
A ∈ P SD

(and, relatedly, A ≻ 0 to mean that it is PD). In this case, we can write A  B to mean that
A−B  0. Note that  is a partial order. Unlike the real numbers, where every pair is comparable,
for symmetric matrices, some are and some are not. But for pairs to which it does apply, it acts
like a full order, in that, e.g.,
A  B and B  C implies A  C

A  B implies that A + C  B + C,

Lecture Notes on Spectral Graph Methods

91

for symmetric matrices A, B, and C.
By viewing a graph as its Laplacian, we can use this to define an inequality over graphs. In
particular, for graphs G and H, we can write
G  H to mean that LG  LH .

In particular, from our previous results, we know that if G = (V, E) is a graph and H = (V, F )
is a subgraph of G, then LG  LH . This follows since the Laplacian of a graph is the sum of the
Laplacians of its edges: i.e., since F ⊆ E, we have
X
X
X
X
LG =
Le =
Le +
Le 
Le = LH ,
einE

which follows since

P

e∈EF

e∈F

einEF

e∈F

Le  0.

That last expression uses the additive property of the order; now let’s look at the multiplicative
property that is also respected by that order.
If we have a graph G = (V, E) and a graph H = (V, E ′ ), let’s define the graph c · H to be the
same as the graph H, except that every edge is multiplied by c. Then, we can prove relationships
between graphs such as the following.
Lemma 11. If G and H are graphs s.t.
then, for all k we have that

Gc·H
λk (G) ≥ cλk (H).

Proof: The proof is by the min-max Courant-Fischer variational characterization. We won’t do it
in detail. See DS, 09/10/12.
⋄
From this, we can prove more general relationships, e.g., bounds if edges are removed or rewieghted.
In particular, the following two lemmas are almost corollaries of Lemma 11.
Lemma 12. If G is a graph and H is obtained by adding an edge to G or increasing the weight of
an edge in G, then, for all i, we have that λi (G) ≤ λi (H).
Lemma 13. If G = (V, E, W1 ) is a graph and H = (V, E, W2 ) is a graph that differs from G only
in its weights, then
w1 (e)
H.
G  min
e∈E w2 (e)

Given the above discussion, we can use this to define the notion that two graphs approximate
each other, basically by saying that they are close if their Laplacian quadratic forms are close. In
particular, here is the definition.
Definition 31. Let G and H be graphs. We say that H is a c-approximation to H if
1
cH  G  H.
c
As a special case, note that if c = 1 + ǫ, for some ǫ ∈ (0, 1), then we have that the two graphs are
very close.

92
8.7.2

M. W. Mahoney
Expanders and complete graphs are close in that metric

Given this notion of closeness between two graphs, we can now show that constant degree expanders
are sparse approximations of the complete graph. The following theorem is one formalization of
this idea. This establishes the closeness; and, since constant-degree expanders are very sparse, this
result shows that they are sparse approximations of the complete graph. (We note in passing that
it is know more generally that every graph can be approximated by a complete graph; this graph
sparsification problem is of interest in many areas, and we might return to it.)
Theorem 16. For every ǫ > 0, there exists a d > 0 such that for all sufficiently large n, there is a
d regular graph Gn that is a 1 ± ǫ approximation of the complete graph Kn
Proof: Recall that a constant-degree expander is a d-regular graph whose Adjacency Matrix eigenvalues satisfy
|αi | ≤ ǫd,
(15)
for all i ≥ 2, for some ǫ < 1. We will show that graphs satisfying this condition also satisfy the
condition of Def. 31 (with c = 1 + ǫ) to be a good approximation of the complete graph.
To do so, recall that
(1 − ǫ) H  G  (1 + ǫ) H
means that
(1 − ǫ) xT LH x ≤ xT LG x ≤ (1 + ǫ) xT LH x.
Let G be the Adjacency Matrix of the graph whose eigenvalues satisfy Eqn. (15). Given this, recall
that the Laplacian eigenvalues satisfy λi = d − αi , and so all of the non-zero eigenvalues of LG are
in the interval between (1 − ǫ) d and (1 + ǫ) d. I.e., for all x s.t. x ⊥ ~1, we have that
(1 − ǫ) xT x ≤ xT LG x ≤ (1 + ǫ) xT x.
(This follows from Courant-Fischer or by expanding x is an eigenvalue basis.)
On the other hand, for the complete graph Kn , we know that all vectors x that are ⊥ ~1 satisfy
xT LKn x = nxT x.
So, let H be the graph
H=

d
Kn ,
n

from which it follows that
xT LH x = dxT x.
Thus, the graph G is an ǫ-approximation of the graph H, from which the theorem follows.
⋄
For completeness, consider G − H and let’s look at its norm to see that it is small. First note that
(1 − ǫ) H  G  (1 + ǫ) H implies that − ǫH  G − H  ǫH.
Since G and H are symmetric, and all of the eigenvalues of ǫH are either 0 or d, this tells us that
kLG − LH k2 ≤ ǫd.

Lecture Notes on Spectral Graph Methods

8.8

93

Expanders are graphs on which diffusions and random walks mix rapidly

We will have more to say about different types of diffusions and random walks later, so for now we
will only work with one variant and establish one simple variant of the idea that random walks on
expander graphs mix or equilibrate quickly to their equilibrium distribution.
Let G = (V, E, W ) be a weighted graph, and we want to understand something about how random
walks behave on G. One might expect that if, e.g., the graph was a dumbbell graph, then random
walks that started in the one half would take a very long time to reach the other half; on the
other hand, one might hope that if there are no such bottlenecks, e.g., bottlenecks revealed by the
expansion of second eigenvalue, than random walks would mix relatively quickly.
To see this, let pt ∈ Rn , where n is the number of nodes in the graph, be a probability distribution
at time t. This is just some probability distribution over the nodes, e.g., it could be a discrete
Dirac δ-function, i.e., the indicator of a node, at time t = 0; it could be the uniform distribution;
or it could be something else. Given this distribution at time t, the transition rule that governs the
distribution at time t + 1 is:
• To go to pt+1 , move to a neighbor with probability ∼ the weight of the edge. (In the case of
unweighted graphs, this means that move to each neighbor with equal probability.) That is,
to get to pt+1 from pt , sum over neighbors
pt+1 (u) =

X

v:(u,v)∈E

where d(v) =

P

u W (u, v)

W (u, v)
pt (v)
d(v)

is the weighted degree of v.

As a technical point, there are going to be bottlenecks, and so we will often consider a “lazy”
random walk, which removed that trivial bottleneck that the graph is bipartite thus not mixing
(i.e. the stationary distribution doesn’t exist) and only increases the mixing time by a factor of
two (intuitively, on expectation in two steps in the “lazy” walk we walk one step as in the simple
random walk)—which doesn’t matter in theory, since there we are interested in polynomial versus
exponential times, and in practice the issues might be easy to diagnose or can be dealt with in less
aggressive ways. Plus it’s nicer in theory, since then things are SPSD.
By making a random walk “lazy,” we mean the following: Let
pt+1 (u) =

1
1
pt (u) +
2
2

X

v:(u,v)∈E

W (u, v)
pt (v).
d(v)

−1
That is, pt+1 = 12 I+ AD
is replaced with
pt , and so the transition matrix WG = AG DG
−1
1
WG = 2 I + AG DG —this is an asymmetric matrix that is similar in some sense to the normalized
Laplacian.


−1

Then, after t steps, we are basically considering WGt , in the sense that
p0 → pt = W pt−1 = W 2 pt−2 = · · · = W t pt .
Fact. Regardless of the initial distribution, the lazy random walk converges to π(i) =
which is the right eigenvector of W with eigenvalue 1.

Pd(i) ,
j d(j)

94

M. W. Mahoney

Fact. If 1 = ω1 ≥ ω2 ≥ · · · ωn ≥ 0 are eigenvalues of W , with π(i) =
rate of convergence to the stationary distribution.

Pd(i) ,
j d(j)

then ω2 governs the

There are a number of ways to formalize this “rate of mixing” result, depending on the norm used
and other things. In particular, a very good way is with the total variation distance, which is
defined as:
)
(
X
X
1
kp − qkT V D = max
pv −
qv = kp − qk1 .
S⊆V
2
v∈S

v∈S

(There are other measures if you are interested in mixing rates of Markov chains.) But the basic
point is that if 1 − ω2 is large, i.e., you are an expander, then a random walk converges fast. For
example:
Theorem 17. Assume G = (V, E) with |V | = n is d-regular, A is the adjacency matrix of G, and
Â = d1 A is the transition matrix of a random walk on G, i.e., the normalized Adjacency Matrix.
Also, assume λ = max(|µ2 |, |µn |) = αd (recall µi is the i-th largest eigenvalue of A, not Â). Then
||Ât p − u||1 ≤

√

nαt ,

where u is the stationary distribution of the random walk, which is the uniform distribution in the
undirected d-regular graph, and p is an arbitrary initial distribution on V .

c
log nǫ , for some absolute constant c independent of n, then ku− Ât pk ≤ ǫ.
In particular, if t ≥ 1−α
Proof. Let us define the matrix Jˆ = n1 ~1~1⊤ , where, as before, ~1 is the all ones vector of length n.
Note that, for any probability vector p, we have
ˆ = 1 ~1~1⊤ p
Jp
n
1
= ~1 · 1
n
= u.
Now, since Â = d1 A we have µ̂i = µi /d, where µ̂i denotes the ith largest eigenvalue of Â, and the
eigenvectors of Â are equal to those of A. Hence, we have
Ât − Jˆ

2

ˆ
max k(Ât − J)wk
2


= σmax Ât − Jˆ
=

w:kwk2 ≤1

= σmax
(a)

= σmax

=

n
X

1
µ̂ti vi vi⊤ − ~1~1⊤
n
i=1
!
n
X
µ̂ti vi vi⊤

i=2
t
max{|µ̂2 |, |µ̂tn |}
t

=α,

!

Lecture Notes on Spectral Graph Methods
where (a) follows since v1 =

√1 ~
1
n

95

and µ̂1 = 1. Then,
Ât p − u

1

√
n Ât p − u 2
√
ˆ
≤ n Ât p − Jp
2
√
≤ n Ât − Jˆ 2 p
√
≤ nαt ,

≤

2

which concludes the proof.
This theorem shows that if the spectral gap is large (i.e. α is small), then we the walk mixes rapidly.
This is one example of a large body of work on rapidly mixing Markov chains. For example, there
are extensions of this to degree-heterogeneous graphs and all sorts of other thigns Later, we might
revisit this a little, when we see how tight this is; in particular, one issue that arises when we discuss
local and locally-biased spectral methods is that how quickly a random walk mixes depends on not
only the second eigenvalue but also on the size of the set achieving that minimum conductance
value.

96

M. W. Mahoney

9

(02/19/2015): Expanders, in theory and in practice (2 of 2)

Reading for today.
• Same as last class.
Here, we will describe how expanders are the metric spaces that are least like low-dimensional
Euclidean spaces (or, for that matter, any-dimensional Euclidean spaces). Someone asked at the
end of the previous class about what would an expander “look like” if we were to draw it. The
point of these characterizations of expanders—that they don’t have good partitions, that they
embed poorly in low dimensional spaces, etc.—is that you can’t draw them to see what they look
like, or at least you can’t draw them in any particularly meaningful way. The reason is that if
you could draw them on the board or a two-dimensional piece of paper, then you would have an
embedding into two dimensions. Relatedly, you would have partitioned the expander into two parts,
i.e., those nodes on the left half of the page, and those nodes on the right half of the page. Any such
picture would have roughly as many edges crossing between the two halves as it had on either half,
meaning that it would be a non-interpretable mess. This is the reason that we are going through
this seemingly-circuitous characterizations of the properties of expanders—they are important, but
since they can’t be visualized, we can only characterize their properties and gain intuition about
their behavior via these indirect means.

9.1

Introduction to Metric Space Perspective on Expanders

To understand expanders from a metric space perspective, and in particular to understand how
they are the metric spaces that are least like low-dimensional Euclidean spaces, let’s back up a bit
to the seemingly-exotic subject of metric spaces (although in retrospect it will not seem so exotic
or be so surprising that it is relevant).
• Finite-dimensional Euclidean space, i.e., Rn , with n < ∞, is an example of a metric space
that is very nice but that is also quite nice/structured or limited.
• When you go to infinite-dimensional Hilbert spaces, things get much more complex; but ∞dimensional RKHS, as used in ML, are ∞-dimensional Hilbert spaces that are sufficiently
regularized that they inherit most of the nice properties of Rn .
• If we measure distances in Rn w.r.t. other norms, e.g., ℓ1 or ℓ∞ , then we step outside the
domain of Hilbert spaces to the domain of Banach spaces or normed vector spaces.
• A graph G = (V, E) is completely characterized by its shortest path or geodesic metric; so the
metric space is the nodes, with the distance being the geodesic distance between the nodes.
Of course, you can modify this metric by adding nonnegative weights to edges like with some
nonlinear dimensionality reduction methods. Also, you can assign a vector to vertices and
thus view a graph geometrically. (We will get back to the question of whether there are other
distances that one can associate with a graph, e.g., resistance of diffusion based distances;
and we will ask what is the relationship between this and geodesic distance.)
• The data may not be obviously a matrix or a graph. Maybe you just have similarity/dissimilarity
information, e.g., between DNA sequences, protein sequences, or microarray expression levels.
Of course, you might want to relate these things to matrices or graphs in some way, as with
RKHS, but let’s deal with metrics first.

Lecture Notes on Spectral Graph Methods

97

So, let’s talk aobut metric spaces more generally. The goal will be to understand how good/bad
things can be when we consider metric information about the data.
So, we start with a definition:
Definition 32. (X, d) is a metric space if
• d : X × X → R+ (nonnegativity)
• d(x, y) = 0 iff x = y
• d(x, y) = d(y, x) (symmetric)
• d(x, y) ≤ d(x, z) + d(z, y) (triangle inequality)
The idea is that there is a function over the set X that takes as input pairs of variables that
satisfies a generalization of what our intuition from Euclidean distances is: namely, nonnegativity,
the second condition above, symmetry, and the triangle inequality. Importantly, this metric does
not need to come from a dot product, and so although the intuition about distances from Euclidean
spaces is the motivation, it is significantly different and more general. Also, we should note that if
various conditions are satisfied, then various metric-like things are obtained:
• If the second condition above is relaxed, but the other conditions are satisfied, then we have
a psuedometric.
• If symmetry is relaxed, but the other conditions are satisfied, then we have a quasimetric.
• If the triangle inequality is relaxed , but the other conditions are satisfied, then we have a
semimetric.
We should note that those names are not completely standard, and to confuse matters further
sometimes the relaxed quantities are called metrics—for example, we will encounter the so-called
cut metric describing distances with respect to cuts in a graph, which is not really a metric since
the second condition above is not satisfied.
More generally, the distances can be from a Gram matrix, a kernel, or even allowing algorithms in
an infinite-dimensional space.
Some of these metrics can be a little counterintuitive, and so for a range of reasons it is useful
to ask how similar or different two metrics are, e.g., can we think of a metric as a tweak of a
low-dimensional space, in which case we might hope that some of our previous machinery might
apply. So, we have the following question:
Question 1. How well can a given metric space (X, d) be approximated
by ℓ2 , where ℓ2 is the
P
metric space (Rn , || · ||), where ∀x, y ∈ Rn , we have ||x − y||2 = ni=1 (xi − yi )2 .

The idea here is that we want to replace the metric d with something d′ that is “nicer,” while still
preserving distances—in that case, since a lot of algorithms use only distances, we can work with
d′ in the nicer place, and get results that are algorithmically and/or statistically better without
introducing too much error. That is, maybe it’s faster without too much loss, as we formulated it
before; or maybe it is better, in that the nicer place introduced some sort of smoothing. Of course,

98

M. W. Mahoney

we could ask this about metrics other than ℓ2 ; we just start with that since we have been talking
about it.
There are a number of ways to compare metric spaces. Here we will start by defining a measure of
distortion between two metrics.
Definition 33. Given a metric space (X, d) and our old friend the metric space (Rn , ℓ2 ), and a
mapping f: X → Rn :
1 )−f (x2 )||2
• expansion(f ) = maxx1 ,x2 ∈X ||f (xd(x
1 ,x2 )
1 ,x2 )
• contraction(f ) = max ||f (xd(x
1 )−f (x2 )||

• distortion(f ) = expansion(f ) · contraction(f )
As usual, there are several things we can note:
• An embedding with distortion 1 is an isometry. This is very limiting for most applications
of interset, which is OK since it is also unnecessarily strong notion of similarity for most
applications of interest, so we will instead look for low-distortion embeddings.
• There is also interest in embedding into ℓ1 , which we will return to below when talking about
graph partitioning.
• There is also interest in embedding in other “nice” places, like trees, but we will not be talking
about that in this class.
• As a side comment, a Theorem of Dvoretzky says that any embedding into normed spaces,
ℓ2 is the hardest. So, aside from being something we have already seen, this partially justifies
the use of ℓ2 and the central role of ℓ2 in embedding theory more generally.
Here, we should note that we have already seen one example (actually, several related examples)
of a low-distortion embedding. Here we will phrase the JL lemma that we saw before in our new
nomenclature.
Theorem 18 (JL Lemma). Let X be an n-point set in Euclidean
i.e., X ⊂ ℓn2 , and fix
 space,

n
ǫ ∈ (0, 1]. Then ∃ a (1 + ǫ)-embedding of X into ℓk2 , where k = O log
.
ǫ2
That is, Johnson-Lindenstrauss says that we can map xi → f (x) such that distance is within 1 ± ǫ
of the original.
A word of notation and some technical comments: For x ∈ Rd and p ∈ [1, ∞), the ℓp norm of
P
1/p
d
p
x is defined as ||x||p =
|x
|
. Let ℓdp denote the space Rd equipped with the ℓp norm.
i
i=1

Sometimes we are interested in embeddings into some space ℓdp , with p given but the dimension d
unrestricted, e.g., in some Euclidean space s.t. X embeds well. Talk about: ℓp = the space of all
P
p 1/p . In this case,
sequences (x1 , x2 , . . .), with ||x||p < ∞, with ||x||p defined as ||x||p = ( ∞
i=1 |xi | )
embedding into ℓp is shorthand for embedding into ℓdp for some d.
Here is an important theorem related to this and that we will return to later.

Lecture Notes on Spectral Graph Methods

99

Theorem 19 (Bourgain). Every n-point metric space (X, d) can be embedded into Euclidean space
ℓ2 with distortion ≤ O(log n).
Proof Idea. (The proof idea is nifty and used in other contexts, but we won’t use it much later,
except to point out how flow-based methods do something similar.) The basic idea is given (X, d),
map each point x → φ(x) in O(log2 n)-dimensional space with coordinates equal to the distance
to S ⊆ X where S is chosen randomly. That is, given (X, d), map every point x ∈ X to φ(x), an
O(log2 n)-dimensional vector, where coordinates in φ(·) correspond to subsets S ⊆ X, and s.t. the
s-th in φ(x) is d(x, S) = mins∈S d(x, s). Then, to define the map, specify a collection of subsets we
use selected carefully but randomly—select O(log n) subsets of size 1, O(log n) subsets of size 2, of
size 4, 8, . . ., n2 . Using that, it works, i.e., that is the embedding.
Note that the dimension of the Euclidean space was originally O(log2 n), but it has been improved
to O(log n), which I think is tight. Note also that the proof is algorithmic in that it gives an efficient
randomized algorithm.
Several questions arise:
• Q: Is this bound tight? A: YES, on expanders.
• Q: Let c2 (X, d) be the distortion of the embedding of X into ℓ2 ; can we compute c2 (X, d) for
a given metric? A: YES, with an SDP.
• Q: Are there metrics such that c2 (X, d) ≪ log n? A: YES, we saw it with JL, i.e., highdimensional Euclidean spaces, which might be trivial since we allow the dimension to float in
the embedding, but there are others we won’t get to.

9.1.1

Primal

The problem of whether a given metric space is γ-embeddable into ℓ2 is polynomial time solvable.
Note: this does not specify the dimension, just whether there is some dimension; asking the same
question with dimension constraints or a fixed dimension is in general much harder. Here, the
condition that the distortion ≤ γ can be expressed as a system of linear inequalities in Gram
matrix correspond to vectors φ(x). So, the computation of c2 (x) is an SDP—which is easy or hard,
depending on how you view SDPs—actually, given an input metric space (X, d) and an ǫ > 0, we
can determine c2 (X, d) to relative error ≤ ǫ in poly(n, 1/ǫ) time.
Here is a basic theorem in the area:
Theorem 20 (LLR). ∃ a poly-time algorithm that, given as input a metric space (X, d), computes
c2 (X, d), where c2 (X, d) is the least possible distortion of any embedding of (X, d) into (Rn , ℓ2 ).
Proof. The proof is from HLW, and it is based on semidefinite programming. Let (X, d) be the
metric space, let |X| = n, and let f : X → ℓ2 . WLOG, scale f s.t. contraction(f ) = 1. Then,
distortion(f ) ≤ γ iff
d(xi , xj )2 ≤ ||f (xi ) − f (xj )||2 ≤ γ 2 d(xi , xj )2 .
(16)

100

M. W. Mahoney

Then, let ui = f (xi ) be the i-th row of the embedding matrix U , and let Z = U U T . Note that
Z ∈ P SD, and conversely, if Z ∈ P SD, then Z = U U T , for some matrix U . Note also:
||f (xi ) − f (xj )||2 = ||ui − uj ||2

= (ui − uj )T (ui − uj )

= uTi ui + uTj uj − 2uTi uj
= Zii + Zjj − 2Zij .

So, instead of finding a ui = f (xi ) s.t. (16) holds, we can find a Z ∈ P SD s.t.
d(xi , xj )2 ≤ Zii + Zjj − 2Zij ≤ γ 2 d(xi , xj )2 .

(17)

Thus, c2 ≤ γ iff ∃Z ∈ SP SD s.t. (17) holds ∀ij. So, this is an optimization problem, and we can
solve this with simplex, interior point, ellipsoid, or whatever; and all the usual issues apply.
9.1.2

Dual

The above is a Primal version of the optimization problem. If we look at the corresponding Dual
problem, then this gives a characterization of c2 (X, d) that is useful in proving lower bounds. (This
idea will also come up later in graph partitioning, and elsewhere.) To go from the Primal to the
Dual, we must take a nonnegative linear combination of constraints. So we must write Z ∈ P SD
in such a way, since that is the constraint causing a problem; the following lemma will do that.
P
Lemma 14. Z ∈ P SD iff ij qij zij ≥ 0, ∀Q ∈ P SD.

Proof. First, we will consider rank 1 matrices; the general result will follow since general PSD
matrices are a linear combination of rank-1 PSD matrices of the form qq T , i.e., Q = qq T .
First, start with the ⇐ direction: for q ∈ Rn , let Q be P SD matrix s.t. Qij = qi qj ; then
X
X
q t Zq =
qi Zij qj =
Qij zij ≥ 0,
ij

ij

where the inequality follows since Q is P SD. Thus, Z ∈ P SD.
For the ⇒ direction: let Q be rank-1 PSD matrix; thus, it has the form Q = qq T or Qij = qi qj , for
q ∈ Rn . Then,
X
X
Qij zij =
qi Zij qj ≥ 0,
ij

ij

where the inequality follows since A is P SD.
P
P
Thus, since Q ∈ P SD implies that Q = i qi qiT = i Ωi , with Ωi being a rank-i PSD matrix, the
lemma follows by working through things.
Now that we have this characterization of Z ∈ P SD in terms of a set of (nonnegative) linear
combination of constraints, we are ready to get out Dual problem which will give us the nice
characterization of c2 (X, d).
P
Recall finding an embedding f (xi ) = ui iff finding a matrix Z iff ij qij zij ≥ 0, ∀Q ∈ SP SD. So,
the Primal constraints are:

Lecture Notes on Spectral Graph Methods
I.

P

101

qij zij ≥ 0 for all QǫP SD

II. zii + zjj − 2zij ≥ d(xi , xj )2
III. γ 2 d(xi , xj )2 ≥ zii + zjj − 2zij ,
which hold ∀ij. Thus, we can get the following theorem.
Theorem 21 (LLR).
v P
u
u
P d(xi , xj )2
t PPij >0 ij
C2 (X, d) =
max
− (Pij <0) Pij d(xi , xj )2
(P ǫP SD,P.1=0)
Proof. The dual program is the statement that for γ < C2 (X, d), thre must exist a non-negative
combination of the constraints of the primal problem that yields a contradiction.
So, we will assume γ < C2 (x, d) and look for a contradiction, i.e., look for a linear combination of
constraints such that the primal gives a contradiction. Thus, the goal is to construct
a nonnegative
P
linear combination of primal constraints to give a contradiction s.t. Q.Z = qij zij ≥ 0.
Recall that the cone of PSD matrices is convex.

The goal is to zero out the zij .
P
(I) above says that Q ·Z =P ij qij zij ≥ 0. Note that since PSD cone is convex, a nonnegative linear
combination of the form k αk hQ, Zi = P · z, for some P ∈ P SD. So, modifying first constraint
from the primal, you get
I’.

P

ij Pij Zij = P · Z ≥ 0, for some P ǫP SD
To construct P , choose the elements such that you zero out zij in the following manner.

• If Pij > 0, multiply second constraint from primal by Pij /2, (i.e., the constraint d(xi , xj )2 ≤
zii + zjj − 2zij )
• If Pij < 0, multiply third constraint from primal by −Pij /2, (i.e., the constraint zii +
zjj − 2zij ≤ γ 2 d(xi , xj )2 )

• If Pij = 0, multiply by 0 constraints involving zij .
This gives

Pij
Pij
(zii + zjj − 2zij ) ≥
d(xi , xj )2
2
2
Pij 2
Pij
−
γ d(xi , xj )2 ≥ −
(zii + zjj − 2zij )
2
2
from which it follows that you can modify the other constraints from primal to be:
II’.
III’.

P

P

Pij
ij,Pij >0 2 (zii

+ zjj − 2zij ) ≥

Pij
ij,Pij <0 2 (zii

+ zjj − 2zij ) ≥

P
P

Pij
2
ij,Pij >0 2 d(xi , xj )
Pij 2
2
ij,Pij <0 2 γ (d(xi , xj ) )

102

M. W. Mahoney

If we add those two things, then we get,
X
X Pij
X Pij
Pii zii +
(zii + zjj ) +
(zii + zjj ) ≥ RHS Sum,
2
2
i

ij

ij:Pij >0

and so
X

X

Pii zii +

i

ij:Pij 6=0

Pij
(zii + zjj ) ≥ RHS Sum,
2

P
P
and so, since we choose P s.t. P · ~1 = ~0, (i.e.
j Pij = 0 for all i, and
i Pij = 0 for all j by
symmetry) we have that


X
X
X Pij
X Pij
Pii +
0=
Pij  zij ≥ RHS =
d(xi , xj )2 +
γ 2 d(xi , xj )2
2
2
i

j:Pij 6=0

ij:Pij >0

ij:Pij <0

So, it follows that

0≥

X

Pij d(xi , xj )2 +

ij:Pij >0

γ 2 d(xi , xj )2 .

ij:Pij <0

This last observation is FALSE if

P

and so the theorem follows.

X

γ2 < P

2
ij:Pij >0 Pij d(xi , xj )

2
ij:Pij <0 (−Pij )d(xi , xj )

(In brief, adding the second and third constraints above,
0≥

P

2
ij,Pij >0 Pij d(xi , xj )

+

P

ij,Pij <0 Pij γ

2 (d(x

2
i , xj ) )

This will be false if you choose γ to be small—in particular, it will be false if γ 2 ≤ top/bottom,
from which the theorem will follow.

9.2

Metric Embedding into ℓ2

We will show that expanders embed poorly in ℓ2 —this is the basis for the claim that they are the
metric spaces that are least like low-dimensional spaces in a very strong sense.
It is easy to see that an expander can be embedded into ℓ2 with distortion O(log n) (just note that
any graph can be embedded with distortion equal to its diameter)—in fact, any metric space can
be embedded with that distortion. We will show that this result is tight, and thus that expanders
are the worst.
The basic idea for showing that expanders embed poorly in ℓ2 is: If G is a d-regular, ǫ-expander,
then λ2 of AG is < d − δ for δ = δ(d, ǫ) 6= δ(n). The vertices of a bounded degree graph can be
paired up s.t. every pair of vertices are a distance Ω(log n). We can then let B be a permutation
matrix for the pairing, and use the matrix P = dI − A + 2δ (B − I).
Note: We can have a simpler proof, using the theorem of Bourgain that expanders don’t embed
well in ℓ2 , since we can embed in Rdiameter , and diameter(expander) = log n. But we go through
this here to avoid (too much) magic.
Start with the following definitions:

Lecture Notes on Spectral Graph Methods

103

Definition 34. A Hamilton cycle in a graph G = (V, E) is a cycle that visits every vertex exactly
once (except for the start and end vertices).
Definition 35. A matching is a set of pairwise non-adjacent edges, i.e., no two edges share a
common vertex. A vertex is matched if it is incident to an edge. A perfect matching is a matching
that matched all the vertices of a graph.
The following theorem is the only piece of magic we will use here:
Theorem 22. A simple graph with n ≥ 3 edges is Hamiltonian if every vertex has degree ≥ n2 .
Note that if every vertex has degree ≥ n2 , then the graph is actually quite dense, and so from
Szemeredi-type results relating dense graphs to random graphs it might not be so surprising that
there is a lot of wiggle room.
Note: A cycle immediately gives a matching.
Thus, we have the following lemma:
Lemma 15. Let G = (V, E) be a d-regular graph, with |V | = n If H = (V, E ′ ) is a graph with the
same vertex set as G, in which two vertices u and v are adjacent iff dG (u, v) ≥ ⌊logk n⌋. Then, H
has a matching with n/2 edges.
Proof. Since G is a d-regular graph, hence for any vertex x ∈ V and any value r, it has ≤ kr
vertices y ∈ V can have dG (x, y) ≤ r, i.e., only that many vertices are within a distance r. If
r = ⌊logk n⌋ − 1, then ∃ ≤ n2 vertices within distance r; that is, at least half of the nodes of G are
further than logk n − 2 from x; this means every node in H has at least degree n/2. So H has a
Hamilton cycle and thus a perfect matching, and by the above theorem the lemma follows.
Finally we get to the main theorem that says that expanders embed poorly in ℓ2 —note that this
is a particularly strong statement or notion of nonembedding, as by Bourgain we know any graph
(with the graph distance metric) can be embedded into ℓ2 with distortion O(log n), so expander is
the worst case in this sense.
Theorem 23. Let d ≥ 3, and let ǫ > 0. If G = (V, E) is a (n, d)-regular graph with λ2 (AG ) ≤ d − ǫ
and |V | = n, then
C2 (G) = Ω(log n)
where the constant inside the Ω depends on d, ǫ.
Proof. To prove the lower bound, we use the characterization from the last section that for the
minimum distortion in embedding a metric space (X, d) into l2 , denoted by C2 (X, d), is:
v P
u
u
p d(xi , xj )2
t Ppij >0 ij
(18)
C2 (X, d) =
max−
→ −
→
− pij <0 pij d(xi , xj )2
P ∈P SD,P. 1 = 0
and we will find some P that is feasible that gives the LB.

Assume B is the adjacency matrix of the matching in H, whose existence is proved in the previous
lemma. Then, define
ǫ
P = (dI − AG ) + (B − I).
2

104

M. W. Mahoney

→ −
−
→
Then, we claim that P 1 = 0 . To see this, notice both (dI − AG ) and I − B are Laplacians (not
normalized), as B is the adjacency matrix of a perfect matching (i.e. 1-regular graph). Next, we
→
−
claim that P ∈ PSD. This proof of this second claim is because, for any x ⊥ 1 , we have
xT (dI − AG )x ≥ dxT x − xT Ax ≥ (d − λ2 )||x||2 ≥ ǫ||x||2
(by the assumption on λ2 ); and
X
X
xT (B − I)x =
2xi xj −
x2i
i

(i,j)∈B

X

=

(i,j)∈B

≥ −2

(2xi xj − x2i − x2j )

X

since ||x||2 =

x2i + x2j

P

(i,j)∈B

x2i + x2j

(i,j)∈B

= −2||x||2
The last line is since ||x||2 =
sum exactly once.

P

(i,j)∈B

x2i + x2j and since B is a matching so each i shows up in the

So, we have that
ǫ
xT P x = xT (dI − AG )x + xT (B − I)x
2
2||x||2 ǫ
≥ ǫ||x||2 −
2
= 0.
Next evaluate the numerator and the denominator.
X
−
d(i, j)2 Pij = dn
Pij <0

X

Pij >0

d(i, j)2 Pij ≥

ǫ
n⌊logd n⌋2
2

where the latter follows since the distances of edges in B are at least ⌊logd n⌋. Thus, for this P , we
have that:
v P
r
u
ǫ
2
u
p d(xi , xj )2
2 n⌊logk n⌋
t Ppij >0 ij
≥
∼ Θ(log n)
− pij <0 pij d(xi , xj )2
dn

and thus, from (18), that C2 is at least this big, i.e., that:
C2 (G) ≥ Ω(log n)

References
[1] N. Linial and A. Wigderson, ”Expander graphs and their applications”

Lecture Notes on Spectral Graph Methods

10

105

(02/24/2015): Flow-based Methods for Partitioning Graphs (1
of 2)

Reading for today.
• “Multicommodity max-flow min-cut theorems and their use in designing approximation algorithms,” in JACM, by Leighton and Rao
• “Efficient Maximum Flow Algorithms,” in CACM, by Goldberg and Tarjan

10.1

Introduction to flow-based methods

Last time, we described the properties of expander graphs and showed that they have several
“extremal” properties. Before that, we described a vanilla spectral partitioning algorithms, which
led to the statement and proof of Cheeger’s Inequality. Recall that one direction viewed λ2 as a
relaxation of the conductance or expansion problem; while the other direction gave a “quadratic”
bound as well as a constructive proof of a graph partitioning algorithm. The basic idea was to
compute a vector, show that it is a relaxation of the original problem, and show that one doesn’t
loose too much in the process. Later, we will see that there are nice connections between these
methods and low-dimensional spaces and hypothesized manifolds.
Lest one think that this is the only way to compute partitions, we turn now to a very different
method to partition graphs—it is based on the ideas of single-commodity and multi-commodity
flows. It is not a spectral method, but it is important to know about for spectral methods (e.g.,
when/why spectral methods work and how to diagnose things when they don’t work as one expects):
the reason is basically that flow-based graph algorithms “succeed” and “fail” in very different ways
than spectral methods; and the reason for this is that they too implicitly involve embedding the
input graph in a metric/geometric place, but one which is very different than the line/clique that
spectral methods implicitly embed the input into.

10.2

Some high-level comments on spectral versus flow

Recall that the key idea underlying graph partitioning algorithms is take a graph G = (V, E) and
spread out the vertices V in some abstract space while not spreading out edges E too much, and
then to partition the vertices in that space into tow (or more) sets.
• Spectral methods do this by putting nodes on an eigenvector and then partitioning based
on a sweep cut over the partitions defined by that eigenvector. For spectral methods, several
summary points are worth noting.
– They achieve “quadratic” worst-case guarantees from Cheeger’s Inequality. This quadratic
factor is “real,” e.g., it is not an artifact of the analysis and there are graphs on which
it is achieved.
– They are “good” when graph has high conductance or expansion, i.e., when it is a good
expander (in either the degree-weighted or degree-unweighted sense).
– They are associated with some sort of underlying geometry, e.g., as defined by where the
nodes get mapped to in the leading eigenvector; but it is not really a metric embedding

106

M. W. Mahoney
(or at least not a “good” metric embedding since it only preserves average distances—
which is typical of these ℓ2 -based methods—and not the distance between every pair of
nodes.
• Flow-based methods do this by using multi-commodity flows to reveal bottlenecks in the
graph and then partitioning based on those bottlenecks. To contrast with spectral methods,
note the following summary points for flow-based methods.
– They achieve O(log n) worst-case guarantees. This O(log n) is “real,” in that it too is
not an artifact of the analysis and there are graphs on which it is achieved.
– They are “good” when the graph has sparse cuts, i.e., when it is not a good expander.
– They are also associated with a geometry, but one which is very different than that of
spectral methods. In particular, although not immediately-obvious, they can be viewed
as embedding a finite metric space of graph into ℓ1 metric, i.e., a very different metric
than before, and then partitioning there.

The point here is that these two methods are in many ways complementary, in the sense that they
succeed and fail in different places. Relatedly, while “real” data might not be exactly one of the
idealized places where spectral or flow-based methods succeed or fail, a lot of graph-based data
have some sort of low-dimensional structure, and a lot of graph-based data are sufficiently noisy
that it is fruitful to view them as having expander-like properties.
Finally, the comments about spectral methods being good on expanders and flow methods being
good on non-expanders might seem strange. After all, most people who use spectral methods are
not particularly interested in partitioning graphs that do not have good partitions (and instead
they take advantage of results that show that spectral methods find good partitions in graphs that
are “morally” low-dimensional, e.g., graphs like bounded-degree planar graphs and well-shaped
meshes). The reason for this comment is that the metric we have been using to evaluate cluster
quality is the objective function and not the quality of the clusters. By this measure, the quadratic
of a constant (which is the expansion value of an expander) is a constant, while the quadratic
√
of, e.g., 1/ n is very large, i.e., the spectral guarantee is nontrivial in one case and not in the
other. For flow-based methods by contrast, the O(log n) is much larger than the constant value
√
of an expander and so gives trivial results, while this is much smaller than, e.g., 1/ n, leading
to nontrivial results on the objective for morally low-dimensional graphs. That these qualitative
guides are the opposite of what is commonly done in practice is a real challenge for theory, and it
is a topic we will return to later.

10.3

Flow-based graph partitioning

Recall the basic ideas about flow and the single commodity flow problem:
• G = (V, E) is a graph, and each edge e ∈ E has capacity C(e), which is the maximum amount
of flow allowed through it. Also, we are given a source s and a sink t.
• (We are going to be applying it to graph partitioning by trying to use network flow ideas to
reveal bottlenecks in the graph, and then cut there.)
• So, the goal is to route as much flow s → t without violating any of the constraints.

Lecture Notes on Spectral Graph Methods

107

• Max-flow is the maximum amount of such flow.
• Min-Cut = the minimum amount of capacity to be removed from a network to disconnect s
from t; that is,
X
M in − Cut =
min
c(e).
U ⊂V :s∈U,t∈Ū

e∈(U,Ū )

(Note that there is no ratio here, since we can assume that demand = 1, WLOG, but that
we won’t be able to do this later.)
• Weak duality is one thing and is relatively easy to show:
Claim 3. maxf low ≤ mincut
Proof. For all U ⊆ V that has s and t on opposite sides, all flows from s → t must be routed
through edges in (U, Ū ), so the total flow is bounded by the capacity in mincut.
• Strong duality is another thing and is harder:
Claim 4. maxf low = mincut
Proof. Suffices to show that mincut bound is always achievable is harder. We won’t do it
here.
All of this discussion so far is for k = 1 single-commodity flow.
There are also multi-commodity versions of this flow/cut problem that have been widely studied.
Here is the basic definition.
Definition 36. Given k ≥ 1, each with a source si and sink ti and also given demands Di for each,
i.e., we have (si , ti , Di ) for each i ∈ [k], the multi-commodity flow problem is to simultaneously
route Di units of flow from si to ti , for all i, while respecting capacity constraints, i.e., s.t. the
total amount of all commodities passing through any edge ≤ capacity of that edge. There are several
variants, including:
• Max throughput flow: maximize the amount of flow summed over all commodities.
• Max concurrent flow: specify a demand Di for each commodity, and then maximize the fraction of demand Di that can be simultaneously shipped or routed by flow:
max f s.t. f Di units of flow go from si to ti .
i.e., the maximum f s.t. f Di units of capacity i are simultaneously routed without violating
the capacity constraints.
That is, for this multicommodity flow problem, if the flow of commodity i along edge (u, v) is
fi (u, v), then an assignment of flow satisfies:
• Capacity constraints.

k
X
i=1

|fi (u, v)| ≤ c(u, v)

108

M. W. Mahoney
• Flow conservation
X

w∈V

fi (u, w) = 0 when u 6= si , ti

For all u, v
• Demand satisfaction.

X

fi (u, v) = −fi (v, u)

fi (si , w) =

w∈V

X

fi (w, t) = Di

w∈V

Then the goal is to find a flow that maximizes one of the above variants of the problem.
We can also define a related cut problem.
Definition 37. The MinCut or Sparsest Cut Ξ—of an undirected multicommodity flow problem—
is the mincut over all cuts of the ratio of the capacity of cut to the demand of the cut, i.e.,
Ξ = min cut = ρ = Ξ = min
U ⊆V

C(U, Ū )
,
D(U, Ū )

where
C(U, Ū ) =

X

C(e)

e∈(U,Ū )

D(U, Ū ) =

X

Di

i:si ∈U
ti ∈Ū or vice versa

That is, C(U, Ū ) is the sum of capacities linking U to Ū ; and D(U, Ū ) is the sum of demands with
source and sinks on opposite sides of the (U, Ū ) cut.
Finally, we point out the following two variants (since they will map to the expansion and conductance objectives when we consider the application of multi-commodity flow to graph partitioning).
There are, of course, other variants of flow problems that we don’t consider that are of interest if
one is primarily interested in flow problems.
Definition 38. The Uniform Multi-Commodity Flow Problem (UMFP): here, there is a demand
for every pair of nodes, and the demand for every commodity is the same, i.e., they are uniform,
WLOG we take to equal 1. The Product Multicommodity Flow Problem (PMFP): here, there is a
nonnegative weight π(·), for all nodes u ∈ V , i.e., π : V → R+ . The weights of demands between a
pair of nodes u and v are π(vi )π(vj ).
Here are several comments about the last definition.
• UMFP is a special case of PMFP with π(i) = 1.
• If π(i) = 1, then we will get a way to approximate the expansion objective.
• If π(v) = deg(v), then we will get a way to approximate the conductance objective.

Lecture Notes on Spectral Graph Methods

109

The MaxFlow-MinCut for single commodity is nice since it relates two fundamental graph theoretic
entities (that are in some sense dual) via the min-max theorem. But, prior to LR, very little was
known about the relationship between MaxFlow and MinCut in this more general multi-commodity
flow setting for general graphs. For certain special graphs, it was known that there was an equality,
i.e., a zero duality gap; but for general graphs, it is only known that MaxFlow is within a fraction k
of the MinCut. This results can be obtained since each commodity can be optimized separately in
the obvious trivial way by using k1 of the capacity of the edges—this might be ok for k = Θ(1), but
it is is clearly bad if k ∼ n or k ∼ n2 . Somewhat more technically, if we consider the LP formulation
of the Max Multicommodity Flow Problem, then we can make the following observations.
• The dual of this is the LP relaxation of the Min Multicut Problem, i.e., the optimal integral
solution to the dual is the Mun Multicut.
• In general, the vertices of the dual polytope are NOT integral.
• But, for single commodity flow, they are integral, and so MaxFlow-MinCut theorem is a
consequence of LP duality.
• For certain special graphs, it can be shown that they are integral, in which case one has zero
duality gap for those graphs.
• Thus, for multicommodity: MaxFlow = MinFractional, i.e., relaxed, Multicut.
Here are several facts that we spend some time discussion:
Fact 1 If we have k commodities, then one can show that max-flow/min-cut gap ≤ O(log k).
This can be shown directly, or it can be shown more generally via metric embedding methods.
Fact 2 If certain conditions are satisfied, then the duality gap = 0. If one look at dual polytope,
then whether or not this is the case depend on whether the optimal solution is integral or not.
This, in turn, depends on special structure of the input.
Fact 3 For k commodities, LR showed that the worst case (over input graph) gap Ω(log k). LLR
interpreted this geometrically in terms of embeddings. The worst case is achieved, and it is achieved
on expanders.
Here, we will spend some time showing these results directly. Then, next time we will describe it
more generally from the metric embedding perspective, since that will highlight better the similarities and differences with spectral methods.

10.4

Duality gap properties for flow-based methods

Here, we will show that there is a non-zero duality gap of size Θ(log n). We will do it in two steps:
first, by illustrating a particular graph (any expander) that has a gap at least Ω(log n); and second,
by showing that the gap is O(log n), i.e., is never worse than that.
Let’s start by showing that there is a graph for which the duality gap is nonzero by showing a
graph for which it is at least Θ(log n).

110

M. W. Mahoney

Theorem 24 (LR). ∀n, ∃ an n-node UMFP with MaxFlow f and MinCut Ξ s.t.


Ξ
f ≤O
log n
Proof. Consider any graph with certain expansion properties. In particular, let G be a 3-regular
n-node graph with unit edge capacities s.t.
C(U, Ū ) = |(U, Ū )| ≥ Const min{|U |, |Ū |},

∀U, V

i.e., an expander. (Such graphs exist by Margoulis, zig-zag, etc. constructions; and moreover a
randomly-selected 3-regular graph satisfies this w.h.p.) The first claim is that:
(U, Ū )|
U ⊆V |U | · |Ū |
Const
≥ min
U ⊆V max{|U |, |Ū |}
Const
.
=
n−1

Ξ =

The second claim is that:

Claim 5. The MaxFlow for UMFP is ≤

min

6
(n−1)(log n−1) ,

which is Θ(log n) smaller than MinCut.

n
Proof of claim. Since the graph
 is 3-regular, ∃ ≤ 2 nodes within a distance log n − 3 of v ∈ V .
n
So, for at least half of the 2 commodities, the shortest path connecting si and ti has at least
log n − 2 edges. To sustain a flow of f for such a commodity, at least f (log n − 2) capacity must be
used by
commodity. To sustain a flow f , ∀ n2 commodities, the capacity of the network must be

≥ 21 n2 f (log n − 2).

Since the graph is 3-regular with unit capacity, the total capacity is ≤
 
1 n
3n
.
f (log n − 2) ≤ CAP ACIT Y ≤
2 2
2

3n
2 .

So,

So,
f

3n
(log n − 2)
6
=
(n − 1)(log n − 2)
6Ξ
Const
≤
since Ξ ≥
Const (log n − 2)
n−1


Ξ
= O
.
log n
≤

n
2



That is, the MaxFlow for the UMFP on an expander G is ≥ Θ(log n) factor smaller than the
MinCut.
An expander has diameter O(log n), and so for expanders, the gap can’t be worse than Θ(log n),
but the following shows that this is true more generally.

Lecture Notes on Spectral Graph Methods

111

Theorem 25 (LR). The MinCut of UMFP can never be more than Θ(log n) factor larger than
MaxFlow, i.e.,


Ξ
Ω
≤ f ≤ Ξ.
log n
Proof. We can do this by showing a polynomial-time algorithm that finds a cut (U, Ū ) s.t.
C(U, Ū )
≤ O(f (log n))
|U | · |Ū |
Ū )
where the LHS is the ratio cost of (U, Ū ); and recall that Ξ = minS⊆V C(U,
. The algorithm is
|U |·|Ū|
based on LP Dual—the dual of Multicommodity Flow is the problem of assigning a fixed weight
(where the weights are thought of as distances) to edges of g s.t. one maximizes the cumulative
distance between source and sink pairs. (We won’t actually go through this now since there is a
more elegant and enlightening formulation.)

In the UMFP, the demand across cut (U, Ū ) is given by: D(U, Ū ) = |U ||Ū |. So, in particular, the
mincut is given by:
min cut:

where C(U, Ū ) =

P

C(U, Ū )
U ⊆V |U ||Ū |
E(U, Ū )
= min
U ⊆V |U ||Ū |

Ξ = min

if all capacities = 1,

e∈(U,Ū) C(e).

This is, if the demands are uniform and all the capacities are equal to one, then from UMFP we
get our old friend, the sparsest cut ∼ best expansion. Among other things, this implies that the
O(log n) approximation for general MultiCommodity Flow in “inherited” by the algorithm for the
sparsest cuts problem, and many other related problems as well. In particular, one way to use flow
is the following: check all 2n cuts, and use the single-commodity zero-duality gap result to show
that we can take the one with the best single cwommodity flow to get the one with the best mincut.
What these results say is that, instead, we can consider the all-pairs multi-commodity flow problem
and check a lot less and get a result that is only a little worse.

10.5

Algorithmic Applications

Let’s go back to our old friend the sparsest cut problem, and here we will make explicit connections
with flow based graph partitioning by viewing it from an optimization perspective. This will in
turn provide us with an algorithm (that is mentioned in the proof of the above theorem) to solve
the problem.
Recall that in the sparsest cut problem, we are given: a graph G = (V, E); a cost function c(e), ∀e ∈
E, i.e., c : E → R+ ; and k pairs of nodes/vertices (si , ti ). We will write the problem as an Integer
Program (IP). To do so,
• Let x(e), e ∈ E, where x(e) ∈ {0, 1} to indicate if an edge e is cut;

112

M. W. Mahoney
• let y(i), i ∈ [k], where y(i) ∈ {0, 1} to indicate if commodity i is cut, i.e., is disconnected by
this cut; and
• let Pi , i = 1, 2, . . . , k be the set of paths between si and ti .

Then, what we want is to solve:
P
c(e)x(e)
min Pe∈E
k
d(i)y(i)
Xi=1
s.t.
x(e) ≥ y(i),

∀P ∈ Pi ,

e∈P

y(i) ∈ {0, 1},

x(e) ∈ {0, 1},

∀i = 1, 2, . . . , k

i ∈ [k]
e∈E

We want to consider the relaxation of this to the case where replace the last two constraints by
y(i) ≥ 0 and x(e) ≥ 0 In doing so, note that if (x, y) is a feasible fractional solution, then (αx, αy)
is also a feasible fractional solution
Pk with the same objective function value. So, WLOG, we can
choose the normalization α s.t.
i=1 d(i)y(i) = 1 to get the following LP:
min

X

c(e)x(e)

e∈E

s.t.

k
X

d(i)y(i) = 1

i=1

X

e∈P

x(e) ≥ y(i),

y(i) ≥ 0

∀P ∈ Pi ,

∀i = 1, 2, . . . , k

, x(e) ≥ 0.

Below, we will show that we can compute a cut with sparsity ratio within a factor of O(log k) of
this optimal fractional solution.
BTW, before we do that, let’s write the LP dual:
max
s.t.

α
X

p∈Pi

f (p) ≥ αd(i),

k
X
X

i=1 p∈Pi (e)

f (p) ≥ 0,

∀i ∈ [k]

f (p) ≤ c(e),

∀e ∈ E

∀P ∈ Pi , i ∈ [k].

This is the Max Concurrent Flow problem, and O(log k) approximation gives an approximate
MaxFlow-MinCut Theorem.
So, to solve this sparsest cut problem, our strategy will be:
• Solve the LP (either the primal or the dual).

Lecture Notes on Spectral Graph Methods

113

• Round the solution to an integral value.
Thus, there are some similarities with spectral—we first solve something to get a real-valued solution, and then we have to “round” to get an integral solution, and the ball game will be to show
that we don’t loose too much. Of course, solving the LP will implicitly embed us in a very different
place than solving an eigenvector problem, so we will expect to see different artifactual properties
between the two approximation algorithms.
The above discussion gives algorithms that run in polynomial time:
• Off the shelf LP (due to a connection with LP).
• Algorithms for approximately optimal flows of distance functions (i.e., take advantage of the
structure of the LP).
• Fastest update of sparsest cut algorithm is Õ(n2 ), with Benczur-Karger sparsification.
• Standard algorithm that runs in something like Õ(n3/2 ) with push-relabel methods.
• Finally, there is a lot of work on using Laplacian-based solvers to do better, and we may
return to these later.
Important Note: The way you would actually solve this in practice is to use some sort of pushrelabel code, which is relatively fast, as opposed to the general LP procedure just described, which
is easier to analyze theoretically.

10.6

Flow Improve

Here is an aside that with luck we will get back to later. These algorithms have a running time that
large enough that it can be challenging to apply to very large graphs—e.g., O(n3/2 ) or especially
O(n2 ) is certainly too large for “massive” graphs. (Implementations of faster algorithms are still
very much a topic of research.) A question arises, can we do something like “local spectral” (which,
recall, consisted roughly of a few steps of a random walk), to do a local flow improvement?
The answer is YES—and here is it. The so-called Improve algorithm of AL as well as the related
MQI method: this algorithm is useful by itself for improving partitions from, say, Metis or some
other very fast procedures; and it is useful as a way to speed up spectral and/or as one way to
combine spectral-based algorithms with flow-based algorithms. In more detail, the goal of a local
improvement algorithm is: Given a partition, find a strictly better partition. A local improvement
algorithm is useful in the following contexts:
• METIS – post process with a flow based improvement heuristic.
• Vanilla spectral: post process with improvement method.
• Local improvement at one step online iterative algorithm.
MQI and Improve essentially construct and solve a sequence of s-t MinCut problems on a modified
quotient cut objective to add and remove vertices from a proposed cut. (We won’t describe them
in detail now.) Here is the basic theorem stating their running time and approximation quality
bounds.

114

M. W. Mahoney

Theorem 26. Let A ⊆ V s.t. π(A) ≤ π(Ā), and let S = Improve(A) be the output of the Flow
Improve Algorithm. Then
|∂S|
I. If C ⊆ A, ( i.e., ∀C ⊆ A) then Q(S) ≤ Q(C) (where Q(S) = Vol
)
(S)

II. If C is such that
π(A)
π(Ā)
π(A ∩ C)
≥
+ǫ
,
π(C)
π(V )
π(V )

for some ǫ

i.e., if C is ǫ-more-correlated with A than random, i.e., if the fraction of C that is in A is
ǫ-better than random, then Q(S) ≤ 1ǫ Q(C) i.e., bound on nearby cuts.
III. The algorithm takes time: (1) π(V )2 iterations if vertex weights π(V ) ∈ Z; and (2) m
iterations if the edges are unweighted.

Lecture Notes on Spectral Graph Methods

11

115

(02/26/2015): Flow-based Methods for Partitioning Graphs (2
of 2)

Reading for today.
• Same as last class.
Recall from last time that we are looking at flow-based graph partitioning algorithm. Last time, we
covered the basics of flow-based methods, and we showed how they are very different than spectral
methods. This time, we will discuss flow-based graph partitioning from an embedding perspective.
We will see that flow-based algorithms implicitly embed the data in a metric space, but one that is
very different than the place where spectral-based algorithms embed the data. Thus, not only do
they run different steps operationally and get incomparable quality of approximation bounds, but
also they implicitly put the data in a very different place—thus “explaining” many of the empirical
results that are empirically observed.
(BTW, I have made some comments that spectral methods can scale up to much larger input graphs
by using diffusion and random walk ideas, a topic we will get back to later. For the moment, let
me just note that the way flow is used is not immediately relevant for such “massive” data. For
example, the running time of a typical flow-based algorithm will be O(n2 ), since it involves a multicommodity variant of flow; single commodity variants of flow-based algorithms run in O(n3/2 ) time;
and more recent work has focused on using Laplacian solver ideas to do even better, i.e., to run
in time that is nearly-linear in the size of the input. There is a lot of interest, mostly from within
TCS so far, in this area; and these fast solvers hold the promise to make these methods applicable
to much larger graphs. I’m hoping to return to some of these Laplacian solver topics at the end of
the semester, and I’m also planning on giving at least give a brief introduction to some of the ideas
about how to couple spectral methods later.)

11.1

Review some things about ℓ1 and ℓ2

Let’s review a few things about ℓ1 and ℓ2 and related topics.
P
1/p
k
Definition 39. The ℓp -norm on Rk is ||x||p =
|x
|
. A finite metric space (X, ρ) is
i=1 i

realized in ℓkp if ∃f : X → Rk s.t. ρ(x, y) = ||f (x) − f (y)||2 , and it is an ℓp -metric if it can be
realized by ℓkp for some k. The metric induced by ℓp is: d(x, y) = ||x − y||p , ∀x, y ∈ ℓp .

For flow-based methods, we will be most interested in ℓ1 , while for spectral-based methods, we are
interested in ℓ2 . The ℓp norm (except for p = ∞, which we won’t discuss here) is usually of at most
more theoretical interest.
The spaces ℓ1 and ℓ2 are very different. (Think ℓ1 regression, i.e., least absolute deviations, versus
ℓ2 regression, i.e., least-squares regression; or think ℓ1 regularized ℓ2 regression, i.e., the lasso, versus
ℓ2 regularized ℓ2 regression, i.e., ridge regression. These differences are often “explained” in terms
of differences between the unit ball in the ℓ1 norm versus the unit ball in the ℓ2 norm, with the
former being “pointy” and the latter being “smooth.” In particular, note that in those situations
ℓ1 often has some sort of connection with sparsity and sparse solutions.)
Here is a comparison between ℓ1 and ℓ2 with respect to the spectral/flow algorithms for the graph
partitioning problem we have been considering.

116

M. W. Mahoney
• ℓ2 norm:
– Good for dimension reduction.
– Efficient polynomial time algorithm to compute embedding of any finite metric space.
– Connections to low-dimensional manifolds, diffusions, etc.
• ℓ1 norm:
– No good for dimension reduction.
– NP-hard to compute the optimal embedding.
– Connections to partitions/cuts, multicommodity flow, etc.

The following is a fundamental result in the area that is also central to understanding why flowbased graph partitioning algorithms work.
Theorem 27 (Bourgain). Every n-point metric space d admits an α-distortion embedding into ℓp ,
∀p, with α = O(log n).
Proof idea. The proof is similar to but more general than the proof for the corresponding embedding
claim for ℓ2 . The idea is: to use the so-called Frechet embedding method, where the embedding is
given by the distance from points to carefully randomly chosen subsets of nodes.
Note that we saw the ℓ2 version of this before. Note also that the original embedding had 2n
dimensions, but LLR proved that it can be done with O(log2 n) dimensions.

11.2

Connection between ℓ1 metrics and cut metrics

First, recall what a metric is.
Definition 40. A metric is a function d : V × V → R s.t.: (1) d(x, y) = 0 if x = 0 (and sometimes
= 0 iff x = y); (2) d(x, y) = d(y, x); and (3) d(x, y) + d(x, z) ≤ d(x, y)
(Recall also that sometimes the word “metric” if one or more of those conditions is/are relaxed.)
Next, recall the definition of the “cut metric,” and recall that this is really not a metric but is
instead just a semi-metric.
Definition 41. Given G = (V, E) and a set S ⊂ V , δS is the “cut metric” for S if
δS (i, j) = |χS (i) − χS (j)|,
where
χS (i) =
Thus
δS (i, j) =





0 if i ∈ S
1 otherwise.

0 if i, j ∈ S, or i, j ∈ S̄
1 otherwise.

(That is, if δs (x, y) is the indicator of x and y being on different sides of S.)

Lecture Notes on Spectral Graph Methods

117

There are important connection between ℓ1 metrics and Cut metrics. In particular:
• There exists a representation of the ℓ1 metrics as a conical combination of cut metrics.
• Cut metrics are the extreme rays of the ℓ1 cone.
• For these reasons, instead of minimizing the ratio of linear functions over the convex cone,
we can minimize the ratio over the extreme rays of the convex cone. Minimum ratio function
over cone ⇐⇒ minimum over extreme rays.
We’ll spend most of today going over these results.
n

Fact: An n-point metric space can be associated with a vector in R( 2 ) , with each coordinate
corresponding to a pair of vertices.
¯ Then, αd¯ + (1 − α)d¯′ is
Fact: Given a metric d, we will refer to the corresponding vector as d.
¯ αd¯ is a metric. So, the set of all metrics forms a
a metric, ∀α ∈ [0, 1]. In addition, ∀α ≥ 0, ∀d,
n
convex cone in R( 2 ) . In somewhat more detail, we have the following result:
Claim 6. The set of ℓ1 metrics is a convex cone, i.e., if d1 and d2 ∈ ℓ1 metrics, and if λ1 , λ2 ∈ R+ ,
then λ1 d1 + λ2 d2 ∈ ℓ1 metrics.
Proof. Recall that the line metric is an ℓ1 metric. Let d(i) (x, y) = |xi − yi |, for x, y ∈ Rc . If d ∈ ℓ1
metric, then it is the sum of line metrics.
Fact. The analogous result does not hold for ℓ2 .
Next, we have the following theorem:
Theorem 28. Let d be a finite ℓ1 metric. Then, d can be written as
X
d=
αS δS ,
S⊂[n]

for some constant αS and cut metrics δS . That is,
X
CU T n = {d : d =
αS δS , α ≥ 0}
S⊆V

= Positive cone generated by cut metrics
= All n-point subsets of Rn , under the ℓ1 norm.

Proof. Consider any metric d ∈ CU T n . Then, ∀S with αS > 0, we have a dimension, and in that
dimension, we can put
(
αS if x ∈ S̄
0
if x ∈ S.
So, CU T n ⊆ ℓ1

Metrics.

For the other direction, consider a set of n points from Rn . Take one dimension d and sort the
points in increasing values along that dimension. Say that we get v1 , . . . , vk as the set of distinct

118

M. W. Mahoney

values; then define k − 1 cut metrics: Si = {x : xd < vi+1 }, and let αi = vi+1 − vi , i.e., k − 1 coeff.
So, along this dimension, we have that
|xd − yd | =

k
X

αi δSi ,

i=1

But, one can construct cut metrics for every dimension. So, we have cut metrics in CU T n , ∀
n-point metrics ℓ1 ; thus, ℓ1 ⊆ CU T .

11.3

Relating this to a graph partitioning objective

Why is this result above useful? The usefulness of this characterization is that we are going to
want to to optimize functions, and rather than optimize functions over all cut metrics, i.e., over
the extreme rays, we will optimize over the full convex cone, i.e., over ℓ1 metrics.
This leads us to the following lemma:
Lemma 16. Let C ⊂ Rn be a convex cone, and let f, g : Rn,+ → R+ be linear functions. And
(x)
exists. Then
assume that minx∈C fg(x)
min
x∈C

f (x)
=
min
g(x) x in extreme rays of

C

f (x)
.
g(x)

P
Proof. Let x0 be the optimum. Since x0 ∈ C, we have that x0 = i ai yi , where ai ∈ R+ and
yi ∈ extreme rays of C. Thus,
P
P
f ( i ai y i )
f (ai yi )
f (x0 )
= P
= Pi
g(x0 )
g( i ai yi )
i g(ai yi )
f (aj yj )
≥
where j is the min value
g(aj yj )
f (yi )
=
,
g(yi )
where the first and third line follow by linearity, and where the second line follows since
P
α
α
Pi i ≥ min j
j
βj
i βi
in general.

To see the connections of all of this to sparsest cut problem, recall that given a graph G = (V, E)
we define the conductance hG and sparsity φG as follows:
E(S, S̄)
min{|S|, |S̄|}
E(S, S̄)
φG := min 1
,
S⊆V
n |S||S̄|
hG := min
S⊆V

Lecture Notes on Spectral Graph Methods

119

and also that:
hG ≤ φG ≤ 2hG .
(This normalization might be different than what we had a few classes ago.)
Given this, we can write sparsest cut as the following optimization problem:
Lemma 17. Solving
φG = min
S⊆V

E(S, S̄)
1
n |S||S̄|

is equivalent to solving:
min

X

dij

(ij)∈E

s.t.

X

dij = 1

ij∈V

d ∈ ℓ1 metric
Proof. Let δS = the cut metric for S. Then,

So,

P
|E(S, S̄)|
ij∈E δS (i, j)
= P
|S| · |S̄|
∀ij δS (i, j)
P

ij∈E δS (i, j)
φG = min P
S
∀ij δS (i, j)

Since ℓ1 -metrics are linear combinations of cut metrics, and cut metrics are extreme rays of ℓ1 from
the above lemma, this ratio is minimized at one of the extreme rays of the cone. So,
P

ij∈E dS (ij)
φG = min P
.
d∈ℓ1
∀ij dS (ij)

Since this is invariant to scaling, WLOG we can assume

11.4

P

∀ij

dij = 1; and we get the lemma.

Turning this into an algorithm

It is important to note that the above formulation is still intractable—we have just changed the
notation/characterization. But, the new notation/characterization suggests that we might be able
to relax (optimize the same objective function over a larger set) the optimization problem—as we
did with spectral, if you recall.
So, the relaxation we will consider is the following: relax the requirement that d ∈ ℓ1 Metric to

120

M. W. Mahoney

d ∈ Any Metric. We can do this by adding 3
X
λ∗ = min
dij

n
3

triangle inequalities to get the following LP:

ij∈E

s.t.

X

dij = 1

∀ij∈V

dij ≥ 0

dij = dji
dij ≤ dik + djk

∀i, j, k

triples

(Obviously, since there are a lot of constraints, a naive solution won’t be good for big data, but we
will see that we can be a bit smarter.) Clearly,
λ∗ ≤ φ∗ = Solution with d ∈ ℓ1

Metric constraint

(basically since we are minimizing over a larger set). So, our goal is to show that we don’t loose
too much, i.e., that:
φ∗ ≤ O(log n)λ∗ .
Here is the Algorithm. Given as input a graph G, do the following:
• Solve the above LP to get a metric/distance d : V × V → R+ .
• Use the (constructive) Bourgain embedding result to embed d into an ℓ1 metric (with, of
course an associated O(log n) distortion).
• Round the ℓ1 metric (the solution) to get a cut.
– For each dimension/direction, covert the ℓ1 embedding/metric along that to a cut metric
representation.
– Choose the best.
Of course, this is what is going on under the hood—if you were actually going to do it on systems
of any size you would use something more specialized, like specialized flow or push-relabel code.
Here are several things to note.
• If we have ℓ1 embedding with distortion factor ξ then can approximate the cut up to ξ.
• Everything above is polynomial time, as we will show in the next theorem.
• In practice, we can solve this with specialized code to solve the dual of corresponding multicommodity flow.
• Recall that one can “localize” spectral by running random walks from a seed node. Flow is
hard to localize, but recall the Improve algorithm, but which is still Õ(n3/2 ).
• We can combine spectral and flow, as we will discuss, in various ways.
Theorem 29. The algorithm above is a polynomial time algorithm to provide an O(log n) approximation to the sparsest cut problem.

Lecture Notes on Spectral Graph Methods

121

Proof. First, note that solving the LP is a polynomial time computation to get a metric d∗ . Then,
note that the Bourgain embedding lemma is constructive. Finding an embedding of d∗ to d ∈
O(log2 n)
ℓ1
with
write d as a linear combination of O(n log2 n) cut
Pdistortion O(log n). So, we can
metrics d = S∈S αS δS , where |S| = O(n log2 n). Note:
P
P
ij∈E δS (ij)
ij∈E dij
min P
≤ P
S∈S
∀ij δS (ij)
∀ij dij
P
∗
ij∈E dij
≤ O(log n) P
∗ ,
∀ij dij
where the first inequality follows since d is in the cone of cut metrics, and where the second
inequality follows from Bourgain’s theorem. But,
P

ij∈E

P

d∗ij

∗
∀ij dij

=

d′

P

′

P

ij∈E dS (ij)
P
≤ min P
min
,
′
∀S
is metric
∀ij dS (ij)
∀ij dij
ij∈E

dij

where the equality follows from the LP solution and the inequality follows since LP is a relaxation
of a cut metric. Thus,
P
P
ij∈E dS (ij)
ij∈E δS (ij)
≤ O(log n) min P
.
min P
S∈S
∀S
∀ij δS (ij)
∀ij dS (ij)
This establishes the theorem.

So, we can also approximate the value of the objective—how do we actually find a cut from this?
(Note that sometimes in the theory of approximation algorithms you don’t get anything more than
an approximation to the optimal number, but that is somewhat dissatisfying if you want you use
the output of the approximation algorithm for some downstream data application.)
To see this:
• Any ℓ1 metric can be written
P as a conic combination of cut metrics—in our case, with
n
σ
O(n log ) nonzeros—d = S αS δS .

• So, pick the best cut from among the ones with nonzero α in the cut decomposition of dσ .

11.5

Summary of where we are

Above we showed that
E(S, S̄)
S⊂V |S||S̄|
X
= min
dij

φG = min

ij∈E

s.t.

X

dij = 1

ij∈V

d ∈ ℓ1 metric

122

M. W. Mahoney

can be approximated by relaxing it to
min

X

dij

ij∈E

s.t.

X

dij = 1

ij∈V

d ∈ Metric
This relaxation is different than the relaxation associated with spectral, where we showed that
φ=

min

x∈{0,1}V

can be relaxed to
d − λ2 = min
x⊥~1

Aij |xi − xj |2
1 P
2
ij |xi − xj |
n
Aij (xi − xj )2
1 P
2
ij (xi − xj )
n

which can be solved with the second eigenvector of the Laplacian.
Note that these two relaxations are very different and incomparable, in the sense that one is not
uniformly better than the other. This is related to them succeeding and failing in different places,
and it is related to them parametrizing problems differently, and it can be used to diagnose the
properties of how each class of algorithms performs on real data. Later, we will show how to
generalize this previous flow-based result and combine it with spectral.
Here are several questions that the above discussion raises.
• What else can you relax to?
• In particular, can we relax to something else and improve the O(log n) factor?
• Can we combine these two incomparable ideas to get better bounds in worst-case and/or in
practice?
• Can we combine these ideas to develop algorithms that smooth or regularize better in applications for different classes of graphs?
• Can we use these ideas to do better learning, e.g., semi-supervised learning on graphs?
We will address some of these questions later in the term, as there is a lot of interest in these and
related questions.

Lecture Notes on Spectral Graph Methods

12

123

(03/03/2015): Some Practical Considerations (1 of 4): How
spectral clustering is typically done in practice

Reading for today.
• “A Tutorial on Spectral Clustering,” in Statistics and Computing, by von Luxburg
Today, we will shift gears. So far, we have gone over the theory of graph partitioning, including
spectral (and non-spectral) methods, focusing on why and when they work. Now, we will describe
a little about how and where these methods are used. In particular, for the next few classes, we
will talk somewhat informally about some practical issues, e.g., how spectral clustering is done in
practice, how people construct graphs to analyze their data, connections with linear and kernel
dimensionality reduction methods. Rather than aiming to be comprehensive, the goal will be to
provide illustrative examples (to place these results in a broader context and also to help people
define the scope of their projects). Then, after that, we will get back to some theoretical questions
making precise the how and where. In particular, we will then shift to talk about how diffusions
and random walks provide a robust notion of an eigenvector and how they can be used to extend
many of the vanilla spectral methods we have been discussing to very non-vanilla settings. This
will then lead to how we can use spectral graph methods for other related problems like manifold
modeling, stochastic blockmodeling, Laplacian solvers, etc.
Today, we will follow the von Luxburg review. This review was written from a machine learning
perspective, and in many ways it is a very good overview of spectral clustering methods; but beware:
it also makes some claims (e.g., about the quality-of-approximation guarantees that can be proven
about the output of spectral graph methods) that—given what we have covered so far—you should
immediately see are not correct.

12.1

Motivation and general approach

The motivation here is two-fold.
• Clustering is an extremely common method for what is often called exploratory data analysis.
For example, it is very common for a person, when confronted with a new data set, to try to
get a first view of the data by identifying subsets of it that have similar behavior or properties.
• Spectral clustering methods in particular are a very popular class of clustering methods. They
are usually very simple to implement with standard linear algebra libraries; and they often
outperform other methods such as k-means, hierarchical clustering, etc.
The first thing to note regarding general approaches is that Section 2 of the von Luxburg review
starts by saying “Given a set of data points x1 , . . . , xn and some notion of similarity sij ≥ 0 between
all pairs of data points xi and xj ...” That is, the data are vectors. Thus, any graphs that might
be constructed by algorithms are constructed from primary data that are vectors and are useful as
intermediate steps only. This will have several obvious and non-obvious consequences. This is a
very common way to view the data (and thus spectral graph methods), especially in areas such as
statistics, machine learning, and areas that are not computer science algorithms. That perspective
is not good or bad per se, but it is worth emphasizing that difference. In particular, the approach
we will now discuss will be very different than what we have been discussing so far, which is more

124

M. W. Mahoney

common in CS and TCS and where the data were a graph G = (V, E), e.g., the single web graph out
there, and thus in some sense a single data point. Many of the differences between more algorithmic
and more machine learning or statistical approaches can be understood in terms of this difference.
We will revisit it later when we talk about manifold modeling, stochastic blockmodeling, Laplacian
solvers, and related topics.

12.2

Constructing graphs from data

If the data are vectors with associated similarity information, then an obvious thing to do is to
represent that data as a graph G = (V, E), where each vertex v ∈ V is associated with a data
point xi an edge e = (vi vj ) ∈ E is defined if sij is larger than some threshold. Here, the threshold
could perhaps equals zero, and the edges might be weighted by sij . In this case, an obvious idea
to cluster the vector data is to cluster the nodes of the corresponding graph.
Now, let’s consider how to specify the similarity information sij . There are many ways to construct
a similarity graph, given the vectors {xi }ni=1 data points as well as pairwise similarity (or distance)
information sij (of dij ). Here we describe several of the most popular.
• ǫ-NN graphs. Here, we connect all pairs of points with distance dij ≤ ǫ. Since the distance
“scale” is set (by ≤ ǫ), it is common not to including the weights. The justification is that,
in certainly idealized situations, including weights would not incorporate more information.
• k-NN graphs. Here, we connect vertex i with vertex j if vi is among the k-NN of vi ,
where NNs are given by the distance dij . Note that this is a directed graph. There are two
common ways to make it undirected. First, ignore directions; and second, include an edge if
(vi connects to vj AND vj connects to vi ) or if (vi connects to vj OR vj connects to vi ). In
either of those cases, the number of edges doesn’t equal k; sometimes people filter it back to
exactly k edges per node and sometimes not. In either case, weights are typically included.
• Fully-connected weighted graphs. Here, we connect all points with a positive similarity
to each other. Often, we want the similarity function to represent local neighborhoods, and
so sij is either transformed into another form or constructed to represent this. A popular
choice is the Gaussian similarity kernel
s(xi , xj ) = exp




1
2
kxi − xj k2 ,
2σ 2

where σ is a parameter that, informally, acts like a width. This gives a matrix that has a
number of nice properties, e.g., it is positive and it is SPSD, and so it is good for MLers who
like kernel-based methods. Moreover, it has a strong mathematical basis, e.g., in scientific
computing. (Of course, people sometimes use this sij = s(xi , xj ) information to construct
ǫ-NN or k-NN graphs.)
Note that in describing those various ways to construct a graph from the vector data, we are
already starting to see a bunch of knobs that can be played with, and this is typical of these graph
construction methods.
Here are some comments about that graph construction approach.

Lecture Notes on Spectral Graph Methods

125

• Choosing the similarity function is basically an art. One of the criteria is that typically one is
not interested in resolving differences that are large, i.e., between moderately large and very
large distances, since the goal is simply to ensure that those points are not close and/or since
(for domain-specific reasons) that is the least reliable similarity information.
• Sometimes this approach is of interest in semi-supervised and transductive learning. In this
case, one often has a lot of unlabeled data and only a little bit of labeled data; and one wants
to use the unlabeled data to help define some sort of geometry to act as a prior to maximize
the usefulness of the labeled data in making predictions for the unlabeled data. Although this
is often thought of as defining a non-linear manifold, you should think of it at using unlabeled
data to specify a data-dependent model class to learn with respect to. (That makes sense
especially if the labeled and unlabeled data come from the same distribution, since in that
case looking at the unlabeled data is akin to looking at more training data.) As we will see,
these methods often have an interpretation in terms of a kernel, and so they are used to learn
linear functions in implicitly-defined feature spaces anyway.
• k-NN, ǫ-NN, and fully-connected weighted graphs are all the same in certain very idealized
situations, but they can be very different in practice. k-NN often homogenizes more, which
people often like, and/or it connects points of different “size scales,” which people often
find useful.
• Choosing k, ǫ, and σ large can easily “short circuit” nice local structure, unless (and sometimes
even if) the local structure is extremely nice (e.g., one-dimensional). This essentially injects
large-scale noise and expander-like structure; and in that case one should expect very different
properties of the constructed graph (and thus very different results when one runs algorithms).
• The fully-connected weighted graph case goes from being a rank-one complete graph to being
a diagonal matrix, as one varies σ. An important question (that is rarely studied) is how does
that graph look like as one does a “filtration” from no edges to a complete graph.
• Informally, it is often thought that mutual-k-NN is between ǫ-NN and k-NN: it connects
points within regions of constant density, but it doesn’t connect regions of very different
density. (For ǫ-NN, points on different scales don’t get connected.) In particular, this means
that it is good for connecting clusters of different densities.
• If one uses a fully-connected graph and then sparsifies it, it is often hoped that the “fundamental structure” is revealed and is nontrivial. This is true in some case, some of which we
will return to later, but it is also very non-robust.
• As a rule of thumb, people often choose parameters s.t. ǫ-NN and k-NN graphs are at least
“connected.” While this seems reasonable, there is an important question of whether it
homogenizes too much, in particular if there are interesting heterogeneities in the graph.

12.3

Connections between different Laplacian and random walk matrices

Recall the combinatorial or non-normalized Laplacian
L = D − W,

126

M. W. Mahoney

and the normalized Laplacian
Lsym = D −1/2 LD −1/2 = I − D −1/2 W D −1/2 .
There is also a random walk matrix that we will get to more detail on in a few classes and that for
today we will call the (somewhat non-standard name) random walk Laplacian
Lrw = D −1 L = I − D −1 W = D −1/2 Lsym D 1/2 .
Here is a lemma connecting them.
Lemma 18. Given the above definitions of L, Lsym , and Lrw , we have the following.
I. For all x ∈ Rn ,

1X
x Lsym x =
Wij
2
T

ij

x
x
√ i − pj
di
dj

!2

.

II. λ is an eigenvalue of Lrw with eigenvector u iff λ is an eigenvalue of Lsym with eigenvector
w = D 1/2 u
III. λ is an eigenvalue of Lrw with eigenvector u iff λ and u solve the generalized eigenvalue
problem Lu = λDu.
IV. 0 is an eigenvalue of Lrw with eigenvector ~1 iff 0 is an eigenvalue of Lsym with eigenvector
D 1/2~1
V. Lsym and Lrw are PSD and have n non-negative real-valued eigenvalues 0 = λ1 ≤ · · · ≤ λn .
Hopefully none of these claims are surprising by now, but they do make explicit some of the
connections between different vectors and different things that could be computed, e.g., one might
solve the generalized eigenvalue problem Lu = λDu or run a random walk to approximate u and
then from that rescale it to get a vector for Lsym .

12.4

Using constructed data graphs

Spectral clustering, as it is often used in practice, often involves first computing several eigenvectors
(or running some sort of procedures that compute some sort of approximate eigenvectors) and then
performing k-means in a low-dimensional space defined by them. Here are several things to note.
• This is harder to analyze than the vanilla spectral clustering we have so far been considering.
The reason is that one must analyze the k means algorithm also. In this context, k-means is
essentially used as a rounding algorithm.
• A partial justification of this is provided by the theoretical result on using the leading k
eigenvectors that you considered on the first homework.
• A partial justification is also given by a result we will get to below that shows that it works
in very idealized situations.

Lecture Notes on Spectral Graph Methods

127

We can use different Laplacians in different ways, as well as different clustering, k-means, etc.
algorithms in different ways to get spectral-like clustering algorithms. Here, we describe 3 canonical
algorithms (that use L, Lrw , and Lsym ) to give an example of several related approaches.
Assume that we have n points, x1 , . . . , xn , that we measure pairwise similarities sij = s(xi , xj )
with symmetric nonnegative similarity function, and that we denote the similarity matrix by S =
(Sij )i,j=1,...,n . The following algorithm, let’s call it PopularSpectralClustering, takes as input
a similarity matrix S ∈ Rn×n and a positive integer k ∈ Z+ which is the number of clusters; and it
returns k clusters. It does the following steps.
I. Construct a similarity graph (e.g., with ǫ-NN, k-NN, fully-connected graph, etc.)
II. Compute the unnormalized Laplacian L = D − A.
III.

• If (use L)
then compute the first k eigenvectors u1 , . . . , uk of L,
• else if (use Lrw )
then compute first k generalized eigenvectors u1 , . . . , uk of the generalized eigenvalue
problem Lu = λDu. (Note by the above that these are eigenvectors of Lrw .)
• else if (use Lsym )
then compute the first k eigenvectors u1 , . . . , uk of Lsym .

IV. Let U ∈ Rn×k be the matrix containing the vectors u1 , . . . , uk as columns.
V.

• If (use Lsym )
P 2 1/2
then uij ← uij /
, i.e., normalize U row-wise.
k uik

VI. For i = {1, . . . , n}, let yi ∈ Rk be a vector containing the ith row of U .
VII. Cluster points (yi )i∈[n] in Rk with a k-means algorithm into clusters, call them C1 , . . . , Ck .
VIII. Return: clusters A1 , . . . , Ak , with Ai = {j : yj ∈ Ci }.
Here are some comments about the PopularSpectralClustering algorithm.
• The first step is to construct a graph, and we discussed above that there are a lot of knobs.
In particular, the PopularSpectralClustering algorithm is not “well-specified” or “welldefined,” in the sense that the algorithms we have been talking about thus far are. It might
be better to think of this as an algorithmic approach, with several knobs that can be played
with, that comes with suggestive but weaker theory than what we have been describing so far.
• k-means is often used in the last step, but it is not necessary, and it is not particularly
principled (although it is often reasonable if the data tend to cluster well in the space defined
by U ). Other methods have been used but are less popular, presumably since k-means is
good enough and there are enough knobs earlier in the pipeline that the last step isn’t the
bottleneck to getting good results. Ultimately, to get quality-of-approximation guarantees for
an algorithm like this, you need to resort to a Cheeger-like bound or a heuristic justification
or weaker theory that provides justification in idealized cases.

128

M. W. Mahoney
• In this context, k-means is essentially a rounding step to take a continuous embedding, provided by the continuous vectors {yi }, where yi ∈ Rk , and put them into one of k discrete
values. This is analogous to what the sweep cut did. But we will also see that this embedding,
given by {yi }ni=1 can be used for all sorts of other things.
• Remark: If one considers the k-means objective function, written as an IP and then relaxes
it (from having the constraint that each data point goes into one of k clusters, written as
an orthogonal matrix with one nonzero per column, to being a general orthogonal matrix),
then you get an objective function, the solution to which can be computed by computing a
truncated SVD, i.e., the top k singular vectors. This provides a 2-approximation to the kmeans objective. There are better approximation algorithms for the k-means objective, when
measured by the quality-of-approximation, but this does provide an interesting connection.
• The rescaling done in the “If (use Lsym ) then” is typical of many spectral algorithms, and
it can be the source of confusion. (Note that the rescaling is done with respect to (PU )ii =
U U T ii , i.e., the statistical leverage scores of U , and this means that more “outlying” points
get down-weighted.) From what we have discussed before, it should not be surprising that
we need to do this to get the “right” vector to work with, e.g., for the Cheeger theory we
talked about before to be as tight as possible. On the other hand, if you are approaching this
from the perspective of engineering an algorithm that returns clusters when you expect them,
it can seem somewhat ad hoc. There are many other similar ad hoc and seemingly ad hoc
decisions that are made when engineering implementations of spectral graph methods, and
this lead to a large proliferation of spectral-based methods, many of which are very similar
“under the hood.”

All of these algorithms take the input data xi ∈ Rn and change the representation in a lossy way to
get data points yi ∈ Rk . Because of the properties of the Laplacian (some of which we have been
discussing, and some of which we will get back to), this often enhances the cluster properties of
the data.
In idealized cases, this approach works as expected, as the following example provides. Say that
we sample data points from R from four equally-spaced Gaussians, and from that we construct
a NN graph. (Depending on the rbf width of that graph, we might have an essentially complete
graph or an essentially disconnected graph, but let’s say we choose parameters as the pedagogical
example suggests.) Then λ1 = 0; λ2 , λ3 , and λ4 are small; and λ5 and up are larger. In addition,
v1 is flat; and higher eigenfunctions are sinusoids of increasing frequency. The first few eigenvectors
can be used to split the data into the four natural clusters (they can be chosen to be worse linear
combinations, but they can be chosen to split the clusters as the pedagogical example suggests).
But this idealized case is chosen to be “almost disconnected,” and so it shouldn’t be surprising that
the eigenvectors can be chosen to be almost cluster indicator vectors. Two things: the situation
gets much messier for real data, if you consider more eigenvectors; and the situation gets much
messier for real data, if the clusters are, say, 2D or 3D with realistic noise.

12.5

Connections with graph cuts and other objectives

Here, we will briefly relate what we have been discussing today with what we discussed over the last
month. In particular, we describe the graph cut point of view to this spectral clustering algorithm.
I’ll follow the notation of the von Luxburg review, so you can go back to that, even though this is

Lecture Notes on Spectral Graph Methods

129

very different than what we used before. The point here is not to be detailed/precise, but instead
to remind you what we have been covering in another notation that is common, especially in ML,
and also to derive an objective that we haven’t covered but that is a popular one to which to add
constraints.
To make connections with the PopularSpectralClustering algorithm and MinCut, RatioCut,
and NormalizedCut, recall that


k
k
X
cut Ai , Āi
1 X W Ai , Āi
=
,
RatioCut(A1 , . . . , Ak ) =
2
|Ai |
|Ai |
i=1

i=1

where cut(A1 , . . . , Ak ) =

1
2

Pk

i=1


Ai , Āi .

First, let’s consider the case k = 2 (which is what we discussed before). In this case, we want to
solve the following problem:

min RatioCut A, Ā .
(19)
A⊂V

Given A ⊂ V , we can define a function f = (f1 , . . . , fn )T ∈ Rn s.t.
( p
|Ā|/|A|
if vi ∈ A
p
.
fi =
− |A|/|Ā|
if vi ∈ Ā

(20)

In this case, we can write Eqn. (19) as follows:
min f T Lf

A⊂V

s.t. f ⊥ ~1

f defined as in Eqn. (20)
√
kf k = n.

In this case, we can relax this objective to obtain
min f T Lf

A⊂V

s.t. f ⊥ ~1

kf k =

√
n,

which can then be solved by computing the leading eigenvectors of L.
Next, let’s consider the case k > 2 (which is more common in practice). In this case, given a partition
of the vertex set V into k sets Ai , . . . , Ak , we can define k indicator vectors hj = (hij , . . . , hnj )T by
 p
1/ |Aj |
vi ∈ Aj , i ∈ [n], j ∈ [k]
hij =
.
(21)
0
otherwise
Then, we can set the matrix H ∈ Rn×k as the matrix containing those k indicator vectors as
columns, and observe that H T H = I, i.e., H is an orthogonal matrix (but a rather special one,
since it has only one nonzero per row).
We note the following observation; this is a particular way to write the RatioCut problem as a
Trace problem that appears in many places.

130

M. W. Mahoney

Claim 7. RatioCut(A1 , . . . , Ak ) = Tr(H T LH)
Proof. Observe that
hTi Lhi

cut Ai , Āi
=
|Ai |



and also that
hTi Lhi = H T LH
Thus, we can write
RatioCut (A1 , . . . , Ak ) =

k
X

hi Lhi =



ii

k
X

.

H T LH

i=1

i=1



ii


= Tr H T LH .

So, we can write the problem of
min RatioCut (A1 , . . . , Ak )
as follows:
min Tr H T LH

A1 ,...,Ak

s.t. H T H = I



H defined as in Eqn. (21).
We can relax this by letting the entries of H be arbitrary elements of R (still subject to the overall
orthogonality constraint on H) to get
min Tr H T LH

H∈Rn×k

s.t. H T H = I,



and the solution to this is obtained by computing the first k eigenvectors of L.
Of course, similar derivations could be provided for the NormalizedCut objective, in which case we
get similar results, except that we deal with degree weights, degree-weighted constraints, etc. In
particular, for k > 2, if we define indicator vectors hj = (hij , . . . , hnj )T by
hij =



p
1/ Vol(Aj )
0

vi ∈ Aj , i ∈ [n], j ∈ [k]
.
otherwise

then the problem of minimizing NormalizedCut is
min Tr H T LH

A1 ,...,Ak



s.t. H T DH = I

H defined as in Eqn. (22),

(22)

Lecture Notes on Spectral Graph Methods

131

and if we let T = D 1/2 H, then the spectral relaxation is


min Tr T T D −1/2 LD −1/2 T
T ∈Rn×k

s.t. T T T = I,

and the solution T to this trace minimization problem is given by the leading eigenvectors of Lsym .
Then H = D −1/2 T , in which case H consists of the first k eigenvectors of Lrw , or the first k
generalized eigenvectors of Lu = λDu.
Trace optimization problems of this general for arise in many related applications. For example:
• One often uses this objective as a starting point, e.g., to add sparsity or other constraints, as
in one variation of “sparse PCA.”
• Some of the methods we will discuss next time, i.e., LE/LLE/etc. do something very similar
but from a different motivation, and this provides other ways to model the data.
• As noted above, the k-means objective can actually be written as an objective with a similar
constraint matrix, i.e., if H is the cluster indicator vector for the points, then H T H = I and
H has one non-zero per row. If we relax that constraint to be any orthogonal matrix such
that H T H = I, then we get an objective function, the solution to which is the truncated
SVD; and this provides a 2 approximation to the k-means problem.

132

M. W. Mahoney

13

(03/05/2015): Some Practical Considerations (2 of 4): Basic
perturbation theory and basic dimensionality reduction

Reading for today.
• “A kernel view of the dimensionality reduction of manifolds,” in ICML, by Ham, et al.
Today, we will cover two topics: the Davis-Kahan-sin (θ) theorem, which is a basic result from
matrix perturbation theory that can be used to understand the robustness of spectral clustering in
idealized cases; and basic linear dimensionality reduction methods that, while not spectral graph
methods by themselves, have close connections and are often used with spectral graph methods.

13.1

Basic perturbation theory

One way to analyze spectral graph methods—as well as matrix algorithms much more generally—
is via matrix perturbation theory. Matrix perturbation theory asks: how do the eigenvalues and
eigenvectors of a matrix A change if we add a (small) perturbation E, i.e., if we are working the the
matrix Ã = A + E? Depending on the situation, this can be useful in one or more of several ways.
• Statistically. There is often noise in the input data, and we might want to make claims
about the unobserved processes that generate the observed data. In this case, A might be
the hypothesized data, e.g., that has some nice structure; we observe and are working with
Ã = A + E, where E might be Gaussian noise, Gaussian plus spiked noise, or whatever; and
we want to make claims that algorithms we run on Ã say something about the unobserved A.
• Algorithmically. Here, one has the observed matrix A, and one wants to make claims
about A, but for algorithmic reasons (or other reasons, but typically algorithmic reasons if
randomness is being exploited as a computational resource), one performs random sampling
or random projections and computes on the sample/projection. This amounts to constructing
a sketch Ã = A + E of the full input matrix A, where E is whatever is lost in the construction
of the sketch, and one wants to provide guarantees about A by computing on Ã.
• Numerically. This arises since computers can’t represent real numbers exactly, i.e., there
is round-off error, even if it is at the level of machine precision, and thus it is of interest to
know the sensitivity of problems and/or algorithms to such round-off errors. In this case, A
is the answer that would have been computed in exact arithmetic, while Ã is the answer that
is computed in the presence of round-off error. (E.g., inverting a non-invertible matrix is very
sensitive, but inverting an orthogonal matrix is not, as quantified by the condition number of
the input matrix.)
The usual reference for matrix perturbation theory is the book of Stewart and Sun, which was
written primarily with numerical issues in mind.
Most perturbation theorems say that some notion of distance between eigenstuff, e.g., eigenvalues
of subspaces defined by eigenvectors of A and Ã, depend on the norm of the error/perturbation E,
often times something like a condition number that quantifies the robustness of problems. (E.g., it
is easier to estimate extremal eigenvalues than eigenvectors that are buried deep in the spectrum
of A, and it is easier if E is smaller.)

Lecture Notes on Spectral Graph Methods

133

For spectral graph methods, certain forms of matrix perturbation theory can provide some intuition
and qualitative guidance as to when spectral clustering works. We will cover one such results that
is particularly simple to state and think about. When it works, it works well; but since we are only
going to describe a particular case of it, when it fails, it might fail ungracefully. In some cases,
more sophisticated variants of this result can provide guidance.
When applied to spectral graph methods, matrix perturbation theory is usually used in the following
way. Recall that if a graph has k disconnected components, then 0 = λ1 = λ2 = . . . = λk =< λk+1 ,
and the corresponding eigenvectors v1 , v2 , . . . , vk can be chosen to be the indicator vectors of the
connected components. In this case, the connected components are a reasonable notion of clusters,
and the k-means algorithm should trivially find the correct clustering. If we let A be the Adjacency
Matrix for this graph, then recall that it splits into k pieces. Let’s assume that this is the idealized
unobserved case, and the data that we observe, i.e., the graph we are given or the graph that we
construct is a noisy version of this, call it Ã = A + E, where E is the noise/error. Among other
things E will introduce “cross talk” between the clusters, so they are no longer disconnected. In
this case, if E is small, then we might hope that perturbation theory would show that only the
top k eigenvectors are small, well-separated from the rest, and that the k eigenvectors of Ã are
perturbed versions of the original indicator vectors.
As stated, this is not true, and the main reason for this is that λk+1 (and others) could be very
small. (We saw a version of this before, when we showed that we don’t actually need to compute
the leading eigenvector, but instead any vector whose Rayleigh quotient was similar would give
similar results—where by similar results we mean results on the objective function, as opposed
to the actual clustering.) But, if we account for this, then we can get an interesting perturbation
bound. (While interesting, in the context of spectral graph methods, this bound is somewhat weak,
in the sense that the perturbations are often much larger and the spectral gap are often much larger
than the theorem permits.)
This result is known as the Davis-Kahan theorem; and it is used to bound the distance between the
eigenspaces of symmetric matrices under symmetric perturbations. (We saw before that symmetric
matrices are much “nicer” than general matrices. Fortunately, they are very common in machine
learning and data analysis, even if it means considering correlation matrices XX T or X T X. Note
that if we relaxed this requirement here, then this result would be false, and to get generalizations,
we would have to consider all sorts of other messier things like pseudo-spectra.)
To bound the distance between the eigenspaces, let’s define the notion of an angle (a canonical or
principal angle) between two subspaces.
Definition 42. Let V1 and V2 be two p-dimensional subspaces of Rd , and let V1 and V2 be two
orthogonal matrices (i.e., V1T V1 = I and V2T V2 = I) spanning V1 and V2 . Then the principal
angles {θi }di=1 are s.t. cos(θi ) are the singular values of V1T V2 .
Several things to note. First, for d = 1, this is the usual definition of an angle between two
vectors/lines. Second, one can define angles between subspaces of different dimensions, which is of
interest if there is a chance that the perturbation introduces rank deficiency, but we won’t need
that here. Third, this is actually a full vector of angles, and one could choose the largest to be the
angle between the subspaces, if one wanted.
Definition 43. Let sin (θ (V1 , V2 )) be the diagonal matrix with the sine of the canonical angles
along the diagonal.

134

M. W. Mahoney

Here is the Davis-Kahan-sin (θ) theorem. We won’t prove it.
Theorem 30 (Davis-Kahan). Let A, E ∈ Rn×n be symmetric matrices, and consider Ã = A + E.
Let S1 ⊂ R be an interval; and denote by σS1 (A) the eigenvalues
  of A in S1 , and by V1 the
eigenspace corresponding to those eigenvalues. Ditto for σS1 Ã and Ṽ1 . Define the distance
between the interval S1 and the spectrum of A outside of S1 as
δ = min{kλ − sk : λ is eigenvalue of A, λ ∈
/ S1 , s ∈ S1 }.




Then the distance d V1 , Ṽ1 = k sin θ V1 , Ṽ1 k between the two subspaces V1 and Ṽ1 can be bounded
as
 kEk

,
d V1 , Ṽ1 ≤
δ
where k · k denotes the spectral or Frobenius norm.
What does this result mean? For spectral clustering, let L be the original (symmetric, and SPSD)
hypothesized matrix, with k disjoint clusters, and let L̃ be the perturbed observed matrix. In
addition, we want to choose the interval such that the first k eigenvalues of both L and L̃ are in
it, and so let’s choose the interval as follows. Let S1 = [0, λk ] (where we recall that the first k
eigenvalues of the unperturbed matrix equal 0); in this case, δ = |λk − λk+1 |, i.e., δ equals the
“spectral gap” between the kth and the (k + 1)st eigenvalue.
Thus, the above theorem says that the bound on the distance d between the subspaces defined by
the first k eigenvectors of L and L̃ is less if: (1) the norm of the error matrix kEk is smaller; and
(2) the value of δ, i.e., the spectral gap, is larger. (In particular, note that we need the angle to be
less than 90 degrees to get nontrivial results, which is the usual case; otherwise, rank is lost).
This result provides a useful qualitative guide, and there are some more refined versions of it, but
note the following.
• It is rarely the case that we see a nontrivial eigenvalue gap in real data.
• It is better to have methods that are robust to slow spectral decay. Such methods exist, but
they are more involved in terms of the linear algebra, and so many users of spectral graph
methods avoid them. We won’t cover them here.
• This issue is analogous to what we saw with Cheeger’s Inequality, were we saw that we got
similar bounds on the objective function value for any vector whose Rayleigh quotient was
close to the value of λ2 , but the actual vector might change a lot (since if there is a very small
spectral gap, then permissible vectors might “swing” by 90 degrees).
• BTW, although this invalidates the hypotheses of Theorem 30, the results of spectral algorithms might still be useful, basically since they are used as intermediate steps, i.e., features
for some other task.
That being said, knowing this result is useful since it suggests and explains some of the eigenvalue
heuristics that people do to make vanilla spectral clustering work.
As an example of this, recall the row-wise reweighting we was last time. As a general rule, eigenvectors of orthogonal matrices are robust, but not otherwise in general. Here, this manifests itself

Lecture Notes on Spectral Graph Methods

135

in whether or not the components of an eigenvector on a given component are “bounded away
from zero,” meaning that there is a nontrivial spectral gap. For L and Lrw , the eigenvectors are
indicator vectors, so there is no need to worry about this, since they will be as robust as possible
to perturbation. But for Lsym , the eigenvector is D 1/2~1A , and if there is substantial degree variability then this is a problem, i.e., for low-degree vertices their entries can be very small, and it is
difficult to deal with them under perturbation. So, the row-normalization is designed to robustify
the algorithms.
This “reweigh to robustify” is an after-the-fact justification. One could alternately note that all the
results for degree-homogeneous Cheeger bounds go through to degree-heterogeneous cases, if one
puts in factors of dmax /dmin everywhere. But this leads to much weaker bounds than if one considers
conductance and incorporates this into the sweep cut. I.e., from the perspective of optimization
objectives, the reason to reweigh is to get tighter Cheeger’s Inequality guarantees.

13.2

Linear dimensionality reduction methods

There are a wide range of methods that do the following: construct a graph from the original data;
and then perform computations on the graph to do feature identification, clustering, classification,
regression, etc. on the original data. (We saw one example of this when we constructed a graph,
computed its top k eigenvectors, and then performed k-means on the original data in the space
thereby defined.) These methods are sometimes called non-linear dimensionality reduction methods
since the constructed graphs can be interpreted as so-called kernels and since the resulting methods
can be interpreted as kernel-based machine learning methods. Thus, they indirectly boil down to
computing the SVD—indirectly in that it is in a feature space that is implicitly defined by the
kernel. This general approach is used for many other problems, and so we will describe it in
some detail.
To understand this, we will first need to understand a little bit about linear dimensionality reduction
methods (meaning, basically, those methods that directly boil down to the computing the SVD or
truncated SVD of the input data) as well as kernel-based machine learning methods. Both are large
topics in its own right, and we will only touch the surface.
13.2.1

PCA (Principal components analysis)

Principal components analysis (PCA) is a common method for linear dimensionality that seeks to
find a “maximum variance subspace” to describe the data. In more detail, say we P
are given {xi }ni=1 ,
m
with each xi ∈ R , and let’s assume that the data have been centered in that i xi = 0. Then,
our goal is to find a subspace P , and an embedding ~yi = P ~xi , where P 2 = P , s.t.
1X
||P xi ||2
Var(~y ) =
n
i

is largest, i.e., maximize the projected variance, or where
1X
Err(~y ) =
||xi − P xi ||22
n
i

is smallest, i.e., minimize the reconstruction error. Since Euclidean spaces are so structured,
the solution to these two problems is identical, and is basically given by computing the SVD or
truncated SVD:

136

M. W. Mahoney
• Let C =

1
n

P

T
i xi xi ,

i.e., C ∼ XX T .

• Define the variance as Var(~y ) = Trace(P CP )
P
T
• Do the eigendecomposition to get C = m
i=1 λi êi êi , where λ1 ≥ λ2 ≥ · · · λm ≥ 0.
P
• Let P = di=1 êi êTi , and then project onto this subspace spanning the top d eigenfunctions
of C.
13.2.2

MDS (Multi-Dimensional Scaling)

A different method (that boils down to taking advantage of the same structural result the SVD)
is that of Multi-Dimensional Scaling (MDS), which asks for the subspace that best preserves interpoint distances. In more detail, say we are given {xi }ni=1 , with xi ∈ RD , and let’s assume that the
P
data are centered in that i xi = 0. Then, we have n(n−1)
pairwise distances, denoted ∆ij . The
2
goal is to find vectors ~yi such that:
||~yi − ~yj || ≈ ∆ij
We have the following lemma:
Lemma 19. If ∆ij denotes the Euclidean distance of zero-mean vectors, then the inner products
are
!
X

1 X 2
2
2
2
∆ik + ∆kj − ∆ij −
Gij =
∆kl
2
k

kl

Since the goal is to preserve dot products (which are a proxy for and in some cases related to
distances), we will choose ~
yi to minimize
X
Err(~y ) =
(Gij − ~yi · ~yj )2
ij

The spectral decomposition of G is given as
G=

n
X

λi v̂i v̂iT

i=1

where λ1 ≥ λ2 ≥ · · · λn ≥ 0. In this case, the optimal approximation is given by
p
yiξ = λξ vξi

for ξ = 1, 2, . . . , d, with d ≤ n, which are simply scaled truncated eigenvectors. Thus G ∼ X T X.

13.2.3

Comparison of PCA and MDS

At one level of granularity, PCA and MDS are “the same,” since they both boil down to computing
a low-rank approximation to the original data. It is worth looking at them in a little more detail,
since they come from different motivations and they generalize to non-linear situations in different
ways. In addition, there are a few points worth making as a comparison with some of the graph
partitioning results we discussed.
To compare PCA and MDS:

Lecture Notes on Spectral Graph Methods

137

P
• Cij = n1 k xki xkj is a m × m covariance matrix and takes roughly O((n + d)m2 ) time to
compute.
• Gij = ~xi · ~xj is an n × n Gram matrix and takes roughly O((m + d)n2 ) time to compute.
Here are several things to note:
• PCA computes a low-dimensional representation that most faithfully preserves the covariance
structure, in an “averaged” sense. It minimizes the reconstruction error
EP CA =

X
i

m
X
(xi · eξ )eξ ||22 ,
||xi =
ξ=1

or equivalently it finds a subspace with minimum variance. The basis
Pfor this subspace is
given by the top m eigenvectors of the d × d covariance matrix C = n1 i xi xTi .

• MDS computes a low-dimensional representation of the high-dimensional data that most
faithfully preserve inner products, i.e., that minimizes
EM DS =

X
ij

(xi · xj − φi · φj )2

It does so by computing the Gram matrix of inner products Gij = xi ·xj , so G ≈ X T X. It the
m
top m eigenvectors of p
this are {vi }m
i=1 and the eigenvalues are {λi }i=1 , then the embedding
MDS returns is Φiξ = λξ vξi .

• Although MDS is designed to preserve inner products, it is often motivated to preserve pairwise distances. To see the connection, let
Sij = ||xi − xj ||2
be the matrix of squared inter-point distances. If the points are centered, then a Gram matrix
consistent with these squared distances can be derived from the transformation
G=−
where u =

√1 (1, · · ·
n

, 1).



1
I − uuT S I − uuT
2

Here are several additional things to note with respect to PCA and MDS and kernel methods:
• One can “kernelize” PCA by writing everything in terms of dot products. The proof of
this
is to say that we can “map” the data A to a feature space F with Φ(X). Since C =
1 Pn
T
j=1 φ(xj )φ(xj ) is a covariance matrix, PCA can be computed from solving the eigenvalue
n
problem: Find a λ > 0 and a vector v 6= 0 s.t.
n

1X
(φ(xj ) · v)φ(xj ).
λv = Cv =
n
j=1

(23)

138

M. W. Mahoney
So, all the eigenvectors vi with λP
i must be in the span of the mapped data, i.e., v ∈
n
n
Span{φ(x1 ), . . . , φ(xn )}, i.e., v =
i=1 αi φ(xi ) for some set of coefficients {αi }i=1 . If we
multiply (23) on the left by φ(xk ), then we get
λ(φ(xk ) · v) = (φ(xk ) · Cv),

k = 1, . . . , n.

If we then define
Kij = (φ(xi ), φ(xj )) = k(xi , xj ) ∈ Rn×n ,

(24)

then to compute the eigenvalues we only need
λ~
α = K~
α,

α = (α1 , . . . , αn )T .

Note that we need to normalize (λk , αk ), and we can do so by K̂ = K − 1n K − K1n − 1n K1n .
To extract features of a new pattern φ(x) onto v k , we need
k

(v · φ(x)) =

m
X
i=1

αki φ(xi ) ·

φ(x) =

m
X

αki k(xi , x).

(25)

i=1

So, the nonlinearities enter:
– The calculation of the matrix elements in (24).
– The evaluation of the expression (25).
But, we can just compute eigenvalue problems, and there is no need to go explicitly to the
high-dimensional space. For more details on this, see “An Introduction to Kernel-Based
Learning Algorithms,” by by Müller et al. or “Nonlinear component analysis as a kernel
eigenvalue problem,” by Schölkopf et al.
• Kernel PCA, at least for isotropic kernels K, where K(xi , xj ) = f (||xi − xj ||), is a form of
MDS and vice versa. For more details on this, see “On a Connection between Kernel PCA
and Metric Multidimensional Scaling,” by Williams and “Dimensionality Reduction: A Short
Tutorial,” by Ghodsi. To see this, recall that
– From the distances-squared, {δij }ij , where δij = ||xi − xj ||22 = (xi − xj )T (xi − xj ), we
can construct a matrix A with Aij = − 21 δij .

– Then, we can let B = HAH, where H is a “centering” matrix (H = I − n1 1.1T ). This
can be interpreted as centering, but really it is just a projection matrix (of a form not
unlike we we have seen).
P
– Note that B = HX(HX)T , (and bij = (xi − x̄)T (xj − x̄), with x̄ = n1 i xi ), and thus
B is SPSD.

– In the feature space, δ̃ij is the Euclidean distance:
δ̃ij

= (φ(xi ) − φ(xj ))T (φ(xi ) − φ(xj ))
= ||φ(xi ) − φ(xj )||22

= 2(1 − r(δij )),

where the last line follows since with an isotropic kernel, where k(xi , xj ) = r(δij ). (If
Kij = f (||xi − xj ||), then Kij = r(δij ) (r(0) = 1).) In this case, A is such that Aij =
r(δij ) − 1, A = K − 1.1T , so (fact) HAH = HKH. The centering matrix annihilates
11T , so HAH = HKH.

Lecture Notes on Spectral Graph Methods

139

So, KM DS = − 21 (I − eeT )A(I − eeT ), where A is the matrix of squared distances.
So, the bottom line is that PCA and MDS take the data matrix and use SVD to derive embeddings
from eigenvalues. (In the linear case both PCA and MDS rely on SVD and can be constructed
in O(mn2 ) time (m > n).) They are very similar due to the linear structure and SVD/spectral
theory. If we start doing nonlinear learning methods or adding additional constraints, then these
methods generalize in somewhat different ways.
13.2.4

An aside on kernels and SPSD matrices

The last few comments were about “kernelizing” PCA and MDS. Here, we discuss this kernel issue
somewhat more generally.
Recall that, given a collection X of data points, which are often but not necessarily elements of Rm ,
techniques such as linear Support Vector Machines (SVMs), Gaussian Processes (GPs), Principle
Component Analysis (PCA), and the related Singular Value Decomposition (SVD), identify and
extract structure from X by computing linear functions, i.e., functions in the form of dot products,
of the data. (For example, in PCA the subspace spanned by the first k eigenvectors is used to
give a k dimensional model of the data with minimal residual; thus, it provides a low-dimensional
representation of the data.)
Said another way, these algorithms can be written in such a way that they only “touch” the data
via the correlations between pairs of data points. That is, even if these algorithms are often written
in such as way that they access the actual data vectors, they can be written in such a way that they
only accesses the correlations between pairs of data vectors. In principle, then, given an “oracle”
for a different correlation matrix, one could run the same algorithm by providing correlations from
the oracle, rather than the correlations from the original correlation matrix.
This is of interest essentially since it provides much greater flexibility in possible computations; or,
said another way, it provides much greater flexibility in statistical modeling, without introducing
too much additional computational expense. For example, in some cases, there is some sort of
nonlinear structure in the data; or the data, e.g. text, may not support the basic linear operations
of addition and scalar multiplication. More commonly, one may simply be interested in working
with more flexible statistical models that depend on the data being analyzed, without making
assumptions about the underlying geometry of the hypothesized data.
In these cases, a class of statistical learning algorithms known as kernel-based learning methods have
proved to be quite useful. These methods implicitly map the data into much higher-dimensional
spaces, e.g., even up to certain ∞-dimensional Hilbert spaces, where information about their mutual positions (in the form of inner products) is used for constructing classification, regression, or
clustering rules. There are two points that are important here. First, there is often an efficient
method to compute inner products between very complex or even infinite dimensional vectors. Second, while general ∞-dimensional Hilbert spaces are relatively poorly-structured objects, a certain
class of ∞-dimensional Hilbert spaces known as Reproducing kernel Hilbert spaces (RKHSs) are
sufficiently-heavily regularized that—informally—all of the “nice” behaviors of finite-dimensional
Euclidean spaces still hold. Thus, kernel-based algorithms provide a way to deal with nonlinear
structure by reducing nonlinear algorithms to algorithms that are linear in some (potentially ∞dimensional but heavily regularized) feature space F that is non-linearly related to the original
input space.

140

M. W. Mahoney

The generality of this framework should be emphasized. There are some kernels, e.g., Gaussian rbfs,
polynomials, etc., that might be called a priori kernels, since they take a general form that doesn’t
depend (too) heavily on the data; but there are other kernels that might be called data-dependent
kernels that depend very strongly on the data. In particular, several of the methods to construct
graphs from data that we will discuss next time, e.g., Isomap, local linear embedding, Laplacian
eigenmap, etc., can be interpreted as providing data-dependent kernels. These methods first induce
some sort of local neighborhood structure on the data and then use this local structure to find a
global embedding of the data into a lower dimensional space. The manner in which these different
algorithms use the local information to construct the global embedding is quite different; but in
general they can be interpreted as kernel PCA applied to specially-constructed Gram matrices.
Thus, while they are sometimes described in terms of finding non-linear manifold structure, it is
often more fruitful to think of them as constructing a data-dependent kernel, in which case they
are useful or not depending on issues related to whether kernel methods are useful or whether
mis-specified models are useful.

Lecture Notes on Spectral Graph Methods

14

141

(03/10/2015): Some Practical Considerations (3 of 4): Nonlinear dimension reduction methods

Reading for today.
• “Laplacian Eigenmaps for dimensionality reduction and data representation,” in Neural Computation, by Belkin and Niyogi
• “Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph
partitioning, and data set parameterization,” in IEEE-PAMI, by Lafon and Lee
Today, we will describe several related methods to identify structure in data. The general idea is
to do some sort of “dimensionality reduction” that is more general than just linear structure that
is identified by a straightforward application of the SVD or truncated SVD to the input data. The
connection with what we have been discussing is that these procedures construct graphs from the
data, and then they perform eigenanalysis on those graphs in order to construct “low-dimensional
embeddings” of the data. These (and many other related) methods are often called non-linear
dimension reduction methods. In some special cases they identify structure that is meaningfully
non-linear; but it is best to think of them as constructing graphs to then construct data-dependent
representations of the data that (like other kernel methods) is linear in some nonlinearly transformed
version of the data.

14.1

Some general comments

The general framework for these methods is the following.
• Derive some (typically sparse) graph from the data, e.g., by connecting nearby data points
with an ǫ-NN or k-NN rule.
• Derive a matrix from the graph (viz. adjacency matrix, Laplacian matrix).
• Derive an embedding of the data into Rd using the eigenvectors of that matrix.
Many algorithms fit this general outline. Here are a few things worth noting about them.
• They are not really algorithms (like we have been discussion) in the sense that there is a
well-defined objective function that one is trying to optimize (exactly or approximately) and
for which one is trying to prove running time or quality-of-approximation bounds.
• There exists theorems that say when each of these method “works,” but those theoretical
results have assumptions that tend to be rather strong and/or unrealistic.
• Typically one has some sort of intuition and one shows that the algorithm works on some
data in certain specialized cases. It is often hard to generalize beyond those special cases, and
so it is probably best to think of these as “exploratory data analysis” tools that construct
data-dependent kernels.
• The intuition underlying these methods is often that the data “live” on a low-dimensional
manifold. Manifolds are very general structures; but in this context, it is best to think of
them as being “curved” low-dimensional spaces. The idealized story is that the processes

142

M. W. Mahoney
generating the data have only a few degrees of freedom, that might not correspond to a linear
subspace, and we want to reconstruct or find that low-dimensional manifold.
• The procedures used in constructing these data-driven kernels depend on relatively-simple
algorithmic primitives: shortest path computations; least-squares approximation; SDP optimization; and eigenvalue decomposition. Since these primitives are relatively simple and
well-understood and since they can be run relatively quickly, non-linear dimensionality reduction methods that use them are often used to explore the data.
• This approach is often of interest in semi-supervised learning, where there is a lot of unlabeled
data but very little labeled data, and where we want to use the unlabeled data to construct
some sort of “prior” to help with predictions.

There are a large number of these and they have been reviewed elsewhere; here we will only review
those that will be related to the algorithmic and statistical problems we will return to later.

14.2

ISOMAP

ISOMAP takes as input vectors xi ∈ RD , i = 1, . . . , n, and it gives as output vectors yi ∈ Rd , where
d ≪ D. The stated goal/desire is to make near (resp. far) points stay close (resp. far); and the idea
to achieve this is to preserve geodesic distance along a submanifold. The algorithm uses geodesic
distances in an MDS computation:
I. Step 1. Build the nearest neighbor graph, using k-NN or ǫ-NN. The choice here is a bit of an
art. Typically, one wants to preserve properties such as that the data are connected and/or
that they are thought of as being a discretization of a submanifold. Note that the k-NN here
scales as O(n2 D).
II. Step 2. Look at the shortest path or geodesic distance between all pairs of points. That is,
compute geodesics. Dijkstra’s algorithm for shortest paths runs in O(n2 log n + n2 k) time.
III. Step 3. Do Metric Multi-Dimensional scaling (MDS) based on A, the shortest path distance
matrix. The top k eigenvectors of the Gram matrix then give the embedding. They can be
computed in ≈ O(nd ) time.
Advantages
• Runs in polynomial time.
• There are no local minima.
• It is non-iterative.
• It can be used in an “exploratory” manner.
Disadvantages
• Very sensitive to the choice of ǫ and k, which is an art that is coupled in nontrivial ways with
data pre-processing.

Lecture Notes on Spectral Graph Methods

143

• No immediate “out of sample extension”’ since so there is not obvious geometry to a graph,
unless it is assumed about the original data.
• Super-linear running time—computation with all the data points can be expensive, if the
number of data points is large. A solution is to choose “landmark points,” but for this to
work one needs to have already sampled at very high sampling density, which is often not
realistic.
These strengths and weaknesses are not peculiar to ISOMAP; they are typical of other graph-based
spectral dimensionality-reduction methods (those we turn to next as well as most others).

14.3

Local Linear Embedding (LLE)

For LLE, the input is vectors xi ∈ RD , i = 1, . . . , n; and the output is vectors yi ∈ Rd , with d ≪ D.
Algorithm
Step 1 : Construct the Adjacency Graph There are two common variations:
I. ǫ neighborhood
II. K Nearest neighbor Graph
Basically, this involves doing some sort of NN search; the metric of closeness or similarity
used is based on prior knowledge; and the usual (implicit or explicit) working assumption
is that the neighborhood in the graph ≈ the neighborhood of the underlying hypothesized
manifold, at least in a “local linear” sense.
Step 2 : Choosing weights That is, construct the graph. Weights Wij must be chosen for all
edges ij. The idea is that each input point and its k-NN can be viewed as samples from a small
approximately linear patch on a low-dimensional submanifold and we can choose weights Wij
to get small reconstruction error. That is, weights are chosen based on the projection of each
data point on the linear subspace generated by its neighbors.
P
P
min i ||xi − j Wij xj ||22 = Φ(W )
s.t. Wij = 0 if (ij) 6∈ E
P
j Wij = 1, ∀i

Step 3 : Mapping to Embedded Co-ordinates Compute output y ∈ Rd by solving the same
equations, but now with the yi as variables. That is, let
X
X
Ψ(y) =
||yi −
Wij yj ||2 ,
i

j

for a fixed W , and then we want to solve
min Ψ(y)
P
s.t.
i yi = 0
P
1
T
i yi yi = I
N

To solve this minimization problem reduces to finding eigenvectors corresponding to the d + 1
lowest eigenvalues of the the positive definite matrix (I − W )′ (I − W ) = Ψ.

144

M. W. Mahoney

Of course, the lowest eigenvalue is uninteresting for other reasons, so it is typically not included.
Since we are really computing an embedding, we could keep it if we wanted, but it would not be
useful (it would assign uniform values to every node or values proportional to the degree to every
node) to do downstream tasks people want to do.

14.4

Laplacian Eigenmaps (LE)

For LE, (which we will see has some similarities with LE), the input is vectors xi ∈ RD , i = 1, . . . , n;
and the output is vectors yi ∈ Rd , with d ≪ D. The idea is to compute a low-dimensional
representation that preserves proximity relations (basically, a quadratic penalty on nearby points),
mapping nearby points to nearby points, where “nearness” is encoded in G.
Algorithm
Step 1 : Construct the Adjacency Graph Again, there are two common variations:
I. ǫ-neighborhood
II. k-Nearest Neighbor Graph
Step 2 : Choosing weights We use the following rule to assign weights to neighbors:
(
||x −x ||2
− i 4t j
e
if vertices i & j are connected by an edge
Wij =
0
otherwise
Alternatively, we could simply set Wij = 1 for vertices i and j that are connected by an
edge—it should be obvious that this gives similar results as the above rule under appropriate
limits, basically since the exponential decay introduces a scale that is a “soft” version of this
“hard” 0-1 rule. As a practical matter, there is usually a fair amount of “cooking” to get
things to work, and this is one of the knobs to turn to cook things.
Step 3 : Eigenmaps Let
Ψ(y) =

X wij ||yi − yj ||2
p
Dii · Djj
i,j

P
where D = Diag{ i wij : j = 1(1)n}, and we compute y ∈ Rd such that
min Ψ(y)
P
s.t.
i yi = 0 centered
1 P
and N i yi yiT = I unit covariance

that is, s.t. that is minimized for each connected component of the graph. This can be
computed from the bottom d + 1 eigenvectors of L = I − D −1/2 W D −1/2 , after dropping the
bottom eigenvector (for the reason mentioned above).
LE has close connections to analysis on manifold, and understanding it will shed light on when it
is appropriate to use and what its limitations are.
• Laplacian in Rd : ∆f = −

P

a∂ 2 f
i ∂x2i

Lecture Notes on Spectral Graph Methods

145

• Manifold Laplacian: change is measured along tangent space of the manifold.
The weighted graph ≈ discretized representation of the manifold. There are a number of analogies,
e.g., Stokes Theorem (which classically is a statement about the integration of differential forms
which generalizes popular theorems from vector calculus about integrating over a boundary versus
integration inside the region, and which thus generalizes the fundamental theorem of calculus):
• Manifold:
• Graph:

P

R

M

||∇f ||2 =

ij (fi

R

M

f ∆f

− fj )2 = f T Lf

An extension of LE to so-called Diffusion Maps (which we will get to next time) will provide
additional insight on these connections.
Note that the Laplacian is like a derivative, and so minimizing it will be something like minimizing
the norm of a derivative.

14.5

Interpretation as data-dependent kernels

As we mentioned, these procedures can be viewed as constructing data-dependent kernels. There
are a number of technical issues, mostly having to do with the discrete-to-continuous transition,
that we won’t get into in this brief discussion. This perspective provides light on why they work,
when they work, and when they might not be expected to work.
2 = (x − x )T (x − x ) = dissimilarities. Then A is
• ISOMAP. Recall that for MDS, we have δij
i
j
i
j
a matrix s.t. Aij = −δij , and B = HAH, with H = In − n1 11T , a “centering” or “projection”
2 , as
matrix. So, if the kernel K(xi , xj ) is stationary, i.e., if it is a function of ||xi − xj ||2 = δij
it is in the above construction, then K(xi , xj ) = r(δij ), for some r that scales s.t. r(0) = 1.
2 is the Euclidean distance in feature space, and if A s.t. A
Then δ̃ij
ij = r(δij ) − 1, then
T
T
A = K − 11 . The “centering matrix” H annihilates 11 , so HAH = HKH. (See the
Williams paper.) So,

KISOM AP = HAH = HDH = (I − 11T )D(I − 11T )
where D is the squared geodesic distance matrix.
P
• LE. Recall that LE minimizes ψ T Lψ = 12 ij (psii − ψj )2 Wij , and doing this involved computing eigenvectors of L or L, depending on the construction. The point is that L has close
connections to diffusions on a graph—think of it in terms of a continuous time Markov chain:
∂ψ(t)
= −Lψ(t).
∂t
The solution is a Green’s function or heat kernel, related to the matrix exponential of L:
Kt = exp(−Lt) =

X
ξ

φξ φTξ e−λξ t ,

146

M. W. Mahoney
where φξ , λξ are the eigenvectors/eigenvalues of L. Then, ψ(t) = Kt ψ(0) is the general
solution, and under assumptions one can show that
1
KL = L+
2
is related to the “commute time” distance of diffusion:
+
+
+
Cij ∼ L+
ii + Ljj − Lij = Lji .

For the difference between the commute time in a continuous time Markov chain and the
geodesic distance on the graph, think of the dumbbell example; we will get to this in more
detail next time.
• LLE. Recall that this says that we are approximating each point as a linear combination of
neighbors. Let (Wn )ij , i, j ∈ [n] be the weight of a point xj in the expansion of xi ; then one
can show that

Kn (xi , xj ) = (I − Wn )T (I − Wn ) ij
is PD on the domain X : xi , . . . , xn . Also, one can show that if λ is the largest eigenvalue of
(I − W )T (I − W ) = M , then

KLLE = (λ − 1)I + W T + W − W T W
is a PSD matrix, and thus a kernel. Note also that, under assumptions, we can view M =
(I − W T (I − W ) as L2 .

14.6

Connection to random walks on the graph: more on LE and diffusions

Recall that, give a data set consisting of vectors, we can construct a graph G = (V, E) in one of
several ways. Given that graph, consider the problem of mapping G to a line s.t. connected points
stay as close together as possible.
Let ~y = (y1 , . . . , yn )T be the map, and by a “good” map we will
P
mean one that minimized ij (yi − yj )2 Wij under appropriate constraints, i.e., it will incur a big
penalty if Wij is large and yi and yj are far apart. This has important connections with diffusions
and random walks that we now discuss.
We start with the following claim:
P
Claim 8. 21 i,j wij (yi − yj )2 = y T Ly

Proof. Recall that Wij is symmetric and that Dii =

P

j

Wij . Then:

1X
1X
wij (yi − yj )2 =
wij yi2 + wij yj2 − 2wij yi yj
2
2
i,j
i,j
X
X
=
Dii yi2 −
wij yi yj
i

ij

T

= y Ly

where L = D − W , W is the weight matrix and L is Laplacian, a symmetric, positive semidefinite
matrix that can be thought of as an operator on functions defined on vertices of G.

Lecture Notes on Spectral Graph Methods

147

The following, which we have seen before, is a corollary:
Claim 9. L is SPSD.
Proof. Immediate, since Wij > 0.
Recall that the solution to
arg min y T Ly
s.t.y T Dy = 1,
where D gives a measure of the importance of vertices, turns out to be given by solving a generalized
eigenvector problem
Ly = λDy,
i.e., computing the bottom eigenvectors of that eigenproblem. Actually, we usually solve,
arg min y T Ly
s.t. yD~1 = 0
y T Dy = 1,
the reason being that since ~1 = (1, . . . , 1) is an eigenvector with eigenvalue 0, it is typically removed.
As we saw above, the condition y T D~1 = 0 can be interpreted as removing a “translation invariance”
in y and so is removed. Alternatively, it is uninteresting if these coordinates are going to be used
simple to provide an embedding for other downstream tasks. Also, the condition y T Dy = 1 removes
an arbitrary scaling factor in the embedding, which is a more benign thing to do if we are using
the coordinates as an embedding.
From the previous discussion on Cheeger’s Inequality, we know that putting the graph on a line
(and, in that setting, sweeping to find a good partition) can reveal when the graph doesn’t “look
like” a line graph. Among other things, the distances on the line may be distorted a lot, relative
to the original distances in the graph (even if they are preserved “on average”).
Thus, more generally, the goal is to embed a graph in Rn s.t. distances are “meaningful,” where
meaningful might mean the same as or very similar to distances in the graph. Let G = (V, E, W ).
We know that if W is symmetric, W = W T , and point-wise positive, W (x, y) ≥ 0, then we can
interpret
Ppairwise similarities as probability mass transitions in a Markov chain on a graph. Let
d(x) = z W (x, z) = the degree of node x. Then let P be the n × n matrix with entries
P (x, w) =

W (x, y)
d(x)

which is the transition probability of going from x to y in one step, which equals the first order
neighborhood of the graph. Then, P t is higher order neighborhoods of the graph; this is sometimes
interpreted as ≈ the “intrinsic geometry” of an underlying hypothesized manifold.
If the graph is connected, as it usually is due to data preprocessing, then
lim P t (x, y) = φ0 (y),

t→∞

148

M. W. Mahoney

where φ0 is the unique stationary distribution
d(x)
φ0 = P
z d(z)

of the associated Markov chain. (Note The Markov chain is reversible, as detailed-balance is
satisfied: φ0 (x)P1 (x, y) = φ0 (y)P1 (y, x).)
Thus, graph G defines a random walk. For some node i the probability going from i to j is
P
wij
, where di =
Pij′ =
j wij . Consider if you are at node i and you are move from i in the
di
following way:

move to a neighbor chosen u.a.r, w.p.= 1 w.p. 1
2
di
stay at node i
w.p. 12
1
1
Then the transition matrix P ∈ Rn×m = I + P ′ . This is the so-called “lazy” random walk.
2
2
di
Fact: If G is connected, then for any measure initial v on the vertex, limt→∞ P t v = P
= φ0 (i).
j dj
This φ converges to the stationary distribution. P is related to the normalized Laplacian.
If we look at the pre-asymptotic state, for 1 ≪ t ≪ ∞, we could define similarity between vertex
x and z in terms of the similarity between two density Pt (x, ·) and Pt (z, ·). That is,
• For t ∈ (0, ∞), we want a metric between nodes s.t. x and z are close if Pt (x, ·) and Pt (z, ·)
are close.
• There are various notions of closeness: e.g., ℓ1 norm (see the Szummer and Jaakkola reference); Kullback-Leibler divergences; ℓ2 distances; and so on.
• The ℓ2 distance is what we will focus on here (although we might revisit some of the others
later).
In this setting, the ℓ2 distance is defined as
Dt2 (x, z) = ||Pt (x, ·) − Pt (z, ·)||21/φ0
X (Pt (x, y) − Pt (z, y))2
.
=
φ0 (y)
y
(So, in particular, the weights
high-density domains.)

1
φ0 (y)

(26)

penalize discrepancies on domains of low density more than

This notion of distance between points will be small if they are connected by many path. The
intuition is that if there are many paths, then there is some sort of geometry, redundancy, and
we can do inference like with low-rank approximations. (BTW, it is worth thinking about the
difference between minimum spanning trees and random spanning trees, or degree and spectral
measures of ranking like eigenvector centrality.)

Lecture Notes on Spectral Graph Methods

15

149

(03/12/2015): Some Practical Considerations (4 of 4): More
on diffusions and semi-supervised graph construction

Reading for today.
• “Transductive learning via spectral graph partitioning,” in ICML, by Joachims
• “Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions,” in ICML, by
Zhu, Ghahramani, and Lafferty
• “Learning with local and global consistency,” in NIPS, by Zhou et al.
Today, we will continue talking about the connection between non-linear dimensionality reduction
methods and diffusions and random walks. We will also describe several methods for constructing
graphs that are “semi-supervised,” in the sense that that there are labels associated with some of
the data points, and we will use these labels in the process of constructing the graph. These will
give rise to similar expressions that we saw with the unsupervised methods, although there will be
some important differences.

15.1

Introduction to diffusion-based distances in graph construction

Recall from last time that we have our graph G, and if we run a random walk to the asymptotic
state, then we converge to the leading non-trivial eigenvector. Here, we are interested in looking at
the pre-asymptotic state, i.e., at a time t such that 1 ≪ t ≪ ∞, and we want to define similarity
between vertex x and z, i.e., a metric between nodes x and z, such that x and z are close if Pt (x, ·)
and Pt (z, ·) are close. Although there are other distance notions one could use, we will work with
the ℓ2 distance. In this setting, the ℓ2 distance is defined as
Dt2 (x, z) = ||Pt (x, ·) − Pt (z, ·)||21/φ0
X (Pt (x, y) − Pt (z, y))2
.
=
φ0 (y)
y

(27)

Suppose the transition matrix P has q left and right eigenvectors and eigenvalues |λ0 | ≥ |λ1 | ≥
. . . ≥ |λn | ≥ 0 s.t.
φTj P

= λj φTj

P ψj = λψj ,
where we note that λ0 = 1 and ψ0 = ~1. Normalize s.t. φk ψℓ = δkℓ so that
kφℓ k21/φ0

=

kψℓ k2φ0

=

X φ2 (x)
ℓ

x

X

φ0 (x)

=1

ψℓ2 (x)φ0 (x) = 1,

x

i.e., normalize the left (resp. right) eigenvector of P w.r.t. 1/φ0 (resp. φ0 ). If Pt (x, y) is the kernel
of the tth iterate of P , then we have the following spectral decomposition:
X
Pt (x, y) =
λtj ψj (x)φj (y).
(28)
j

150

M. W. Mahoney

(This is essentially a weighted PCA/SVD of P t , where as usual
P Pthe first k terms provide the “best”
rank-k approximation, where the “best” is w.r.t. kAk2 = x y φ0 (x)A(x, y)2 φ0 (y).) If we insert
Eqn. (28) into Eqn. (27), then one can show that the L2 distance is:
Dt2 (x, z) =

n−1
X

2
λ2t
j (ψj (x) − ψj (z))

(29)

2
λ2t
j (ψj (x) − ψj (z))

(30)

(ψj (x) − ψj (z))2 .

(31)

j=1

≈
≈

k
X
j=1

k
X
j=1

Eqn. (29) says that this provides a legitimate distance, and establishing this is on HW2. The approximation of Eqn. (30) holds if the eigenvalues decay quickly, and the approximation of Eqn. (31)
holds if the large eigenvalues are all roughly the same size. This latter expression is what is provided
by LE.

15.2

More on diffusion-based distances in graph construction

Recall from last time that LE can be extended to include all of the eigenvectors and also to
weighting the embedding by the eigenvalues. This gives rise to an embedding that is based on
diffusions, sometimes it is called Diffusion Maps, that defines a distance in the Euclidean space
that is related to a diffusion-based distance on the graph. In particular, once we choose k in some
way, then here is the picture of the Diffusion Map:

λt1 ψ1 (x)


..
Map Ψt : X → 

.
λtk ψk (x)


and so
Dt2 (x, z) ≈

k
X
j=1

2
λ2t
j (ψj (x) − ψj (z))

≈ ||ψt (x) − ψt (z)||2 .
This is a low-dimensional embedding, where the Euclidean distance in the embedded space corresponds to the “diffusion distance” in the original graph. Here is the picture of the relationships:

G
↔ Rn
|
|
diffusion ↔ || · ||2
Fact: This defines a distance. If you think of a diffusion notion of distance in a graph G, this
identically equals the Euclidean distance || · ||2 between Ψ(i) and Ψ(j). The diffusion notion of

Lecture Notes on Spectral Graph Methods

151

distance is related to the commute time between node i and node j. We will describe this next
time when we talk about resistor networks.
In particular, Laplacian Eigenmaps chooses k = k∗ , for some fixed k∗ and sets t = 0. Under certain
nice limits, this is close to the Laplace-Beltrami operator on the hypothesized manifold. But, more
generally, these results will hold even if the graph is not drawn from such a nice setting.
• None of this discussion assumes that the original vector data come from low-dimensional
manifolds, although one does get a nice interpretation in that case.
• It is fair to ask what happens with spectral decomposition when there is not manifold, e.g.,
a star graph or a constant-degree expander. Many of the ideas wetland about earlier in the
semester would be relevant in this case.

15.3

A simple result connecting random walks to NCUT/conductance

There are connections with graph partitioning, but understanding the connection with random
walkers opens up a lot of other possibilities. Here, we describe one particularly simple result: that
a random walker, starting in the stationary distribution, goes between a set and its complement with
probability that depends on the NCUT of that set. Although this result is simple, understanding
it will be good since it will open up several other possibilities: what if one doesn’t run a random
walk to the asymptotic state; what if one runs a random walk just a few steps starting from an
arbitrary distribution; when does the random walk provide regularization; and so on?
Here, we provide an interpretation for the NCUT objective (and thus related to normalized spectral
clustering as well as conductance): when minimizing NCUT, we are looking for a suit through the
graph s.t. the random walk seldom transitions from A to Ā, where A ⊂ V . (This result says
that conductance/NCUT is not actually looking for good cuts/partitions, but instead should be
interpreted as providing bottlenecks to diffusion.)
Lemma 20. Let P = D −1 W be a random walk transition matrix. Let (Xt )t∈Z+ be the random walk
starting at X0 = π, i.e., starting at the stationary distribution. For disjoint subsets A, B ⊂ V , let
P [B|A] = P [X1 ∈ B|X0 ∈ A]. Then,





N CU T A, Ā = P Ā|A + P A|Ā .
Proof. Observe that
P [X0 ∈ A and X1 ∈ B] =
=

X

P [X0 = i and X1 = j]

i∈A,j∈B

X

πi Pij

i∈A,j∈B

=

X

i∈A,j∈B

=

1
Vol(V )

di Wij
Vol(V ) di
X

i∈A,j∈B

Wij

152

M. W. Mahoney

From this, we have that
P [X0 ∈ A and X1 ∈ B]
P [X0 ∈ A]




X
Vol(A) −1
1


Wij
=
Vol(V )
Vol(V )

P [X1 ∈ B|X0 ∈ A] =

i∈A,j∈B

=

1
Vol(V )

X

Wij

i∈A,j∈B

The lemma follows from this and the definition of NCUT.

15.4

Overview of semi-supervised methods for graph construction

Above, we constructed graphs/Laplacians in an unsupervised manner. That is, there were just
data points that were feature vectors without any classification/regression labels associated with
them; and we constructed graphs from those unlabeled data points by using various NN rules and
optimizing objectives that quantified the idea that nearby data points should be close or smooth in
the graph. The graphs were then used for problems such as data representation and unsupervised
graph clustering that don’t involve classification/regression labels, although in some cases they were
also used for classification or regression problems.
Here, we consider the situation in which some of the data have labels and we want to construct
graphs to help make predictions for the unlabeled data. In between the extremes of pure unsupervised learning and pure supervised learning, there are semi-supervised learning, transductive
learning, and several other related classes of machine learning methods. It is in this intermediate
regime where using labels for graph construction is most interesting.
Before proceeding, here are a few things to note.
• If unlabeled data and labeled data come from the same distribution, then there is no difference
between unlabeled data and test data. Thus, this transductive setup amounts to being able
to see the test data (but not the labels) before doing training. (Basically, the transductive
learner can look at all of the data, including the test data with labels and as much training
data as desired, to structure the hypothesis space.)
• People have used labels to augment the graph construction process in an explicit manner.
Below, we will describe an example of how this is done implicitly in several cases.
• We will see that linear equations of a certain form that involve Laplacian matrices will arise.
This form is rather natural, and it has strong similarities with what we saw previously in the
unsupervised setting. In addition, it also has strong similarities with local spectral methods
and locally-biased spectral methods that we will get to in a few weeks.
We will consider three approaches (those of Joachims, ZGL, and Zhao et al.). Each of these general
approaches considers constructing an augmented graph, with extra nodes, s and t, that connect
to the nodes that have labels. Each can be understood within the following framework. Let

Lecture Notes on Spectral Graph Methods

153

L = D − A = B T CB, where B is the unweighted edge-incidence matrix. Recall, then, that the
s, t-mincut problem is:
X
min
kBxkC,1 =
min
C(u,v) |xu − xv |.
x s.t. xs =1,xt =0
x s.t. xs =1,xt =0
(u,v)∈E
The ℓ2 minorant of this s, t-mincut problem is:

x

s.t.

min

xs =1,xt =0

kBxkC,2 =

x

This latter problem s equivalent to:

x



X

1/2


C(u,v) |xu − xv |2 
min
s.t. s=1,xt =0 (u,v)∈E

.

X
1
C(u,v) |xu − xv |2 =
min
xT Lx.
kBxk2C,2 =
min
2
s.t. xs =1,xt =0
x s.t. xs =1,xt =0
x s.t. xs =1,xt =0
(u,v)∈E
min

The methods will construct Laplacian-based expressions by considering various types of s-t mincut
problems and then relaxing to the associated ℓ2 minorant.

15.5

Three examples of semi-supervised graph construction methods

The Joachims paper does a bunch of things, e.g., several engineering heuristics that are difficult
to relate to a general setting but that no doubt help the results in practice, but the following is
essentially what he does. Given a graph and labels, he wants to find a vector ~y s.t. it satisfies the
bicriteria:
I. it minimizes y T Ly, and
II. it has values

(

1
−1

for nodes in class j
for nodes not in class j

What he does is essentially the following. Given G = (V, E), he adds extra nodes to the node set:
(
s with the current class
t with the other class
Then, there is the issue about what to add as weights when those nodes connect to nodes in V .
The two obvious weights are 1 (i.e., uniform) or equal to the degree. Of course, other choices are
possible, and that is a parameter one could play with. Here, we will consider the uniform case.
Motivated by problem with mincut (basically, what we described before, where it tends to cut off
small pieces), Joachims instead considers NCUT and so arrives at the following problem:
Y = (DS + L)−1 S,
where DS is a diagonal matrix with the row sums of S on the diagonal, and S is a vector corresponding to the class connected to the node with label s. Note that since most of the rows of S

154

M. W. Mahoney

equal zero (since they are unlabeled nodes) and the rest have only 1 nonzero, DS is a diagonal
indicator vector that is sparse.
ZGL is similar to Joachims, except that they strictly enforce labeling on the labeled samples. Their
setting is that they are interpreting this in terms of a Gaussian random field on the graph (which
is like a NN method, except that nearest labels are captures in terms of random vectors on G).
They provide hard constraints on class labels. In particular, they want to solve
min
y

s.t.

1 T
y Ly
2

 1
yi =
0

free

node i is labeled in class j
node i is labeled in another class
otherwise

This essentially involves constructing a new graph from G ,where the labels involve adding extra
nodes, s and t, where s links to the current class and t links to the other class. For ZGL, weights
= ∞ between s and the current class as well as between t and the other class.
Zhao, et al. extend this to account explicitly for the bicriteria that:
I. nearby points should have the same labels; and
II. points on the same structure (clusters of manifold) have the same label.
Note that Point 1 is a “local” condition, in that it depends on nearby data points, while Point 2
is a “global” condition, in that it depends on large-scale properties of clusters. The point out that
different semi-supervised methods take into account these two different properties and weight them
in different ways. Zhao et al. try to take both into account by “iteratively” spreading ZGL to get
a classification function that has local and global consistency properties and is sufficiently smooth
with respect to labeled and unlabeled data.
In more detail, here is the Zhao et al. algorithm.
I. Form the affinity matrix W from an rbf kernel.
II. Construct W̃ = D −1/2 W D −1/2 .
III. Iterate Y (t + 1) = αW̃ Y + (1 − α) S, where α ∈ (0, 1) is a parameter, and where S is a class
label vector.
IV. Let Y ∗ be the limit, and output it.


Note that Ỹ ∗ = (1 − α) 1 − αW̃ S. Alternatively,

Y = (L + αD)−1 S = (D − βA)−1 S,

where Y is the prediction vector, and where S is the matrix of labels. Here, we have chosen/defined
α = 1−β
β to relate the two forms.
Then the “mincut graph” has G = (V, E), with nodes s and t such that
(
s connects to each sample labeled with class j with weight α
t connects to all nodes in G (except s) with weight α(di − si )

Lecture Notes on Spectral Graph Methods

155

The ℓ2 minorant of this mincut graph has:

x

1
kBxk2C,2 =
s.t. xs =1,xt =0 2
x
min

T 
1
αeT s
−αsT
1  
min
y
−αs
αD + L
s.t. xs =1,xt =0 2
0
0
−α(d − s)T


In this case, y solves (αD + L) y = αs.

 
0
1


α(d − s)
y .
T
αe (d − s)
0

We will see an equation of this form below in a somewhat different context. But for the semisupervised learning context, we can interpret this as a class-specific smoothness constraint. To do
so, define


n
n
X
X
1
1
1
kYi − Si k2 
Yj k2 − µ
Yi − p
Wij k √
A(Y ) = 
2
Dii
Djj
j=1
ij=1
to be the “cost” associated with the prediction Y . The first term is a “smoothness constraint,”
which is the sum of local variations, and it reflects that a good classification should not change
much between nearby points. The second term is a “fitting constraint,” and it says that a good
classification function should not change much from the initial assignment provided by the labeled
data. The parameter µ givers the interaction between the two terms. To see how this gives rise to
the previous expression, observe that
∂Q(Y )
|Y =Y ∗ = Y ∗ − W Y ∗ + µ (Y ∗ − S) = 0,
∂Y
from which it follows that
Y∗−
If we define α =

1
1+µ

and β =

µ
1+µ ,

µ
1
WY ∗ −
S = 0.
1+µ
1+µ

so that α + β = 1, then we have that
(I − αW ) Y ∗ = βS,

and thus that

Y ∗ = β (1 − αW )−1 S.

156

M. W. Mahoney

16

(03/17/2015): Modeling graphs with electrical networks

Reading for today.
• “Random Walks and Electric Networks,” in arXiv, by Doyle and Snell

16.1

Electrical network approach to graphs

So far, we have been adopting the usual approach to spectral graph theory: understand graphs via
the eigenvectors and eigenvalues of associated matrices. For example, given a graph G = (V, E), we
defined an adjacency matrix A and considered the eigensystem Av = λv, and wePalso defined the
Laplacian matrix L = D−A and considered the Laplacian quadratic form xT Lx− (ij)∈E (xi −xj )2 .
There are other ways to think about spectral graph methods that, while related, are different in
important ways. In particular, one can draw from physical intuition and define physical-based
models from the graph G, and one can also consider more directly vectors that are obtained from
various diffusions and random walks on G. We will do the former today, and we will do the latter
next time.

16.2

A physical model for a graph

In many physical systems, one has the idea that there is an equilibrium state and that the system
goes back to that equilibrium state when disturbed. When the system is very near equilibrium, the
force pushing it back to the equilibrium state is quadratic in the displacement from equilibrium,
one can often define a potential energy that in linear in the displacement from equilibrium, and
then the equilibrium state is the minimum of that potential energy function.
In this context, let’s think about the edges of a graph G = (V, E) as physical “springs,” in which
case the weights on the edges correspond to a spring constant k. Then, the force, as a function
of the displacement x from equilibrium, is F (x) = kx, and the corresponding potential energy is
U (x) = 12 kx2 . In this case, i.e., if the graph is viewed as a spring network, then if we nail down
some of the vertices and then let the rest settle to an equilibrium position, then we are interested
in finding the minimum of the potential energy
X
(xi − xj )2 = xT Lx,
(ij)∈E

subject to the constraints on the nodes we have nailed down. In this case, the energy is minimized
when the non-fixed vertices have values equal to
xi =

1 X
xj ,
di
(ij)∈E

i.e., when the value on any node equals is the average of the values on its neighbors. (This is the
so-called harmonic property which is very important, e.g., in harmonic analysis.)
As we have mentioned previously and will go into in more detail below, eigenvectors can be unstable
things, and having some physical intuition can only help; so let’s go a little deeper into these
connections.

Lecture Notes on Spectral Graph Methods

157

First, recall that the standard/weighted geodesic graph metric defines a distance d(a, b) between
vertices a and b as the length of the minimum-length path, i.e., number of edges or the sum of
weights over edges, on the minimum-length path connecting a and b. (This is the “usual” notion
of distance/metric on the nodes of a graph, but it will be different than distances/metrics implied
by spectral methods and by what we will discuss today.)
Here, we will model a graph G = (V, E) as an electrical circuit. (By this, we mean a circuit that
arises in electromagnetism and electrical engineering.) This will allow us to use physical analogues,
and it will allow us to get more robust proofs for several results. In addition, it allow us to define
another notion of distance that is closer to diffusions.
As background, here are some physical facts from electromagnetism that we would like to mimic
and that we would like our model to incorporate.
• A basic direct current electrical circuit consists of a battery and one or more circuit elements
connected by wires. Although there are other circuit elements that are possible, here we will
only consider the use of resistors. A battery consists of two distinct vertices, call them {a, b},
one of which is the source, the other of which is the sink. (Although we use the same terms,
“source” and “sink,” as we used with flow-based methods, the sources and since here will obey
different rules.) A resistor between two points a and b, i.e., between two nodes in G, has
an associated (undirected and symmetric) quantity rab called a resistance (and an associated
conductance cab = r1ab ). Also, there is a current Yab and a potential difference Vab between
nodes a and b.
Initially, we can define the resistance between two nodes that are connected by an edge to depend
(typically inversely) on the weight of that edge, but we want to extend the idea of resistance to
a resistance between any two nodes. To do so, an important notion is that of effective resistance,
which is the following. Given a collection of resistors between nodes a and b, they can be replaced
with a single effective resistor with some other resistance. Here is how the value of that effective
resistance is determined.
• If a and b have a node c between them, i.e., the resistors are in series, and there are resistances
r1 = Rac and r2 = Rcb , then the effective resistance between a and b is given by Rab = r1 + r2 .
• If a and b have no nodes between them but they are connected by two edges with resistances
r1 and r2 , i.e., the resistors are in parallel, then the effective resistance between a and b is
given by Rab = 1 +1 1 .
r1

r2

• These rules can be applied recursively.
From this it should be clear that the number of paths as well as their lengths contribute to the
effective resistance. In particular, having k parallel edges/paths leads to an effective resistance that
is decreased by k1 ; and adding the first additional edge between two nodes has a big impact on the
effective resistance, but subsequent edges have less of an effect. Note that this is vaguely similar to
the way diffusions and random walks behave, and distances/metrics they might imply, as opposed
to geodesic paths/distances defined above, but there is no formal connection (yet!).
Let a voltage source be connected between vertices a and b, and let Y > 0 be the net current out
of source a and into course b. Here we define two basic rules that our resistor networks must obey.

158

M. W. Mahoney

Definition 44. The Kirchhoff current law states that the current Yij between vertices i and j
(where Yij = −Yji ) satisfies

 Y
Yij =
−Y

free
j∈N (i)
X

i=a
i=b
,
otherwise

where N (i) refers to the nodes that are neighbors of node i.
Definition 45. The Kirchhoff circuit/potential law states that for every cycle C in the network,
X

Yij Rij = 0.

(ij)∈C

From Definition 45, it follows that there is a so-called potential function on the vertices/nodes of
the graph. This is known as Ohm’s Law.
Definition 46. Ohm’s Law states that, to any vertex i in the vertex set of G, there is an associated
potential, call it Vi , such that for all edges (ij) ∈ E in the graph
Yij Rij = Vi − Vj .
Given this potential function, we can define the effective resistance between any two nodes in G,
i.e., between two nodes that are not necessarily connected by an edge.
Definition 47. Given two nodes, a and b, in G, the effective resistance is Rab =

Va −Vb
.
Y

Fact. Given a graph G with edge resistances Rij , and given some source-sink pair (a, b), the
effective resistance exists, it is unique, and (although we have defined it in terms of a current) it
does not depend on the net current.

16.3

Some properties of resistor networks

Although we have started with this physical motivation, there is a close connection between resistor
networks and what we have been discussing so far this semester.
To see this, let’s start with the following definition, which is a special case of the Moore-Penrose
pseudoinverse.
Definition 48. The Laplacian pseudoinverse is the unique matrix satisfying:
I. L+~1 = 0; and
II. For all w ⊥ ~1 :

L+ w = v s.t. Lv = w and v ⊥ ~1 .

Given this, we have the following theorem. Note that here we take the resistances on edges to be
the inverse of the weights on those edges, which is probably the most common choice.

Lecture Notes on Spectral Graph Methods

159

Theorem 31. Assume that the resistances of the edges of G = (V, E) are given by Rij =
Then, the effective resistance between any two nodes a and b is given by:

1
wij .

Rab = (ea − eb )T L+ (ea − eb )
+
+
= L+
aa − 2Lab + Lbb .

Proof. The idea of the proof is that, given a graph, edge resistances, and net current, there always
exists currents Y and potentials V satisfying Kirchhoff’s current and potential laws; in addition,
the vector of potentials is unique up to a constant, and the currents are unique. I’ll omit the details
of this since it is part of HW2.
Since the effective resistance between any two nodes is well-defined, we can define the total effective
resistance of the graph. (This is sometimes called the Kirchhoff index.)
P
Definition 49. The total effective resistance is Rtot = nij=1 Rij .

Before proceeding, think for a minute about why one might be interested in such a thing. Below,
we will show that the effective resistance is a distance; and so the total effective resistance is the
sum of the distances between all pairs of points in the metric space. Informally, this can be used
to measure the total “size” or “capacity” of a graph. We used a similar thing (but for the geodesic
distance) when we showed that expander graphs had a Θ (log(n)) duality gap. In that case, we did
this, essentially, by exploiting the fact that there was a lot of flow to route and since most pairs of
nodes were distance Θ (log(n)) apart in the geodesic distance.
The quantity Rtot can be expressed exactly in terms of the Laplacian eigenvalues (all of them, and
not just the first one or first few). Here is the theorem (that we won’t prove).
P
Theorem 32. Let λi be the Laplacian eigenvalues. Then, Rtot = n ni=1 λ1i .
Of course, we can get a (weak) bound on Rtot using just the leading nontrivial Laplacian eigenvalue.

Corollary 1.
n
n(n − 1)
≤ Rtot ≤
λ2
λ2
Next, we show that the effective resistance is a distance function. For this reason, it is sometimes
called the resistance distance.
Theorem 33. The effective resistance R is a metric.
Proof. We will establish the three properties of a metric.
First, from the above theorem, Rij = 0 ⇐⇒ i = j. The reason for this is since ei − ej is in
the null space of L+ (which is the span(~1)) iff i = j. Since the pseudoinverse of L has eigenvalues
−1
0, λ−1
2 , . . . , λn , it is PSD, and so Rij ≥ 0.
Second, since the pseudoinverse is symmetric, we have that Rij = Rji .
So, the only nontrivial thing is to show the triangle inequality holds.
To do so, we show two claims.

160

M. W. Mahoney


 1
Claim 10. Let Yab be the vector ea − eb =
−1

0
Vab (a) ≥ Vab (c) ≥ Vab (b), for all c.

at a
at b
elsewhere,

and let Vab = L+ Yab . Then,

Proof. Recall that Vab is the induced potential when we have 1 Amp going in a and
P 1 Amp coming
out of b. For every vertex c, other than a and b, the total flow is 0, which means x∼c R1xc (Vab (x) −
P

C

V (x)

P xc ab
where Cxc = R1xc is the conductance between
Vab (c)) = 0, and it is easy to see Vab (c) = x∼c
x∼c Cxc
x, c. Vab (c) has a value equal to the weighted average of values of Vab (x) at its neighbors. We can
use this to prove the claim by contradiction. Assume that there exists a c s.t. Vab (c) > Vab (a). If
there are several such nodes, then let c be the node s.t. Vab (c) is the largest. In this case, Vab (c) is
larger than the values at its neighbors. This is a contradiction, since Vc is a weighted average of the
potentials at its neighbors. The proof of the other half of the claim is similar. (also Vab (a) ≥ Vab (b)
as Vab (a) − Vab (b) = Rab ≥ 0)

Claim 11. Ref f (a, b) + Ref f (b, c) ≥ Ref f (a, c)
Proof. Let Yab and Ybc be the external current from sending one unit of current from a → b and
b → c, respectively. Note that Yac = Yab + Ybc . Define the voltages Vab = L+ Yab , Vbc = L+ Ybc , and
Vac = L+ Yac . By linearity, Vac = Vab + Vbc . Thus, it follows that
T
T
T
Ref f (a, c) = Yac
Vac = Yac
Vab + Yac
Vbc .

By Claim 10, it follows that
T
Yac
Vab = Vab (a) − Vab (c) ≤ Vab (a) − Vab (b) = Ref f (a, b).
TV ≤ R
Similarly, Yac
bc
ef f (b, c). This establishes the claim.

The theorem follows from these two claims.
Here are some things to note regarding the resistance distance.
• Ref f is non-increasing function of edge weights.
• Ref f does not increase when edges are added.
• Rtot strictly decreases when edges are added and weights are increased.
Note that these observations are essentially claims about the distance properties of two graphs, call
them G and G′ , when one graph is constructed from the other graph by making changes to one or
more edges.
We have said that both geodesic distances and resistances distances are legitimate notions of distances between the nodes on a graph. One might wonder about the relationship between them. In
the same way that there are different norms for vectors in Rn , e.g., the ℓ1 , ℓ2 , and ℓ∞ , and those
norms have characteristic sizes with respect to each other, so too we can talk about the relative
sizes of different distances on nodes of a graph. Here is a theorem relating the resistance distance
with the geodesic distance.

Lecture Notes on Spectral Graph Methods

161

Theorem 34. For Ref f and the geodesic distance d:
I. Ref f (a, b) = d(a, b) iff there exists only one path between a and b.
II. Ref f (a, b) < d(a, b) otherwise.
Proof. If there is only one path P between a and b, then Yij = Y , for all ij on this path (by
Kirchhoff current law), and Vi − Vj = Y Rij . It follows that
X Vi − Vi
X
Va − Vb
Rab =
=
=
Vij = dab .
Y
Y
(ij)∈P

(ij)∈P

If a path between a and b is added, so that now there are multiple paths between a and b, this new
path might use part of the path P . If it does, then call that part of the path P1 ; consider the rest
of P , and call the shorter of these P2 and the larger P3 .
Observe that the current through each edge of P1 is Y ; and, in addition, that the current through
each edge of P2 and P3 is the same for each edge in the path, call them Y2 and Y3 , respectively.
Due to Kirchhoff current law and Kirchhoff circuit/potential law, we have that Y2 + Y3 = Y and
also that Y2 , Y3 > 0, from which it follows that Y2 < Y . Finally,
Rab =
=

Va − Vb
Y
X Vi − Vj
X Vi − Vj
+
Y
Y
(ij)∈P2

(ij)∈P1

X Vi − Vj
X Vi − Vj
=
+
Y
Y2
(ij)∈P1
(ij)∈P2
X
X
Rij
Rij +
=
(ij)∈P1

(ij)∈P2

= d(a, b)

The result follows since Ref f doesn’t increase when edges are added.
In a graph that is a tree, there is a unique path between any two vertices, and so we have the
following result.
Claim 12. The metrics Ref f and d are the same in a tree. That is, on a tree, Ref f (a, b) = d(a, b),
for all nodes a and b.
Fact. Ref f can be used to bound several quantities of interest, in particular the commute time,
the cover time, etc. We won’t go into detail on this.
Here is how Ref f behaves in some simple examples.
tot : Rtot (K ) = n − 1.
• Complete graph Kn . Of all graphs, this has the minimum Ref
n
ef f
f
tot : Rtot (P ) =
• Path graph Pn . Among connected graphs, the path graph has the maximum Ref
n
ef f
f
1
6 (n − 1)n(n + 1).
tot : Rtot (S ) = (n − 1)2 .
• Star Sn . Among trees, this has the minimum Ref
n
ef f
f

162

M. W. Mahoney

16.4

Extensions to infinite graphs

All of what we have been describing so far is for finite graphs. Many problems of interest have to do
with infinite graphs. Perhaps the most basic is whether random walks are recurrent. In addition to
being of interest in its own right, considering this question on infinite graphs should provide some
intuition for how random walked based spectral methods perform on the finite graphs we have been
considering.
Definition 50. A random walk is recurrent if the walker passes through every point with probability
1, or equivalently if the walker returns to the starting point with probability 1. Otherwise, the random
walk is transient.
Note that—if we were to be precise—then we would have to define this for a single node, be precise
about which of those two notions we are considering, etc. It turns out that those two notions are
equivalent and that a random walk is recurrent for one node iff it is recurrent for any node in the
graphs. We’ll not go into these details here.
For irreducible, aperiodic random walks on finite graphs, this discussion is of less interest, since
a random walk will eventually touch every node with probability proportional to its degree; but
consider three of the simplest infinite graphs: Z, Z2 , and Z3 . Informally, as the dimension increases,
there are more neighbors for each node and more space to get lost in, and so it should be harder
to return to the starting node. Making this precise, i.e., proving whether a random walk on these
graphs is recurrent is a standard problem, one version of which appears on HW2.
The basic idea for this that you need to use is to use something called Rayleigh’s Monotonicity Law
as well as the procedures of shorting and cutting. Rayleigh’s Monotonicity Law is a version of the
result we described before, which says that Ref f between two points a and b varies monotonically
with individual resistances. Then, given this, one can use this to do two things to a graph G:
• Shorting vertices u and v: this is “electrical vertex identification.”
• Cutting edges between u and v: this is “electrical edge deletion.”
Both of these procedures involve constructing a new graph G′ from the original graph G (so that
we can analyze G′ and make claims about G). Here are the things you need to know about shorting
and cutting:
• Shorting a network can only decrease Ref f .
• Cutting a network can only increase Ref f .
For Z2 , if you short in “Manhattan circles” around the origin, then this only decreases Ref f , and
you can show that Ref f = ∞ on the shorted graph, and thus Ref f = ∞ on the original Z2 . For Z3 ,
if you cut in a rather complex way, then you can show that Ref f < ∞ on the cut graph, meaning
that Ref f < ∞ on the original Z3 . This, coupled with the following theorem, establish the result
random walks on Z2 are recurrent, but random walks on Z3 are transient.
Theorem 35. A network is recurrent iff Ref f = ∞.
Using these ideas to prove the recurrence claims is left for HW2: getting the result for Z is straightforward; getting it for Z2 is more involved but should be possible; and getting it for Z3 is fairly
tricky—look it up on the web, but it is left as extra credit.

Lecture Notes on Spectral Graph Methods

17

163

(03/19/2015): Diffusions and Random Walks as Robust Eigenvectors

Reading for today.
• “Implementing regularization implicitly via approximate eigenvector computation,” in ICML,
by Mahoney and Orecchia
• “Regularized Laplacian Estimation and Fast Eigenvector Approximation,” in NIPS, by Perry
and Mahoney
Last time, we talked about electrical networks, and we saw that we could reproduce some of the
things we have been doing with spectral methods with more physically intuitive techniques. These
methods are of interest since they are typically more robust than using eigenvectors and they often
lead to simpler proofs. Today, we will go into more detail about a similar idea, namely whether we
can interpret random walks and diffusions as providing robust or regularized or stable analogues
of eigenvectors. Many of the most interesting recent results in spectral graph methods adopt this
approach of using diffusions and random walks rather than eigenvectors. We will only touch on the
surface of this approach.

17.1

Overview of this approach

There are several advantages to thinking about diffusions and random walks as providing a robust
alternative to eigenvectors.
• New insight into spectral graph methods.
• Robustness/stability is a good thing in many situations.
• Extend global spectral methods to local spectral analogues.
• Design new algorithms, e.g., for Laplacian solvers.
• Explain why diffusion-based heuristics work as they do in social network, computer vision,
machine learning, and many other applications.
Before getting into this, step back for a moment, and recall that spectral methods have many nice
theoretical and practical properties.
• Practically. Efficient to implement; can exploit very efficient linear algebra routines; perform
very well in practice, in many cases better than theory would suggest. (This last claim means,
e.g., that there is an intuition in areas such as computer vision and social network analysis
that even if you could solve the best expansion/conductance problem, you wouldn’t want to,
basically since the approximate solution that spectral methods provide is “better.”)
• Theoretically. Connections between spectral and combinatorial ideas; and connections between Markov chains and probability theory that provides a geometric viewpoint.
Recently, there have been very fast algorithms that combine spectral and combinatorial ideas.
They rely on an optimization framework, e.g., solve max flow problems by relating them to these

164

M. W. Mahoney

spectral-based optimization ideas. These use diffusion-based ideas, which are a relatively new trend
in spectral graph theory.
To understand better this new trend, recall that the classical view of spectral methods is based on
Cheeger’s Inequality and involves computing an eigenvector and performing sweep cuts to reveal
sparse cuts/partitions. The new trend is to replace eigenvectors with vectors obtained by running
random walks. This has been used in:
• fast algorithms for graph partitioning and related problems;
• local spectral graph partitioning algorithms; and
• analysis of real social and information networks.
There are several different types of random walks, e.g., Heat Kernel, PageRank, etc., and different
walks are better in different situations.
So, one question is: Why and how do random walks arise naturally from an optimization framework?
One advantage of a random walk is that to compute an eigenvector in a very large graph, a vanilla
application of the power method or other related iterative methods (especially black box linear
algebra methods) might be too slow, and so instead one might run a random walk on the graph to
get a quick approximation.
Let W = AD −1 be the natural random walk matrix, and let L = D − A be the Laplacian. As we
have discussed, it is well-known that the second eigenvector of the Laplacian can be computed by
iterating W .
• For “any” vector y0 (or “any” vector y0 s.t. y0 D −1~1 = 0 or any random vector y0 s.t.
y0 D −1~1 = 0), we can compute D −1 W t y0 ; and we can take the limit as t → ∞ to get
v2 (L) = lim
t→

D −1 W t y0
,
kW t y0 kD−1

where v2 (L) is the leading nontrivial eigenvector of the Laplacian.
• If time is a precious resource, then one alternative is to avoid iterating to convergence, i.e.,
don’t let t → ∞ (which of course one never does in practice, but by this we mean don’t
iterate to anywhere near machine precision), but instead do some sort of “early stopping.” In
that case, one does not obtain an eigenvector, but it is of interest to say something about the
vector that is computed. In many cases, this is useful, either as an approximate eigenvector
or as a locally-biased analogue of the leading eigenvector. This is very common in practice,
and we will look at it in theory.
Another nice aspect of replacing an eigenvector with a random walk or by truncating the power
iteration early is that the vectors that are thereby returned are more robust. The idea should be
familiar to statisticians and machine learners, although in a somewhat different form. Say that
there is a “ground truth” graph that we want to understand but that the measurement we make,
i.e., the graph that we actually see and that we have available to compute with, is a noisy version of
this ground truth graph. So, if we want to compute the leading nontrivial eigenvector of the unseen
graph, then computing the leading nontrivial eigenvector of the observed graph is in general not a

Lecture Notes on Spectral Graph Methods

165

particularly good idea. The reason is that it can be very sensitive to noise, e.g., mistakes or noise
in the edges. On the other hand, if we perform a random walk and keep the random walk vector,
then that is a better estimate of the ground truth eigendirection. (So, the idea is that eigenvectors
are unstable but random walks are not unstable.)
A different but related question is the following: why are random walks useful in the design of fast
algorithms? (After all, there is no “ground truth” model in this case—we are simply running an
algorithm on the graph that is given, and we want to prove results about the algorithm applied
to that graph.) The reason is similar, but the motivation is different. If we want to have a fast
iterative algorithm, then we want to work with objects that are stable, basically so that we can
track the progress of the algorithm. Working with vectors that are the output of random walks will
be better in this sense. Today, we will cover an optimization perspective on this. (We won’t cover
the many applications of these ideas to graph partitioning and related algorithmic problems.)

17.2

Regularization, robustness, and instability of linear optimization

Again, take a step back. What is regularization? The usual way it is described (at least in machine
learning and data analysis) is the following.
We have an optimization problem
min f (x),
x∈S

where f (x) is a (penalty) function, and where S is some constraint set. This problem might not
be particularly well-posed or well-conditioned, in the sense that the solution might change a lot
if the input is changed a little. In order to get a more well-behaved version of the optimization
problem, e.g., one whose solution changes more gradually as problem parameters are varied, one
might instead try to solve the problem
min f (x) + λg(x),
x∈S

where λ ∈ R+ is a parameter, and where g(x) is (regularization) function. The idea is that g(x)
is “nice” in some way, e.g., it is convex or smooth, and λ governs the relative importance of
the two terms, f (x) and g(x). Depending on the specific situation, the advantage of solving the
latter optimization problem is that one obtains a more stable optimum, a unique optimum, or
smoothness conditions. More generally, the benefits of including such a regularization function in
ML and statistics is that: one obtains increased stability; one obtains decreased sensitivity to noise;
and one can avoid overfitting.
Here is an illustration of the instability of eigenvectors. Say that we have a graph G that is basically
an expander except that it is connected to two small poorly-connected components. That is, each
of the two components is well-connected internally but poorly-connected to the rest of G, e.g.,
connected by a single edge. One can easily choose the edges/weights in G so that the leading
non-trivial eigenvector of G has most of its mass, say a 1 − ǫ fraction of its mass, on the first
small component. In addition, one can then easily construct a perturbation, e.g., removing one
edge from G to construct a graph G′ such that G′ has a 1 − ǫ fraction of its mass on the second
component. That is, a small perturbation that consists of removing one edge can completely shift
the eigenvector—not only its direction but also where in Rn its mass is supported.

166

M. W. Mahoney

Let’s emphasize that last point. Recalling our discussion of the Davis-Kahan theorem, as well as the
distinction between the Rayleigh quotient objective and the actual partition found by performing a
sweep cut, we know that if there is a small spectral gap, then eigenvectors can swing by 90 degrees.
Although the example just provided has aspects of that, this example here is even more sensitive:
not only does the direction of the vector change in Rn , but also the mass along the coordinate
axes in Rn where the eigenvector is localized changes dramatically under a very minor perturbation
to G.
To understand this phenomenon better, here is the usual quadratic optimization formulation of the
leading eigenvector problem. For simplicity, let’s consider a d-regular graph, in which case we get
the following.
Quadratic Formulation :

1
d

minx∈Rn xT Lx
s.t.

(32)

kxk = 1
x ⊥ ~1.

This is an optimization over vectors x ∈ Rn . Alternatively, we can consider the following optimization problem over SPSD matrices.
SDP Formulation :

1
d

minX∈Rn×n
s.t.

L•X

(33)

I •X =1

J •X =0
X  0,

Recall that I • X = Tr (X) and that J = 11T . Recall also that if a matrix X is rank-one and thus
can be written as X = xxT , then L • X = xT Lx.
These two optimization problems, Problem 32 and Problem 33, are equivalent, in that if x∗ is the
vector solution to the former and X ∗ is a solution of the latter, then X ∗ = x∗ x∗ T . In particular,
note that although there is no constraint on the SDP formulation that the solution is rank-one, the
solution turns out to be rank one.
Observe that this is a linear SDP, in that the objective and all the constraints are linear. Linear
SDPs, just as LPs, can be very unstable. To see this in the simpler setting of LPs, consider a
convex set S ⊂ Rn and a linear optimization problem:
f (c) = arg min cT x.
x∈S

The optimal solution f (c) might be very unstable to perturbations of c, in that we can have
kc′ − ck ≤ δ

and kf (c′ ) − f (c)k ≫ δ.

(With respect to our Linear SDP, think of the vector x as the PSD variable X and think of the
vector c as the Laplacian L.) That is, we can change the input c (or L) a little bit and the solution
changes a lot. One way to fix this is to introduce a regularization term g(x) that is strongly convex.
So, consider the same convex set S ⊂ Rn and a regularized linear optimization problem
f (c) = arg min cT x + λg(x),
x∈S

where λ ∈ R+ is a parameter and where g(x) is σ-strongly convex. Since this is just an illustrative
example, we won’t define precisely the term σ-strongly convex, but we note that σ is related to

Lecture Notes on Spectral Graph Methods

167

the derivative of f (·) and so the parameter σ determines how strongly convex is the function g(x).
Then, since σ is related to the slope of the objective at f (c), and since the slope of the new objective
at f (c) < δ, strong convexity ensures that we can find a new optimum f (c′ ) at distance < σδ . So,
we have
kc′ − ck ≤ δ ⇒ kf (c′ ) − f (c)k < δ/σ,
(34)
i.e., the strong convexity on g(x) makes the problem stable that wasn’t stable before.

17.3

Structural characterization of a regularized SDP

How does this translate to the eigenvector problem? Well, recall that the leading eigenvector of
the Laplacian solves the SDP, where X appears linearly in the objective and constraints, as given
in Problem 33. We will show that several different variants of random walks exactly optimize
regularized versions of this SDP. In particular they optimize problems of the form
SDP Formulation :

1
d

minX∈Rn×n
s.t.

L • X + λG(X)

(35)

I •X =1

J •X =0
X  0,

where G(X) is an appropriate regularization function that depends on the specific form of the
random walk and that (among other things) is strongly convex.
To give an interpretation of what we are doing, consider the eigenvector decomposition of X, where

∀i pi ≥ 0
 P
X
T
X=
pi vi vi , where
(36)
i pi = 1

i
∀i viT ~1 = 0

I’ve actually normalized things so that the eigenvalues sum to 1. If we do this, then the eigenvalues
of X define a probability distribution. If we don’t regularize in Problem 35, i.e., if we set λ = 0,
then the optimal solution to Problem 35 puts all the weight on the second eigenvector (since
X ∗ = x∗ x∗ T ). If instead we regularize, then the regularization term P
ensures that the weight is
T
∗
spread out on all the eigenvectors, i.e., the optimal solution X =
i αi vi vi , for some set of
n
coefficients {αi }i=1 . So, the solution is not rank-one, but it is more stable.

Fact. If we take this optimization framework and put in “reasonable” choices for G(X), then we
can recover algorithms that are commonly used in the design of fast algorithms and elsewhere.
That the solution is not rank one makes sense from this perspective: if we iterate t → ∞, then all
the other eigendirections are washed out, and we are left with the leading direction; but if we only
iterate to a finite t, then we still have admixing from the other eigendirections.
To see this in more detail, consider the following three types of random walks. (Recall that M =
AD −1 and W = 12 (I + M ).)
• Heat Kernel. Ht = exp (−tL) =
matrix onto that eigendirection.

P∞

k=0

(−t)k k
k! L

=

Pn

−λi t P ,
i
i=1 e

where Pi is a projection

• PageRank. Rγ = γ (I − (1 − γ) M )−1 . This follows since the PageRank vector is the solution to
π(γ, s) = γs + (1 − γ)M π(γ, s),

168

M. W. Mahoney
which can be written as
π(γ, s) = γ

∞
X
t=0

(1 − γ)t M t s = Rγ s.

• Truncated Lazy Random Walk. Wα = αI + (1 − α)M .
These are formal expressions describing the action of each of those three types of random walks,
in the sense that the specified matrix maps the input to the output: to obtain the output vector,
compute the matrix and multiply it by the input vector. Clearly, of course, these random walks
would not be implemented by computing these matrices explicitly; instead, one would iteratively
apply a one-step version of them to the input vector.
Here are the three regularizers we will consider.
• von Neumann entropy. Here, GH = Tr (X log X) =
• Log-determinant. Here, GD = − log det (X).

P

i pi log pi .

• Matrix p-norm, p > 0. Here, Gp = p1 kXkpp = 1p Tr (X p ) =
And here are the connections that we want to establish.

1
p

P

p
i pi .

• G = GH ⇒entropy X ∗ ∼ Ht , with t = λ.
• G = GD ⇒logdet X ∗ ∼ Rγ , with γ ∼ λ.
• G = Gp ⇒p−norm X ∗ ∼ Wαt , with t ∼ λ.
Here is the basic structural theorem that will allow us to make precise this connection between
random walks and regularized SDPs. Note that its proof is a quite straightforward application of
duality ideas.
Theorem 36. Recall the regularized SDP of Problem 35, and let λ = 1/η. If G is a connected,
weighted, undirected graph, then let L be the normalized Laplacian. The the following are sufficient
conditions for X ∗ to be the solution of the regularized SDP.
I. X ∗ = (∇G)−1 (η (λ∗ I − L)), for some λ∗ ∈ R.
II. I • X ∗ = 1.
III. X ∗  0.
Proof. Write the Lagrangian L of the above SDP as
1
L = L • X + G(X) − λ (I • X − 1) − U • X,
η
where λ ∈ R and where U  0. Then, the dual objective function is
h (λ, U ) = min L(X, λ, U ).
X0

Lecture Notes on Spectral Graph Methods

169

Since G(·) is strictly convex, differentiable, and rotationally invariant, the gradient of G over the
positive semi-definite cone is invertible, and the RHS is minimized when
X = (∇G)−1 (η (−L + λ∗ I + U )) ,
where λ∗ is chosen s.t. I • X ∗ = 1. Hence,

1
h (λ∗ , 0) = L • X ∗ + G(X ∗ ) − λ∗ (I · X ∗ − 1)
η
1
= L • X ∗ + G(X ∗ ).
η

By Weak Duality, this implies that X ∗ is the optimal solution to the regularized SDP.

17.4

Deriving different random walks from Theorem 36

To derive the Heat Kernel random walk from Theorem 36, let’s do the following. Since
GH (X) = Tr (X log(X)) − Tr (X) ,

it follows that (∇G) (X) = log(x) and thus that (∇G)−1 (Y ) = exp(Y ), from which it follows that
X ∗ = (∇G)−1 (η (λI − L))
= exp (η (λI − L))

for an appropriate choice of η, λ

= exp (−ηL) exp (ηλ)
Hη
=
,
Tr (Hη )

where the last line follows if we set λ =

−1
η

log (Tr (exp (−ηL)))

To derive the PageRank random walk from Theorem 36, we follow a similar derivation. Since
GD (X) = − log det (X) ,

it follows that (∇G) (X) = −X −1 and thus that (∇G)−1 (Y ) = −Y −1 , from which it follows that
X ∗ = (∇G)−1 (η (λI − L))
= − (η (λI − L))−1

=

D −1/2 R

γ

D −1/2

Tr (Rγ )

for an appropriate choice of η, λ

,

for η, λ chosen appropriately.
Deriving the truncated iterated random walk or other forms of diffusions is similar.
We will go into more details on the connection with PageRank next time, but for now we just state
that the solution can be written in the form
x∗ = c (LG − αLKn )+ Ds,
for a constant c and a parameter γ. That is, it is of the form of the solution to a linear equation,
i.e., L+
G s, except that there is a term that moderates the effect of the graph by adding the Laplacian
of the complete graph. This is essentially a regularization term, although it is not usually described
as such. See the Gleich article for more details on this.

170

17.5

M. W. Mahoney

Interpreting Heat Kernel random walks in terms of stability

Here, we will relate the two previous results for the heat kernel. Again, for simplicity, assume
that G is d-regular. Recall that the Heat Kernel random walk is a continuous time Markov chain,
modeling the diffusion of heat along the edges of G. Transitions take place in continuous time t
with an exponential distribution:


ρ(t)
t
∂ρ(t)
= −L
⇒ ρ(t) = exp − L ρ(0).
∂t
d
d
That is, this describes the way that the probability distribution changes from one step to the
next and how it is related to L. In particular, the Heat Kernel can be interpreted as a Poisson
distribution over the number of steps of the natural random walk W = AD −1 , where we get the
following:
∞ k
X
t
− dt L
−t
e
W k.
=e
k!
k=1

What this means is: pick a number of steps from the Poisson distribution; and then perform that
number of steps of the natural random walk.

So, if we have two graphs G and G′ and they are close, say in an ℓ∞ norm sense, meaning that
the edges only change a little, then we can show the following. (Here, we will normalize the two
graphs so that their respective eigenvalues sum to 1.) The statement analogous to Statement 34 is
the following.
t
t
HG
HG
′
−
kG − G′ k∞ ⇒ k
t
t k1 ≤ tδ.
I • HG I • HG
′

Here, k · k1 is some other norm (the ℓ1 norm over the eigenvalues) that we won’t describe in detail.
Observe that the bound on the RHS depends on how close the graphs are (δ) as well as how long
the Heat Kernel random walk runs (t). If the graphs are far apart (δ is large), then the bound is
weak. If the random walk is run for a long time (t → ∞), then the bound is also very weak. But,
if the walk is nor run too long, then we get a robustness result. And, this follows from the strong
convexity of the regularization term that the heat kernel is implicitly optimizing exactly.

17.6

A statistical interpretation of this implicit regularization result

Above, we provided three different senses in which early-stopped random walks can be interpreted
as providing a robust or regularized notion of the leading eigenvectors of the Laplacian: e.g., in
the sense that, in addition to approximating the Rayleigh quotient, they also exactly optimize
a regularized version of the Rayleigh quotient. Some people interpret regularization in terms of
statistical priors, and so let’s consider this next.
In particular, let’s now give a statistical interpretation to this implicit regularization result. By a
“statistical interpretation,” I mean a derivation analogous to the manner in which ℓ2 or ℓ1 regularized ℓ2 regression can be interpreted in terms of a Gaussian or Laplace prior on the coefficients
of the regression problem. This basically provides a Bayesian interpretation of regularized linear
regression. The derivation below will show that the solutions to the Problem 35 that random
walkers implicitly optimize can be interpreted as a regularized estimate of the pseudoinverse of the
Laplacian, and so in some sense it provides a Bayesian interpretation of the implicit regularization
provided by random walks.

Lecture Notes on Spectral Graph Methods

171

To start, let’s describe the analogous results for vanilla linear regression. For some (statistics)
students, this is well-known; but for other (non-statistics) students, it likely is not. The basic idea
should be clear; and we cover it here to establish notation and nomenclature. Let’s assume that we
see n predictor-response pairs in Rp × R, call them {(xi , yi )}ni=1 , and the goal is to find a parameter
vector β ∈ Rp such that β T xi ≈ yi . A common thing to do is to choose β by minimizing the RSS
(residual sum of squares), i.e., choosing
F (β) = RSS(β) =

n
X
i=1

kyi − β T xi k22 .

Alternatively, we could optimize a regularized version of this objective. In particular, we have
Ridge regression:

min F (β) + λkβk22

Lasso regression:

min F (β) + λkβk1 .

β

β

To derive these two versions of regularized linear regression, let’s model yi as independent random
variables with distribution dependent on β as follows:

yi ∼ N β T x, σ 2 ,
(37)

i.e., each yi is a Gaussian random variable with mean β T xi and known variance σ 2 . This induces
a conditional density for y as follows:
−1
F (β)},
(38)
2σ 2
where the constant of proportionality depends only on y and σ. From this, we can derive the vanilla
least-squares estimator. But, we can also assume that β is a random variable with distribution p(β),
which is known as a prior distribution, as follows:
p (y|β) ∼ exp{

p(β) ∼ exp{−U (β)},

(39)

where we adopt that functional form without loss of generality. Since these two random variables
are dependent, upon observing y, we have information on β, and this can be encoded in a posterior
density, p (β|y), which can be computed from Bayes’ rule as follows:
p (β|y) ∼ p (y|β) p(β)
−1
∼ exp{ 2 F (β) − U (β)}.
2σ
We can form the MAP, the maximum a posteriori, estimate of β by solving

(40)

max p (β|y) iff min − log p (β|y) .
β

β

From this we can derive ridge regression and Lasso regression:
U (β) =
U (β) =

λ
kβk22
2σ 2
λ
kβk1
2σ 2

⇒ Ridge regression
⇒ Lasso regression

To derive the analogous result for regularized eigenvectors, we will follow the analogous setup.
What we will do is the following. Given a graph G, i.e., a “sample” Laplacian L, assume it is a
random object drawn from a “population” Laplacian L.

172

M. W. Mahoney
• This induces a conditional density for L, call it p (L|L).
• Then, we can assume prior information about the population Laplacian L in the form of p (L).
• Then, given the observed L, we can estimate the population Laplacian by maximizing its
posterior density p (L|L).

While this setup is analogous to the derivation for least-squares, there are also differences. In
particular, one important difference between the two approaches is that here there is one data
point, i.e., the graph/Laplacian is a single data point, and so we need to invent a population
from which it was drawn. It’s like treating the entire matrix X and vector y in the least-squares
problem as a single data point, rather than n data points, each of which was drawn from the
same distribution. (That’s not a minor technicality: in many situations, including the algorithmic
approach we adopted before, it is more natural to think of a graph as a single data point, rather
than as a collection of data points, and a lot of statistical theory breaks down when we observe
N = 1 data point.)
In more detail, recall that a Laplacian is an SPSD matrix with a very particular structure, and let’s
construct/hypothesize a population from which it was drawn. To do so, let’s assume that nodes n
in the population and in the sample have the same degrees. If d = (d1 , . . . , dn ) is the degree vector,
and D = deg (d1 , . . . , dn ) is the diagonal degree matrix, then we can define the set
χ = {X : X  0, XD 1/2~1 = 0, rank(X) = n − 1}.
So, the population Laplacian and sample Laplacian are both members of χ. To model L, let’s
use a scaled Wishart matrix with expectation L. (This distribution plays the role of the Gaussian
distribution in the least-squares derivation. Note that this is a plausible thing to assume, but
other assumptions might be possible too.) Let m ≥ n − 1 be a scale parameter (analogous to the
1
Wishart (L, m). Then E [L|L] = L and L
variance), and suppose that L is distributed over χ as m
has the conditional density
p (L|L) ∼

1
m/2

det (L)

exp{


−m
Tr LL+ }.
2

(41)

This is analogous to Eqn (38) above. Next, we can say that L is a random object with prior density
p (L), which without loss of generality we can take to be of the following form:
p (L) ∼ exp{−U (L)},
where U is supported on a subset χ̄ ⊆ χ. This is analogous to Eqn (39) above. Then, observing L,
the posterior distribution for L is the following:
p (L|L) ∼ p (L|L) p (L)
 m

m
∼ exp{ Tr LL+ + log det L+ − U (L)},
2
2

with support determined by χ̄. This is analogous to Eqn (40) above.

If we denote by L̂ the MAP estimate of L, then it follows that L̂+ is the solution of the following
optimization problem.
minX
s.t.

Tr (L • X) +
X ∈ χ̄ ⊆ χ.


2
U X + − log det (X)
m

(42)

Lecture Notes on Spectral Graph Methods

173

If χ̄ = {X : Tr (X) = 1} ∩ χ, then Problem 42 is the same as Problem 35, except for the factor of
log det (x).
This is almost the regularized SDP we had above.
Next, we present a prior that will be related to the PageRank procedure. This will make the
connection with the regularized SDP more precise. In particular, we present a prior for the population Laplacian that permits us to exploit the above estimation framework to show that the MAP
estimate is related to a PageRank computation.
The criteria for the prior are so-called neutrality and invariance conditions. It is to be supported
on χ; and in particular, for any X ∈ χ, it will have rank n − 1 and satisfy XD 1/2 1 = 0. The prior
will depend only on the eigenvalues of the Laplacian (or equivalently of the inverse Laplacian). Let
L+ = τ OΛO be the spectral decomposition of the inverse Laplacian, where τ is a scale parameter.
We will require that the distribution for λ = (λ1 , . . . , λn ) be exchangeable (i.e., invariant under
λ(u)
permutations) and neutral (i.e., λ(v) is independent of 1−λ(v)
, for u 6= v, for all v). The only
non-degenerate possibility is that λ is distributed as a Dirichlet distribution as follows:
p (L) ∼ p(τ )

n−1
Y

λ(v)α−1 ,

(43)

v=1

where α is a so-called shape parameter. Then, we have the following lemma.
Lemma 21. Given the conditional likelihood for L given L in Eqn. (41) and the prior density for
L given in Eqn. (43); if L̂ is the MAP estimate of L, then
L̂+
 
Tr L̂+

solves the regularized SDP, with G(X) = − − log det(X) and with the value of η given in the
proof below.
Proof. For L in the support set of the posterior, we can define τ = Tr (L+ ) and Θ = τ1 L+ , so that
rank(Θ) = n − 1 and Tr (Θ) = 1. Then, p (L) ∼ exp{−U (L)}, where
U (L) = − log{p(τ ) det (Θ)α−1 } = −(α − 1) log det (Θ) − log (p(τ )) .
Thus,
 m

m
Tr LL+ + log det L+ − U (L)}
2
2
−mτ
m + 2(α − 1)
∼ exp{
Tr (LΘ) +
log det (Θ) + g(τ )},
2
2

p (L|L) ∼ exp{−

m(n−1)
where the second line follows since det (L+ ) = τ n−1 det (Θ), and where
log(τ ) +
2
 g(τ ) =
log p(τ ). If L̂ maximizes the posterior likelihood, then define τ̂ = Tr Lˆ+ and Θ̂ = τ1 L̂+ , and so
 
 
Θ̂ must minimize Tr LΘ̂ − η1 log det Θ̂ , where

η=

mτ̂
m + 2(α − 1)

This Θ̂ solves the regularized SDP with G(x) = − log det (X).

174

M. W. Mahoney

Remark. Lemma 21 provides a statistical interpretation of the regularized problem that is optimized by an approximate PageRank diffusion algorithm, in the sense that it gives a general
statistical estimation procedure that leads to the Rayleigh quotient as well as statistical prior related to PageRank. One can write down priors for the Heat Kernel and other random walks; see
the two references if you are interested. Note, however, that the prior for PageRank makes things
particularly simple. The reason is that the extra term in Problem 42, i.e., the log det (X) term,
is of the same form as the regularization function that the approximate PageRank computation
implicitly regularizes with respect to. Thus, we can choose parameters to make this term cancel.
Otherwise, there are extra terms floating around, and the statistical interpretation is more complex.

Lecture Notes on Spectral Graph Methods

18

175

(03/31/2015): Local Spectral Methods (1 of 4): Introduction
and Overview

Reading for today.
• “Spectral Ranking”, in arXiv, by Vigna
• “PageRank beyond the Web,” in SIAM Review, by Gleich
Last time, we showed that certain random walks and diffusion-based methods, when not run to the
asymptotic limit, exactly solve regularized versions of the Rayleigh quotient objective (in addition
to approximating the Rayleigh quotient, in a manner that depends on the specific random walk
and how the spectrum of L decays). There are two ways to think about these results.
• One way to think about this is that one runs almost to the asymptotic state and then one gets
a vector that is “close” to the leading eigenvector of L. Note, however, that the statement of
implicit regularization from last time does not depend on the initial condition or how long
the walk was run. (The value of the regularization parameter, etc., does, but the form of the
statement does not.) Thus ...
• Another way to think about this is that one starts at any node, say a localized “seed set”
of nodes, e.g., in which all of the initial probability mass is on one node or a small number
of nodes that are nearby each other in the graph topology, and then one runs only a small
number of steps of the random walk or diffusion. In this case, it might be more natural/useful
to try to quantify the idea that: if one starts the random walk on the small side of a bottleneck
to mixing, and if one runs only a few steps of a random walk, then one might get stuck in
that small set.
The latter is the basic idea of so-called local spectral methods, which are a class of algorithms that
have received a lot of attention recently. Basically, they try to extend the ideas of global spectral
methods, where we compute eigenvectors, random walks, etc., that reveal structure about the entire
graph, e.g., that find a partition that is quadratically-good in the sense of Cheeger’s Inequality to
the best conductance/expansion cut in the graph, to methods that reveal interesting structure in
locally-biased parts of the graph. Not only do these provide locally-biased versions of global spectral
methods, but since spectral methods are often used to provide a ranking for the nodes in a graph
and/or to solve other machine learning problems, these also can be used to provide a locally-biased
or personalized version of a ranking function and/or to solve other machine learning problems in a
locally-biased manner.

18.1

Overview of local spectral methods and spectral ranking

Here is a brief history of local and locally-biased spectral methods.
• LS: developed a basic locally-biased mixing result in the context of mixing of Markov chains
in convex bodies. They basically show a partial converse to the easy direction of Cheeger’s
Inequality—namely, that if the conductance φ(G) of the graph G is big, then every random
walk must converge quickly—and from this they also show that if the random walk fails to
converge quickly, then by examining the probability distribution that arises after a few steps
one can find a cut of small conductance.

176

M. W. Mahoney
• ST: used the LS result to get an algorithm for local spectral graph partitioning that used
truncated random walks. They used this to find good well-balanced graph partitions in
nearly-linear time, which they then used as a subroutine in their efforts to develop nearly
linear time solvers for Laplacian-based linear systems (a topic to which we will return briefly
at the end of the semester).
• ACL/AC: improved the ST result by computing a personalized PageRank vector. This improves the fast algorithms for Laplacian-based linear solvers, and it is of interest in its own
right, so we will spend some time on it.
• C: showed that similar results can be obtained by doing heat kernel computations.
• AP: showed that similar results can be obtained with an evolving set method (that we won’t
discuss in detail).
• MOV: provided an optimization perspective on these local spectral methods. That is, this is a
locally-biased optimization objective that, if optimized exactly, leads to similar locally-biased
Cheeger-like bounds.
• GM: characterized the connection between the strongly-local ACL and the weakly-local MOV
in terms of ℓ1 regularization (i.e., a popular form of sparsity-inducing regularizations) of ℓ2
regression problems.

There are several reasons why one might be interested in these methods.
• Develop faster algorithms. This is of particular interest if we can compute locally-biased
partitions without even touching all of the graph. This is the basis for a lot of work on nearly
linear time solvers for Laplacian-based linear systems.
• Improved statistical properties. If we can compute locally-biased things, e.g., locally-biased
partitions, without even touching the entire graph, then that certainly implies that we are
robust to things that go on on the other side of the graph. That is, we have essentially
engineered some sort of regularization into the approximation algorithm; and it might be of
interest to quantify this.
• Locally exploring graphs. One might be interested in finding small clusters or partitions that
are of interest in a small part of a graph, e.g., a given individual in a large social graph, in
situations when those locally-biased clusters are not well-correlated with the leading or with
any global eigenvector.
We will touch on all these themes over the next four classes.
P
For now, let G = (V, E), and recall that Vol(G) =
v∈T dv (so, in particular, we have that
Vol(G) = 2|E| = 2m. Also, A is the Adjacency Matrix, W = D −1 A, and L = I − D −1 A is the
random walk normalized Laplacian. For a vector x ∈ Rn , let’s define its support as
Supp(v) = {i ∈ V = [n] : vi 6= 0}.
Then, here is the transition kernel for that vanilla random walk.
(
1
if i ∼ j
P [xt+1 = j|xt = i] = d
0 otherwise.

Lecture Notes on Spectral Graph Methods

177

If we write this as a transition matrix operating on a (row) vector, then we have that
p(t) = sW t ,
where W is the transition matrix, and where s = p(0) is the initial distribution, with ksk1 = 1.
Then, p = ~1T D/(~1T D~1) is the stationary distribution, i.e.,
lim P [xt = i] =

t→∞

di
,
Vol(G)

independent of s = p(0), as long
 as G is connected and not bipartite. (If it is bipartite, then let
W → WLAZY = 12 I + D −1 A , and the same results holds.)
There are two common interpretations of this asymptotic random walk.
• Interpretation 1: the limit of a random walk.
• Interpretation 2: a measure of the importance of a node.
With respect to the latter interpretation, think of an edge as denoting importance, and then what
we want to find is the important nodes (often for directed graphs, but we aren’t considering that
here). Indeed, one of the simplest centrality measures in social graph analysis is the degree of a
node. For a range of reasons, e.g., since that is easy to spam, a refinement of that is to say that
important nodes are those nodes that have links to important nodes.
This leads to a large area known as spectral ranking methods. This area applies the theory of
matrices or linear maps—basically eigenvectors and eigenvalues, but also related things like random
walks—to matrices that represent some sort of relationship between entities. This has a long history,
most recently made well-known by the PageRank procedure (which is one version of it). Here, we
will follow Vigna’s outline and description in his “Spectral Ranking” notes—his description is very
nice since it provides the general picture in a general context, and then he shows that with several
seemingly-minor tweaks, one can obtain a range of related spectral graph methods.

18.2

Basics of spectral ranking

To start, take a step back, and let M ∈ Rn×n , where each column/row of M represents some sort
of entity, and Mij represents some sort of endorsement or approval of entity j from entity i. (So
far, one could potentially have negative entries, with the obvious interpretation, but this will often
be removed later, basically since one has more structure if entries must be nonnegative.) As Vigna
describes, Seeley (in 1949) observed that one should define importance/approval recursively, since
that will capture the idea that an entity is important/approved if other important entities think is
is important/approved. In this case, recursive could mean that
r = rM,
i.e., that the index of the ith node equals the weighted sum of the indices of the entities that
endorse the ith node. This isn’t always possible, and indeed Seeley considers nonnegative matrices
that don’t have any all-zeros rows, in which case uniqueness, etc., follow from the Perron-Frobenius
ideas we discussed before. This involves the left eigenvectors, as we have discussed; one could also
look at the right eigenvectors, but the endorsement/approval interpretation fails to hold.

178

M. W. Mahoney

Later, Wei (in 1952) and Kendall (in 1955) were interested in ranking sports teams, and they said
essentially that better teams are those teams that beat better teams. This involves looking at the
rank induced by
lim M k~1T
k→∞

and then appealing to Perron-Frobenius theory. The significant point here is three-fold.
• The motivation is very different than Seeley’s endorsement motivation.
• Using dominant eigenvectors on one side or the other dates back to mid 20th century, i.e.,
well before recent interest in the topic.
• The relevant notion of convergence in both of these motivating applications is that of convergence in rank (where rank means the rank order of values of nodes in the leading eigenvector,
and not anything to do with the linear-algebraic rank of the underlying matrix). In particular, the actual values of the entires of the vector are not important. This is very different
than other notions of convergence when considering leading eigenvectors of Laplacians, e.g.,
the value of the Rayleigh quotient.
Here is the generalization. Consider matrices M with real and positive dominant eigenvalue λ
and its eigenvector, i.e., vector r such that λr = rM , where let’s say that the dimension of the
eigenspace is one.
Definition 51. The left spectral ranking associated with M is, or is given by, the dominant left
eigenvector.
If the eigenspace does not have dimension one, then there is the usual ambiguity problem (which
is sometimes simply assumed away, but which can be enforced by a reasonable rule—we will see a
common way to do the latter in a few minutes), but if the eigenspace has dimension one, then we
can talk of the spectral ranking. Note that it is defined only up to a constant: this is not a problem
if all the coordinates have the same sign, but it introduces an ambiguity otherwise, and how this
ambiguity is resolved can lead to different outcomes in “boundary cases” where it matters; see the
Gleich article for examples and details of this.
Of course one could apply the same thing to M T . The mathematics is similar, but the motivation is
different. In particular, Vigna argues that the endorsement motivation leads to the left eigenvectors,
while the influence/better-than motivation leads to the right eigenvectors.
The next idea to introduce is that of “damping.” (As we will see, this will have a reasonable
generative “story” associated with it, and it will have a reasonable statistical interpretation, but
it is also important for a technical reason having to do with
 ensuring that the dimension of the
eigenspace is one.) Let M be a zero-one matrix; then M k ij is the number of directed paths from
i → j in a directed graph defined by M . In this case, an obvious idea of measuring the importance
of i, i.e., to measure the number of paths going into j, since they represent recursive endorsements,
which is given by
∞
X

~1 I + M + M 2 + · · · = I +
M k,
k=0

does not work, since the convergence of the equation is not guaranteed, and it does not happen
in general.

Lecture Notes on Spectral Graph Methods

179

If, instead, one can guarantee that the spectral radius of M is less than one, i.e., that λ0 < 1, then
this infinite sum does converge. One way to do this is to introduce a damping factor α to obtain
∞
X

~1 I + αM + α2 M 2 + · · · = I +
(αM )k .
k=0

This infinite sum does converge, as the spectral radius of αM is strictly less than 1, if α <
Katz (in 1953) proposed this. Note that
~1

∞
X
k=0

1
λ0 .

(αM )k = ~1 (I − αM )−1 .

So, in particular, we can compute this index by solving the linear system as follows.
x (I − αM ) = ~1.
This is a particularly well-structured system of linear equations, basically a system of equations
where the constraint matrix can be written in terms of a Laplacian. There has been a lot of work
recently on developing fast solvers for systems of this form, and we will get back to this topic in a
few weeks.
A generalization of this was given by Hubbel (in 1965), who said that one could define a status
index r by using a recursive equation r = v + rM , where v is a “boundary condition” or “exogenous
contribution.” This gives
∞
X
r = v (I − M )−1 = v
M k.
k=0

So, we are generalizing from the case where the exogenous contribution is ~1 to an arbitrary vector v.
This converges if λ0 < 1; otherwise, one could introduce an damping factor (as we will do below).
To see precisely how these are all related, let’s consider the basic spectral ranking equation
λ0 r = rM.
If the eigenspace of λ0 has dimension greater than 1, then there is no clear choice for the ranking r.
One idea in this case is to perturb M to satisfy this property, but we want to apply a “structured
perturbation” in such a way that many of the other spectral properties are not damaged. Here is
the relevant theorem, which is due to Brauer (in 1952), and which we won’t prove.
Theorem 37. Let A ∈ Cn×n , let

λ0 , λ1 , . . . , λn−1

be the eigenvalues of A, and let x ∈ Cn be nonzero vector such that AxT = λ0 xT . Then, for all
vectors v ∈ Cn , the eigenvalues of A + xT v are given by
λ0 + vxT , λ1 , . . . , λn−1 .
That is, if we perturb the original matrix by an rank-one update, where the rank-one update is of
the form of the outer product of an eigenvector of the matrix and an arbitrary vector, then one
eigenvalue changes, while all the others stay the same.

180

M. W. Mahoney

In particular, this result can be used to split degenerate eigenvalues and to introduce a gap into
the spectrum of M . To see this, let’s consider a rank-one convex perturbation of our matrix M by
using a vector v such that vxT = λ0 and by applying the theorem to αM and (1 − α) xT v. If we
do this then we get

λ0 r = r αM + (1 − α) xT v .

Next, note that αM + (1 − α) xT v has the same dominant eigenvalues as M , but with algebraic
multiplicity 1, and all the other eigenvalues are multiplied by α ∈ (0, 1).

This ensures a unique r. The cost is that it introduces extra parameters (α is we set v to be an allones vector, but the vector v if we choose it more generally). These parameters can be interpreted
in various ways, as we will see.
An important consequence of this approach is that r is defined only up to a constant, and so we
can impose the constraint that rxT = λ0 . (Note that if x = ~1, then this says that the sum of r’s
coordinates is λ0 , which if all the coordinates have the same sign means that krk1 = λ0 .) Then we
get
λ0 r = αrM + (1 − α) λ0 v.
Thus,
r (λ0 − αM ) = (1 − α) λ0 v.
From this it follows that
r = (1 − α) λ0 v (λ0 − αM )−1
−1

α
= (1 − α) v 1 − M
λ0
k
∞ 
X
α
M
= (1 − α) v
λ0
k=0
∞
X

= (1 − λ0 β) v
which converges if α < 1, i.e., if β <

(βM )k ,

k=0

1
λ0 .

That is, this approach shows that the Katz-Hubbel index can be obtained as a rank-one perturbation
of the original matrix. In a little bit, we will get to what this rank-one perturbation “means” in
different situations.
To review, we started with a matrix M with possibly many eigenvectors associated with the dominant eigenvalue, and we introduced a structured perturbation get a specific eigenvector associated
with λ0 , given the boundary condition v.
The standard story is that if we start from a generic nonnegative matrix and normalize its rows to
get a stochastic matrix, then we get a Markovian spectral ranking, which is the limit distribution
of the random walk. Here, we are slightly more general, as is captured by the following definition.
Definition 52. Given the matrix M , the sampled spectral ranking of M with boundary condition
v and dumping factor α is
k
∞ 
X
α
M ,
r0 = (1 − α) v
λ0
k=0

for |α| < 1.

Lecture Notes on Spectral Graph Methods

181

The interpretation of the boundary condition (from the sampled to the standard case) is the
following.
• In the damped case, the Markov chain is restarted to a fixed distribution v, and there is a
single stationary distribution which is the limit of every starting distribution.
• In the standard case, v is the starting distribution from which we capture the limit using an
eigenprojection.
While in some sense equivalent, these two interpretations suggest different questions and interpretations, and we will consider both of these over the next few classes.

18.3

A brief aside

Here is an aside we will get back to over the next few classes.
Consider a vanilla random walk, where at each time step, we follow the graph with some probability
β, and we randomly jump to any uniformly-chosen node in the graph with probability 1 − β. This
Markov chain has stationary distribution
1 − β~
1 + βpβ W.
n
This is often known as PageRank, which has received a lot of attention in web ranking, but from
the above discussion it should be clear that it is one form of the general case of spectral ranking
methods. We can also ask for a “personalized” version of this, by which we informally mean a
ranking of the nodes that in some sense is conditioned on or personalized for a given node or a
given seed set of nodes. We can get this by using a personalized PageRank, by randomly jumping
(not to any node chosen uniformly at random but) to a “seed set” s of nodes. This PPR is the
unique solution to
pβ (s) = (1 − β) s + βpβ (s)W,
pβ =

i.e., it is of the same form as the expression above, except that an all-ones vector has been replaced
by a seed set or boundary condition vector V . This latter expression solves the linear equation
(I − βW ) pβ (s) = (1 − β) s.

We can write this expression as an infinite sum as
pβ (s) = (1 − β) s + β

∞
X

β t (sW )t

t=0

Thus, note that the following formulations of PageRank (as well as spectral ranking more generally)
are equivalent.
• (I − βW ) x = (1 − β) s
• (γD + L) z = γs, where β =

1
1+γ

and x = Dz.

As noted above, this latter expression is of the form of Laplacian-based linear equations. It is of
the same form that arises in those semi-supervised learning examples that we discusses. We will
talk toward the end of the term about how to solve equations of this form more generally.

182

M. W. Mahoney

19

(04/02/2015): Local Spectral Methods (2 of 4): Computing
spectral ranking with the push procedure

Reading for today.
• “The Push Algorithm for Spectral Ranking”, in arXiv, by Boldi and Vigna
• “Local Graph Partitioning using PageRank Vectors,” in FOCS, by Andersen, Chung, and
Lang
Last time, we talked about spectral ranking methods, and we observed that they can be computed
as eigenvectors of certain matrices or as the solution to systems of linear equations of certain
constraint matrices. These computations can be performed with a black box solver, but they can
also be done with specialized solvers that take advantage of the special structure of these matrices.
(For example, a vanilla spectral ranking method, e.g., one with a preference vector ~v that is an
all-ones vector ~1, has a large eigenvalue gap, and this means that one can obtain a good solution
with just a few steps of a simple iterative method.) As you can imagine, this is a large topic. Here,
we will focus in particular on how to solve for these spectral rankings in the particular case when
the preference vector ~v is has small support, i.e., when it has its mass localized on a small seed set
of nodes. This is a particularly important use case, and the methods developed for it are useful
much more generally.
In particular, in this class, we will describe how to compute personalized/local spectral rankings
with a procedure called the push procedure. This has several interesting properties, in general, but
in particular when computing locally-biased or personalized spectral rankings in a large graph. As
with the last class, we will follow the Boldi-Vigna’s notes on “The Push Algorithm for Spectral
Ranking” on the topic.

19.1

Background on the method

To start, let M ∈ Rn×n be a nonnegative matrix; and WLOG assume that kM k1 = 1, i.e., assume
that M is stochastic. (Actually, this assumes that M is sub-stochastic and that at least one row
of M sums to 1; this distinction won’t matter for what we do, but it could matter in other cases,
e.g., when dealing with directed graphs, etc.) Then, let v be a nonnegative vector s.t. kvk1 = 1,
i.e., assume that v is a distribution, and let α ∈ [0, 1).
Then, recall that we defined the spectral ranking of M with preference vector v and damping factor
α to be
r = (1 − α) v (I − αM )−1
∞
X
= (1 − α) v
αk M k .
k=0

For this, note that r need not be a distribution unless M is stochastic (although
itis usually applied
h
1
in that case). Note also that the linear operator is defined for all α ∈ 0, ρ(M ) , but estimating
ρ(M ) can be difficult. (We’ll see an interesting example next week of what arises when we push
this parameter to the limit.)
From this perspective, while it is difficult to “guess” what is the spectral ranking r associated with

Lecture Notes on Spectral Graph Methods

183

a preference vector v, it is “easy” to solve the inverse problem: given r, solve
v=

1
r (I − αM ) .
1−α

(While this equation is always true, the resulting preference vector might not be a distribution;
otherwise, one could obtain any spectral ranking using a suitable preference vector.)
While this is obvious, it has important consequences for computing spectral rankings and approximate spectral rankings that we will now describe.
Consider the indicator vector Xx (z) = [x = z]. If we want to obtain the vector (1 − α) Xx as a
spectral ranking, then the associated preference vector v has the simple form:
1
(1 − α) Xx (I − αM )
1−α
X
= Xx − α
Mxy Xy

v =

(44)

x→y

= Xx − α
where note that if Mxy =

1
d(x)

1 X
Xy ,
d(x) x→y

is the natural random walk, then we can take it out of the summation.

This is true in general. The important point for us here is that if v is highly concentrated, e.g.,
if it is an indicator vector of s small set of nodes, and if α is not too close to 1, then most of the
updates done by the linear solver or by an iterative method either don’t update much or update
below a threshold level. Motivated by this, we will discuss the so-called push algorithm—this uses
a particular form of updates to reduce computational burden. This was used in work of Jeh and
Widom and also of Berkhin on personalized page rank, and it was used with this name by ACL.
Although they all apply it to PageRank, it applies for the steady state of Markov chains with
restart, and it is basically an algorithm for spectral ranking with damping (see the Boldi, Lonati,
Santini, Vigna paper).

19.2

The basic push procedure

The basic idea of this approach is that, rather that computing an exact PPR vector by iterating
the corresponding Markov chain (e.g., with vanilla matrix-vector multiplies) until it converges, it
is also possible to consider computing an approximate PPR vector much more efficiently. Recall
that the PPR vector is the unique solution to
πα (s) = (1 − α) s + απα (s)W,
and it can be written as an infinite sum
πα (s) = (1 − α) s + (1 − α)

∞
X

αt (sW )t .

t=1

With this notation, we can define the following notion of approximation of a PPR vector.
Definition 53. An ǫ-approximate PageRank vector πα (s) is any PageRank vector πα (s − r), where
r is nonnegative and r(v) ≤ ǫdv , for all v ∈ V .

184

M. W. Mahoney

Fact. The approximation error of an ǫ-approximate PageRank vector on any set of nodes S can
be bounded in terms of the Vol(S) and ǫ. Here is a basic lemma from ACL.
Lemma 22. For all ǫ-approximate PageRank vectors πα (s − r), and for all S ⊂ V , we have
πα (s)1TS ≥ πα (s − r)1TS ≥ πα (s)1TS − ǫVol(S).
Here is an algorithm to compute an ǫ-approximate PageRank vector; let’s call this algorithm
ApproxPR(s, α, ǫ).
I. Let p = ~0 and r = ~s.
II. While ru ≥ ǫdu for some vertex u,
• Pick and u such that ru ≥ ǫdn
• Apply Push(u)

III. Return the vectors p and r
And here is the Push(u) algorithm that is called by the ApproxPR(s, α, ǫ) algorithm.
I. Let p′ = p and r ′ = r, except for the following updates:
• p′u = pu + αru
• ru′ = (1 − α) r2u
ru
• rv′ = rv + (1 − α) 2d
, for all vertices v such that (u, v) ∈ E.
u
Note that we haven’t specified the order in which the pushes are executed, i.e., in which the Push(u)
algorithm is called by the ApproxPR(s, α, ǫ) algorithm, and so they can be done in different ways,
leading to slightly different algorithms.
Here is the theorem that ACL establishes about this procedure.
Theorem 38. Algorithm ApproxPR(s, α, ǫ) has the following properties.
• For all starting vertices with ksk1 ≤ 1 and for all ǫ ∈ (0, 1], the algorithm returns an ǫapproximate p for pα (s).
• The support of p satisfies
Vol(Supp(p)) ≤
• The running time of the algorithm is O

1
αǫ



2
.
(1 + α)ǫ

.

The idea of the proof—outlined roughly above—is that Algorithm Push(u) preserves the approximate PageRank condition p = π (s − r); and the stopping condition ensures that it is an ǫ
approximation. The running time follows since krk1 = 1 and it decreases by αǫdn at each time
Push(u) is called.

Lecture Notes on Spectral Graph Methods

19.3

185

More discussion of the basic push procedure

The basic idea of the method is to keep track of two vectors, p which is the vector of current
approximations and r which is the vector of residuals, such that the following global invariant is
satisfied at every step of the algorithm.
p + (1 − α) r (I − αM )−1 = (1 − α) v (I − αM )−1 .

(45)

Initially, p = 0 and r = v, and so this invariant is trivially satisfied. Subsequently, at each step the
push method increases p and decreases r to keep the invariant satisfied.
To do so, it iteratively “pushes” some node x. A push on x adds (1 − α) rx Xx to the vector p. To
keep the invariant try, the method must update r. To do so, think of r as a preference vector, in
which we are just trying to solve Eqn. (44). By linearity, if we subtract
!
X
Mxy Xy
rx X x − α
x→y

from r, then the value (1 − α) r (I − αM )−1 will decrease by (1 − α) rx Xx , thus preserving the
invariant.
This is a good choice since
• We zero out the xth entry of r.
• We add a small positive quantities to a small set of entries (small set, if the graph is sparse,
which is the use case we are considering). This increases the ℓ1 norm of p by (1 − α) rx , and
it decreases r by at least the same amount.
Since we don’t create negative entries in this process, it always holds that
kpk1 + krk1 ≤ 1,
and thus we can keep track of two norms at each update. The error in the estimate is then given
by the following.
X
k (1 − α) r (I − αM )−1 k1 = (1 − α) kr
αk M k k1
k≥0

≤ (1 − α) krk1
≤ krk1

X
k≥0

αk kM k k1

So, in particular, we can control the absolute error of the algorithm by controlling the ℓ1 error of
the residual.

19.4

A different interpretation of the same process

Here is a different interpretation of the push method. Some might find useful—if so, good, and if
not, then just ignore the following.
Just as various random walks can be related to diffusion of heat/mass, one can think of PageRank
as related to the diffusion of a substance where some fraction of that substance gets stuck in place

186

M. W. Mahoney

at each time step—e.g., diffusing paint, where the point of paint is that part of the paint dries in
place and then stops diffusing/flowing. (Think here of doing the updates to push asynchronously.)
At each time step, 1 − α fraction of the paint dries in place, and α fraction of the paint does a
lazy random walk. So, we need to keep track of two quantities: the amount of wet paint (that still
moves at the next step) and the amount of dry paint (that is probability mass that won’t move
again). If we let
• p : V → Rn be a vector that says how much pain is stuck at each vertex.
• r : V → Rn be a vector saying how much wet paint remains at each vertex.
Then, at t = 0, r 0 = Xu , and these vectors evolve as
pt+1 = pt + (1 − α) r t
r t = αr Ŵ .

(Note that I have swapped sides where the vector is multiplying, so I think I have inconsistencies
here to fix. Also, I think I refer to α and 1 − α inconsistently, so that must be fixed too)
Given this, if we let p∞ be where paint is dried at the end of the process, then
X
p∞ = (1 − α)
rt
t≥0

= (1 − α)
=

X
t≥0

X

αt r 0 Ŵ t

t≥0

αt Xu Ŵ t

This is simply PPR, with a different scaling and α. Recall here that Ŵ =
P
i
(I + X)−1 = ∞
i=0 X , if the spectral radius of X is less than 1, it follows that

−1
p∞ = (1 − α) Xu I − αŴ

α −1
α
I− W
= (1 − α) Xu 1 −
2
2
−1
= γXu (I − (1 − γ) W ) ,

1
2I

+ 21 W . Since

where the parameter γ is defined in terms of the parameter α.
Note that in this we don’t need to update the above time process but we can ignore time and do it
in an asynchronous manner. That is, we can compute this by solving a linear equation or running
a random walk in which one keeps track of two vectors that has this interpretation in terms of
diffusing paint. But, Jeh-Widom and Berkhin note that rather than doing it with these equations,
one can instead choose a vertex, say that a fraction α of paint at that vertex is dry, and then push
the wet paint to the neighbors according to the above rule. (That is, we can ignore time and do
it asynchronously.) But this just gives us the push process. To see this, let πp,r be the vector of
“dried paint” that we eventually compute. Then
X
πp,r = p + (1 − α)
rαt W t
t≥0

= p + (1 − α) r (I − αW )−1

Lecture Notes on Spectral Graph Methods

187

In this case, the updates we wrote above, written another way, are the following: pick a vertex u
and create p′ and r ′ as follows.
• p′ (u) = p(u) + (1 − α) r(u)
• r ′ (u) = 0
• r ′ (v) = r(v) +

α
d(u) r(u),

for all neighbors v of u.

Then, it can be shown that
πp′ ,r′ = πp,r ,
which is the invariant that we noted before.
So, the idea of computing the approximate PageRank vectors is the following.
• Pick a vertex where r(u) is large or largest and distribute the paint according to that rule.
• Choose a threshold ǫ and don’t bother to process a vertex if r(u) ≤ ǫd(u).
Then, it can be shown that
Lemma 23. The process will stop within

1
ǫ(1−α)

iterations.

(Note there is an inconsistency with α and 1−α, i.e., whether α is the teleporting or non-teleporting
probability, that needs to be fixed.)

19.5

Using this to find sets of low-conductance

This provides a way to “explore” a large graph. Two things to note about this.
• This way is very different that DFS or BFS, which are not so good if the diameter is very
small, as it is in many real-world graphs.
• The sets of nodes that are found will be different than with a geodesic metric. In particular,
diffusions might get stuck in good conductance sets.
Following up on that second point, ACL showed how to use approximate PPR to find sets of low
conductance, if they start from a random vector in a set. Importantly, since they do it with the
Approximate PPR vector, the number of nodes that is touched is proportional to the size of the
output set. So, in particular, if there is a set of small conductance, then the algorithm will run
very quickly.
Here is the basic idea. ACL first did it, and a simpler approach/analysis can be found in AC07.
• Given πv , construct qv (u) =

πv (u)
d(u) .

• Number the vertices, WLOG, such that qv (1) ≥ qv (2) ≥ · · · qv (n), and let Sk = {1, . . . , k}.

188

M. W. Mahoney

Then, it can be shown that starting from a random vertex in the set of low conductance, then one
of the sets has low conductance.
Computationally, these and related diffusion-based methods use only local information in the graph.
If one can look at the entire graph, then one can get information about global eigenvectors, which
approximate sparse cuts. But, if the graphs are huge, then one might be interested in the following
question:
• Given a graph G = (V, E) and a node v ∈ V , find a set of nodes V such that the Cheeger
ratio hS is small.
|E(S,S̄)|
. Before, we were interested in good global clusters, and thus
Recall that hS =
min{Vol(S),Vol(S̄)}
we defined hG = minS⊂V hS and tried to optimize it, e.g., by computing global eigenvectors; but
here we are not interested in global eigenvectors and global clusters.

Note that it is not immediately obvious that diffusions and PageRank are related to this notion of
local clustering, but we will see that they are. In particular, we are interested in the following.
• Given a seed node s, we want to find a small set of nodes that is near s and that is well
connected internally but that is less well connected with the rest of the graph.
The intuition is that if a diffusion is started in such a cluster, then it is unlikely to leave the cluster
since the cluster is relatively poorly-connected with the rest of the graph. This is often true, but
one must be careful, since a diffusion that starts, e.g., at the boundary of a set S can leave S at
the first step. So, the precise statement is going to be rather complicated.
Here is a lemma for a basic diffusion process of a graph G.
Lemma 24. Let C ⊂ V be a set with Cheeger ratio hC , and let t0 be a time parameter. Define Ct0
to be a subset of C such that for all v ∈ Ct0 and all t ≤ t0 , we have that ~1Tv W t~1TC̄ ≤ t0 hC . Then,
the volume of Ct0 is such that
1
Vol (Ct0 ) ≥ Vol (C) .
2
This is a complicated lemma, but it implies that for vertex set C ⊂ V with small Cheeger ratio,
that there are many nodes v ∈ C such that the probability of a random walk starting at v leaves
C is low.
There is a similar result for the PageRank diffusion process. The statement is as follows.
Lemma 25. Let C ⊂ V have Cheeger ratio hC , and let α ∈ (0, 1]. Then, there is a subset Cα ⊆ C
with volume Vol (Cα ) ≥ 12 Vol (C) such that for all v ∈ Cα , the PageRank vector πα (1V ) satisfies
hC
πα (1V ) ~1TC ≥ 1 −
.
α
This is also a complicated lemma, but it implies that for any vertex set C ⊆ V with small hC , that
there are many nodes v ∈ C for which PPR, using v as a start node, is small on nodes outside of C.
These and related results show that there is a relationship or correlation between the Cheeger
ratio and diffusion processes, even for “short” random walks do not reach the asymptotic limit. In

Lecture Notes on Spectral Graph Methods

189

particular, for a set C ⊂ V with small hC , it is relatively hard for a diffusion process started within
C to leave C.
We can use these ideas to get local spectral clustering algorithms. To do this, we need a method for
creating vertex cuts from a PR or random walk vector. Here is the way.
• Say that πα (s) is a PPR vector.
• Then, create a set CS by doing a sweep cut over πα (s).
That is, do the usual sweep cut, except on the vector returned by the algorithm, rather than the
leading nontrivial eigenvector of the Laplacian.
Here is one such lemma that can be proved about such a local spectral algorithm.
Lemma 26. If πα (s) is a PPR vector with ksk1 ≤ 1 and there exists S ⊆ V and a constant δ such
that πα (s)~1S − Vol(S) > δ, then
Vol(G)

hCs

 1/2
 p
12α log 4 Vol(S)/δ
 .
<
δ


This too is a complicated lemma, but the point is that if there exists a set S where the PPR is
d
, then a sweep cut over πα (S) produces a set
much larger than the stationary distribution
Vol
 (G)
p
α log (Vol(S)) .
CS with low Cheeger ratio: O

This result is for PPR; that there is a similar result for approximate PPR, which has a more
complicated statement still.
Two things to note.
• While the statements of these theoretical results is quite complicated, these methods do very
well in practice in many applications. (We won’t discuss this much.)
• These algorithms were originally used as primitives for the Laplacian solvers, a topic to which
we will return in a few weeks.

While one can prove results about the output of these algorithms, one might also be interested
in what these algorithms optimize. Clearly, they approximately optimize something related to a
local version of the global spectral partitioning objective; so the question is what do they optimize
exactly. This is the topic we will turn to next—it will turn out that there are interesting connections
here to with implicit regularization ideas.

190

M. W. Mahoney

20

(04/07/2015): Local Spectral Methods (3 of 4): An optimization perspective on local spectral methods

Reading for today.
• “A Local Spectral Method for Graphs: with Applications to Improving Graph Partitions and
Exploring Data Graphs Locally,” in JMLR, by Mahoney, Orecchia, and Vishnoi
Last time, we considered local spectral methods that involve short random walks started at a small
set of localized seed nodes. Several things are worth noting about this.
• The basic idea is that these random walks tend to get trapped in good conductance clusters,
if there is a good conductance cluster around the seed node. A similar statement holds for
approximate localized random walks, e.g., the ACL push procedure—meaning, in particular,
that one can implement them “quickly,” e.g., with the push algorithm, without even touching
all of the nodes in G.
• The exact statement of the theorems that can be proven about how these procedures can
be used to find good locally-biased clusters is quite technically complicated—since, e.g., one
could step outside of the initial set of nodes if one starts near the boundary—certainly the
statement is much more complicated than that for the vanilla global spectral method.
• The global spectral method is on the one hand a fairly straightforward algorithm (compute
an eigenvector or some other related vector and then perform a sweep cut with it) and on
the other hand a fairly straightforward objective (optimize the Rayleigh quotient variance
subject to a few reasonable constraints).
• Global spectral methods often do very well in practice.
• Local spectral methods often do very well in practice.
• A natural question is: what objective do local spectral methods optimize—exactly, not approximately? Or, relatedly, can one construct an objective that is quickly-solvable and that
also comes with similar locally-biased Cheeger-like guarantees?
To this end, today we will present a local spectral ansatz that will have several appealing properties:
• It can be computed fairly quickly, as a PPR.
• It comes with locally-biased Cheeger-like guarantees.
• It has the same form as several of the semi-supervised objectives we discussed.
• Its solution touches all the nodes of the input graph (and thus it is not as quick to compute
as the push procedure which does not).
• The strongly local spectral methods that don’t touch all of the nodes of the input graph are
basically ℓ1 -regularized variants of it.

Lecture Notes on Spectral Graph Methods

20.1

191

A locally-biased spectral ansatz

Here is what we would like to do.
• We would like to introduce an ansatz for an objective function for locally-biased spectral
graph partitioning. This objective function should be a locally-biased version of the usual
global spectral partitioning objective; its optimum should be relatively-quickly computable; it
should be useful to highlight locally-interesting properties in large data graphs; and it should
have some connection to the local spectral algorithms that we have been discussing.
In addition to having an optimization formulation of locally-biased spectral partitioning methods,
there are at least two reasons one would be interested in such an objective.
• A small sparse cut might be poorly correlated with the second (or even all) global eigenvectors
of L, and so it might be invisible to global spectral methods.
• We might have exogenous information about a specific region of a large graph in which we
are most interested, and so we might want a method that finds clusters near that region, e.g.,
to do exploratory data analysis.
Here is the approach we will take.
• We will start with the usual global spectral partitioning objective function and add to it a
certain locality constraint.
• This program will be a non-convex problem (as is the global spectral partitioning problem),
but its solution will be computable as a linear equation that is a generalization of the PR
spectral ranking method.
• In addition, we will show that it can be used to find locally biased partitions near an input
seed node, it has connections with the ACL push-based local spectral method, etc.
Let’s set notation. The Laplacian is L = D−A; and the normalized
Laplacian is L = D −1/2 LD −1/2 .
Pn
T
The degree-weighted inner product is given by x Dy = i=1 xi yi di . In this case, the weighted
complete graph is given by
1
D11T D,
AKn =
Vol(G)
in which case DKn = DG and thus
LKn = DKn − AKn = DG −

1
D11T D.
Vol(G)

Given this notation, see the left panel of Figure 1 for the usual spectral program Spectral(G),
and see the right panel of Figure 1 for LocalSpectral(G, s, κ), a locally-biased spectral program.
(Later, we’ll call the LocalSpectral objective “MOV,” to compare and contrast it with the “ACL”
push procedure.)

192

M. W. Mahoney

min
s.t.

min

T

x LG x

s.t.

T

x DG x = 1
T

xT L G x
xT DG x = 1
(xT DG 1)2 = 0

2

(x DG 1) = 0

(xT DG s)2 ≥ κ

x ∈ RV

x ∈ RV

Figure 1: Global and local spectral optimization programs. Left: The usual spectral program
Spectral(G). Right: A locally-biased spectral program LocalSpectral(G, s, κ). In both cases, the
optimization variable is the vector x ∈ Rn .
In the above, we assume WLOG that
s is such that



sT Ds = 1
sT D1 = 0

.

This “WLOG” just says that one can subtract off the part of s along the all-ones vector; we could
have parametrized the problem to include this component and gotten similar results to what we
will present below, had we not done this. Note that s can actually be any vector (that isn’t in the
span of the all-ones vector); but it is convenient to think of it as an indicator vector of a small
“seed set” of nodes S ⊂ V .
2
√
The constraint xT Ds ≥ κ says that the projection of the solution x is at least κ in absolute
value, where κ ∈ (0, 1). Here is the interpretation of this constraint.
√
• The vector x must be in a spherical cap centered at s with angle at most arccos ( κ) from s.
• Higher values of κ correspond to finding a vector that is more well-correlated with the seed
vector. While the technical details are very different than with strongly local spectral methods
such as ACL, informally one should think of this as corresponding to shorter random walks
or, relatedly, higher values of the teleportation parameter that teleports the walk back to the
original seed set of nodes.
• If κ = 0, then there is no correlation constraint, in which case we recover Spectral(G).

20.2

A geometric notion of correlation

Although LocalSpectral is just an objective function and no geometry is explicitly imposed, there
is a geometric interpretation of this in terms of a geometric notion of correlation between cuts in
G. Let’s make explicit the geometric notion of correlation between cuts (or, equivalently, between
partitions, or sets of nodes) that is used by LocalSpectral.
Given a cut (T, T̄ ) in a graph G = (V, E), a natural vector in Rn to associate with it is its
indicator/characteristic vector, in which case the correlation between a cut (T, T̄ ) and another cut
(U, Ū ) can be captured by the inner product of the characteristic vectors of the two cuts. Since we
are working on the space orthogonal to the degree-weighted all-ones vector, we’ll do this after we
remove from the characteristic vector its projection along the all-ones vector. In that case, again, a

Lecture Notes on Spectral Graph Methods

193

notion of correlation is related to the inner product of two such vectors for two cuts. More precisely,
given a set of nodes T ⊆ V , or equivalently a cut (T, T̄ ), one can define the unit vector sT as
r


1T
1T̄
vol(T )vol(T̄ )
def
,
sT =
−
2m
vol(T ) vol(T̄ )
in which case

 q
vol(T )vol(T̄ )
1

· vol(T
)
q 2m
sT (i) =
 − vol(T )vol(T̄ ) · 1
2m
vol(T̄ )

if i ∈ T

.

if i ∈ T̄

Several observations are immediate from this definition.

• One can replace sT by sT̄ and the correlation remains the same with any other set, and so
this is well-defined. Also, sT = −sT̄ ; but since here we only consider quadratic functions of
sT , we can consider both sT and sT̄ to be representative vectors for the cut (T, T̄ ).
• Defined this way, it immediately follows that sTT DG 1 = 0 and that sTT DG sT = 1. Thus,
sT ∈ SD for T ⊆ V , where we denote by SD the set of vectors {x ∈ RV : xT DG 1 = 0}; and
sT can be seen as an appropriately normalized version of the vector consisting of the uniform
distribution over T minus the uniform distribution over T̄ .
• One can introduce the following measure of correlation between two sets of nodes, or equivalently between two cuts, say a cut (T, T̄ ) and a cut (U, Ū ):
def

K(T, U ) = (sT DG sU )2 .
Then it is easy to show that: K(T, U ) ∈ [0, 1]; K(T, U ) = 1 if and only if T = U or T̄ = U ;
K(T, U ) = K(T̄ , U ); and K(T, U ) = K(T, Ū ).
• Although we have described this notion of geometric correlation in terms of vectors of the
form sT ∈ SD that represent partitions (T, T̄ ), this correlation is clearly well-defined for other
vectors s ∈ SD for which there is not such a simple interpretation in terms of cuts.
Below we will show that the solution to LocalSpectral can be characterized in terms of a PPR
vector. If we were interested in objectives that had solutions of different forms, e.g., the form of a
heat kernel, then this would correspond to an objective function with a different constraint, and
this would then imply a different form of correlation.

20.3

Solution of LocalSpectral

Here is the basic theorem characterizing the form of the solution of LocalSpectral.
Theorem 39 (Solution Characterization). Let s ∈ Rn be a seed vector such that sT DG 1 = 0,
sT DG s = 1, and sT DG v2 6= 0, where v2 is the second generalized eigenvector of LG with respect to
DG . In addition, let 1 > κ ≥ 0 be a correlation parameter, and let x⋆ be an optimal solution to
LocalSpectral(G, s, κ). Then, there exists some γ ∈ (−∞, λ2 (G)) and a c ∈ [0, ∞] such that
x⋆ = c(LG − γDG )+ DG s.

(46)

194

M. W. Mahoney

Before presenting the proof of this theorem, here are several things to note.
• s and κ are the parameters of the program; c is a normalization factor that rescales the norm
of the solution vector to be 1 (and that can be computed in linear time, given the solution
vector); and γ is implicitly defined by κ, G, and s.
• The correct setting of γ ensures that (sT DG x⋆ )2 = κ, i.e., that x⋆ is found exactly on the
boundary of the feasible region.
• x⋆ and γ change as κ changes. In particular, as κ goes to 1, γ tends to −∞ and x⋆ approaches
s; conversely, as κ goes to 0, γ goes to λ2 (G) and x⋆ tends towards v2 , the global eigenvector.
• For a fixedchoice of G, s, and
 κ, an ǫ-approximate solution to LocalSpectral can be computed

m
1
· log( ǫ ) using the Conjugate Gradient Method; or in time Õ m log( 1ǫ )
in time Õ √
λ2 (G)

using the Spielman-Teng linear-equation solver (that we will discuss in a few weeks), where
the Õ notation hides log log(n) factors. This is true for a fixed value of γ, and the correct
setting of γ can be found by binary search.
While that is theoretically true, and while there is a lot of work recently on developing
practically-fast nearly-linear-time Laplacian-based solvers, this approach might not be appropriate in certain applications. For example, in many applications, one has precomputed
an eigenvector decomposition of LG , and then one can use those vectors and obtain an approximate solution with a small number of inner products. This can often be much faster
in practice.

In particular, solving LocalSpectral is not “fast” in the sense of the original local spectral methods,
i.e., in that the running time of those methods depends on the size of the output and doesn’t depend
on the size of the graph. But the running time to solve LocalSpectral is fast, in that its solution
depends essentially on computing a leading eigenvector of a Laplacian L and/or can be solved with
“nearly linear time” solvers that we will discuss in a few weeks.
While Eqn. (46) is written in the form of a linear equation, there is a close connection between the
solution vector x⋆ and the Personalized PageRank (PPR) spectral ranking procedure.
• Given a vector s ∈ Rn and a teleportation constant α > 0, the PPR vector can be written as
−1

1−α
DG
DG s.
prα,s = LG +
α
By setting γ = − 1−α
α , one can see that the optimal solution to LocalSpectral is proved to be
a generalization PPR.
• In particular, this means that for high values of the correlation parameter κ for which the
corresponding γ satisfies γ < 0, the optimal solution to LocalSpectral takes the form of a
PPR vector. On the other hand, when γ ≥ 0, the optimal solution to LocalSpectral provides
a smooth way of transitioning from the PPR vector to the global second eigenvector v2 .
• Another way to interpret this is to say that for values of κ such that γ < 0, then one could
compute the solution to LocalSpectral with a random walk or by solving a linear equation,
while for values of κ for which γ > 0, one can only compute the solution by solving a linear
equation and not by performing a random walk.

Lecture Notes on Spectral Graph Methods

minimize
s.t.

LG ◦ X

195

maximize

LK n ◦ X = 1

(DG s)(DG s)T ◦ X ≥ κ

α + κβ

s.t. LG  αLKn + β(DG s)(DG s)T

X0

β≥0

α∈R

Figure 2: Left: Primal SDP relaxation of LocalSpectral(G, s, κ): SDPp (G, s, κ). For this primal,
the optimization variable is X ∈ Rn×n such that X is SPSD. Right: Dual SDP relaxation of
LocalSpectral(G, s, κ): SDPd (G, s, κ). For this dual, the optimization variables are α, β ∈ R.
About the last point, we have talked about how random walks compute regularized or robust
versions of the leading nontrivial eigenvector of L—it would be interesting to characterize an algorithmic/statistical tradeoff here, e.g., if/how in this context certain classes of random walk based
algorithms are less powerful algorithmically than related classes of linear equation based algorithms
but that they implicitly compute regularized solutions more quickly for the parameter values for
which they are able to compute solutions.

20.4

Proof of Theorem 39

Here is an outline of the proof, which essentially involves “lifting” a rank-one constraint to obtain
an SDP in order to get strong duality to apply.
• Although LocalSpectral is not a convex optimization problem, it can be relaxed to an SDP
that is convex.
• From strong duality and complementary slackness, the solution to the SDP is rank one.
• Thus, the vector making up the rank-one component of this rank-one solution is the solution
to LocalSpectral.
• The form of this vector is of the form of a PPR.
Here are some more details. Consider the primal SDPp and dual SDPd SDPs, given in the left panel
and right panel, respectively, of Figure 2.

Here are a sequence of claims.
Claim 13. The primal SDP, SDPp is a relaxation of LocalSpectral.
Proof. Consider x ∈ Rn , a feasible vector for LocalSpectral. Then, the SPSD matrix X = xxT is
feasible for SDPp .
Claim 14. Strong duality holds between SDPp and SDPd .

196

M. W. Mahoney

Proof. The program SDPp is convex, and so it suffices to check that Slater’s constraint qualification
conditions hold for SDPp . To do so, consider X = ssT . Then,
2
(DG s) (DG s)T ◦ ssT = sT DG s = 1 > κ.
Claim 15. The following feasibility and complementary slackness conditions are sufficient for a
primal-dual pair X ∗ , α∗ , β ∗ to be an optimal solution. The feasibility conditions are:
LKn ◦ X ⋆ = 1,

(DG s)(DG s)T ◦ X ⋆ ≥ κ,

LG − α⋆ LKn − β ⋆ (DG s)(DG s)T
β

⋆

 0, and

(47)

≥ 0,

and the complementary slackness conditions are:
α⋆ (LKn ◦ X ⋆ − 1) = 0,

⋆

β ⋆ ((DG s)(DG s)T ◦ X ⋆ − κ) = 0, and
⋆

⋆

T

X ◦ (LG − α LKn − β (DG s)(DG s) ) = 0.

(48)
(49)

Proof. This follows from the convexity of SDPp and Slater’s condition.
Claim 16. The feasibility and complementary slackness conditions, coupled with the assumptions
of the theorem, imply that X ∗ is rank one and that β ∗ ≥ 0.
Proof. If we plug v2 in Eqn. (47), then we obtain that v2T LG v2 − α⋆ − β ⋆ (v2T DG s)2 ≥ 0.
But v2T LG v2 = λ2 (G) and β ⋆ ≥ 0. Hence, λ2 (G) ≥ α⋆ . Suppose α⋆ = λ2 (G). As sT DG v2 6= 0, it
must be the case that β ⋆ = 0. Hence, by Equation (49), we must have X ⋆ ◦ L(G) = λ2 (G), which
implies that X ⋆ = v2 v2T , i.e., the optimum for LocalSpectral is the global eigenvector v2 . This
corresponds to a choice of γ = λ2 (G) and c tending to infinity.
Otherwise, we may assume that α⋆ < λ2 (G). Hence, since G is connected and α⋆ < λ2 (G), LG −
α⋆ LKn has rank exactly n − 1 and kernel parallel to the vector 1.
From the complementary slackness condition (49) we can deduce that the image of X ⋆ is in the
kernel of LG − α⋆ LKn − β ⋆ (DG s)(DG s)T .
If β ⋆ > 0, we have that β ⋆ (DG s)(DG s)T is a rank one matrix and, since sT DG 1 = 0, it reduces
the rank of LG − α⋆ LKn by one precisely. If β ⋆ = 0 then X ⋆ must be 0 which is not possible if
SDPp (G, s, κ) is feasible.
Hence, the rank of LG − α⋆ LKn − β ⋆ (DG s)(DG s)T must be exactly n − 2. As we may assume that
1 is in the kernel of X ⋆ , X ⋆ must be of rank one. This proves the claim.
Remark. It would be nice to have a cleaner proof of this that is more intuitive and that doesn’t
rely on “boundary condition” arguments as much.
Now we complete the proof of the theorem. From the claim it follows that, X ⋆ = x⋆ x⋆T where x⋆
satisfies the equation
(LG − α⋆ LKn − β ⋆ (DG s)(DG s)T )x⋆ = 0.

Lecture Notes on Spectral Graph Methods

197

From the second complementary slackness condition, Equation (48), and the fact that β ⋆ > 0, we
√
√
obtain that (x⋆ )T DG s = ± κ. Thus, x⋆ = ±β ⋆ κ(LG − α⋆ LKn )+ DG s, as required.

20.5

Additional comments on the LocalSpectral optimization program

Here, we provide some additional discussion for this locally-biased spectral partitioning objective.
Recall that the proof we provided for Cheeger’s Inequality showed that in some sense the usual
global spectral methods “embed” the input graph G into a complete graph; we would like to say
something similar here.
To do so, observe that the dual of LocalSpectral is given by the following.
maximize α + βκ
s.t.

LG  αLKn + βΩT
β ≥ 0,

where ΩT = DG sT sTT DG . Alternatively, by subtracting the second constraint of LocalSpectral from
the first constraint, it follows that

xT LKn − LKn sT sTT LKn x ≤ 1 − κ.
Then it can be shown that

LKn − LKn sT sTT LKn =

LKT̄
LK T
,
+
vol(T
)
vol(T̄ )

where LKT is the DG -weighted complete graph on the vertex set T . Thus, LocalSpectral is equivalent to
minimize xT LG x
s.t.

xT L K n x = 1


LKT̄
LK T
T
x
+
x ≤ 1 − κ.
vol(T̄ ) vol(T )

The dual of this program is given by the following.
maximize α − β(1 − κ)
s.t.

LG  αLKn − β
β ≥ 0.



LKT̄
LK T
+
vol(T̄ ) vol(T )



Thus, from the perspective of this dual, LocalSpectral can be viewed as “embedding” a combination
of a complete graph Kn and a weighted combination of complete graphs on the sets T and T̄ , i.e.,
KT and KT̄ . Depending on the value of β, the latter terms clearly discourage cuts that substantially
cut into T or T̄ , thus encouraging partitions that are well-correlated with the input cut (T, T̄ ).
If we can establish a precise connection between the optimization-based LocalSpectral procedure and
operational diffusion-based procedures such as the ACL push procedure, then this would provide
additional insight as to “why” the short local random walks get stuck in small seed sets of nodes.
This will be one of the topics for next time.

198

M. W. Mahoney

21

(04/09/2015): Local Spectral Methods (4 of 4): Strongly and
weakly locally-biased graph partitioning

Reading for today.
• “Anti-differentiating Approximation Algorithms: A case study with Min-cuts, Spectral, and
Flow,” in ICML, by Gleich and Mahoney
• “Think Locally, Act Locally: The Detection of Small, Medium-Sized, and Large Communities
in Large Networks,” in PRE, by Jeub, Balachandran, Porter, Mucha, and Mahoney
Last time we introduced an objective function (LocalSpectral) that looked like the usual global
spectral partitioning problem, except that it had a locality constraint, and we showed that its
solution is of the form of a PPR vector. Today, we will do two things.
• We will introduce a locally-biased graph partitioning problem, we show that the solution to
LocalSpectral can be used to compute approximate solutions to that problem.
• We describe the relationship between this problem and what the strongly-local spectral methods, e.g., the ACL push method, compute.

21.1

Locally-biased graph partitioning

We start with a definition.
Definition 54 (Locally-biased graph partitioning problem.). Given a graph G = (V, E), an input
node u ∈ V , a number k ∈ Z+ , find a set of nodes T ⊂ V s.t.
φ(u, k) =

min

T ⊂V :u∈T,Vol(T )≤k

φ(T ),

i.e., find the best conductance set of nodes of volume not greater than k that contains the node u.
That is, rather than look for the best conductance cluster in the entire graph (which we considered
before), look instead for the best conductance cluster that contains a specified seed node and that
is not too large.
Before proceeding, let’s state a version of Cheeger’s Inequality that applies not just to the leading
nontrivial eigenvector of L but instead to any “test vector.”
Theorem 40. Let x ∈ Rn s.t. xT D~1 = 0. Then there exists a t ∈ [n] such that S ≡ SweepCutt (x) ≡
2
T Lx
≥ φ(S)
{i : xi ≥ t} satisfies xxT Dx
8 .
Remark. This form of Cheeger’s Inequality provides additional flexibility in at least two ways.
First, if one has computed an approximate Fiedler vector, e.g., by running a random walk many
steps but not quite to the asymptotic state, then one can appeal to this result to show that Cheegerlike guarantees hold for that vector, i.e., one can obtain a “quadratically-good” approximation to
the global conductance objective function using that vector. Alternatively, one can apply this to
any vector, e.g., a vector obtained by running a random walk just a few steps from a localized seed
node. This latter flexibility makes this form of Cheeger’s Inequality very useful for establishing
bounds with both strongly and weakly local spectral methods.

Lecture Notes on Spectral Graph Methods

199

Let’s also recall the objective with which we are working; we call it LocalSpectral(G, s, κ) or LocalSpectral. Here it is.
xT L G x

min

xT DG x = 1

s.t.

(xT DG 1)2 = 0
(xT DG s)2 ≥ κ
x ∈ Rn

Let’s start with our first result, which says that LocalSpectral is a relaxation of the intractable
combinatorial problem that is the locally-biased version of the global spectral partitioning problem
(in a manner analogous to how the global spectral partitioning problem is a relaxation of the
intractable problem of finding the best conductance partition in the entire graph). More precisely,
we can choose the seed set s and correlation parameter κ such that LocalSpectral(G, s, κ) is a
relaxation of the problem defined in Definition 54.
Theorem 41. For u ∈ V , LocalSpectral(G, v{u} , 1/k) is a relaxation of the problem of finding a
minimum conductance cut T in G which contains the vertex u and is of volume at most k. In
particular, λ(G, v{u} , 1/k) ≤ φ(u, k).
Proof. If we let x = vT in LocalSpectral(G, v{u} , 1/k), then vTT LG vT = φ(T ), vTT DG 1 = 0, and
vTT DG vT = 1. Moreover, we have that
(vTT DG v{u} )2 =

du (2m − vol(T ))
≥ 1/k,
vol(T )(2m − du )

which establishes the lemma.
Next, let’s apply sweep cut rounding to get locally-biased cuts that are quadratically good, thus
establishing a locally-biased analogue of the hard direction of Cheeger’s Inequality for this problem.
In particular, we can apply Theorem 40 to the optimal solution for LocalSpectral(G, v{u} , 1/k) and
obtain a cut T whose conductance is quadratically close to the optimal value λ(G, v{u} , 1/k). By
p
Theorem 41, this implies that φ(T ) ≤ O( φ(u, k)), which essentially establishes the following
theorem.
Theorem 42 (Finding a Cut). Given an unweighted graph G = (V, E),
p a vertex u ∈ V and a
positive integer k, we can find a cut in G of conductance at most O( φ(u, k)) by computing a
sweep cut of the optimal vector for LocalSpectral(G, v{u} , 1/k).
Remark. What this theorem states is that we can perform a sweep cut over the vector that is
the solution to LocalSpectral(G, v{u} , 1/k) in order to obtain a locally-biased partition; and that
this partition comes with quality-of-approximation guarantees analogous to that provided for the
global problem Spectral(G) by Cheeger’s inequality.
We can also use the optimal value of LocalSpectral to provide lower bounds on the conductance
value of other cuts, as a function of how well-correlated they are with the input seed vector s. In
particular, if the seed vector corresponds to a cut U , then we get lower bounds on the conductance
of other cuts T in terms of the correlation between U and T .

200

M. W. Mahoney

Theorem 43 (Cut Improvement). Let G be a graph and s ∈ Rn be such that sT DG 1 = 0, where
DG is the degree matrix of G. In addition, let κ ≥ 0 be a correlation parameter. Then, for all sets
def
T ⊆ V such that κ′ = (sT DG vT )2 , we have that
φ(T ) ≥



λ(G, s, κ)
if κ ≤ κ′
κ′
′
κ · λ(G, s, κ) if κ ≤ κ.

In particular, if s = sU for some U ⊆ V, then note that κ′ = K(U, T ).
Proof. It follows from the results that we established in the last class that λ(G, s, κ) is the same
as the optimal value of SDPp (G, s, κ) which, by strong duality, is the same as the optimal value
of SDPd (G, s, κ). Let α⋆ , β ⋆ be the optimal dual values to SDPd (G, s, κ). Then, from the dual
feasibility constraint LG − α⋆ LKn − β ⋆ (DG s)(DG s)T  0, it follows that
sTT LG sT − α⋆ sTT LKn sT − β ⋆ (sT DG sT )2 ≥ 0.
Notice that since sTT DG 1 = 0, it follows that sTT LKn sT = sTT DG sT = 1. Further, since sTT LG sT =
φ(T ), we obtain, if κ ≤ κ′ , that
φ(T ) ≥ α⋆ + β ⋆ (sT DG sT )2 ≥ α⋆ + β ⋆ κ = λ(G, s, κ).
If on the other hand, κ′ ≤ κ, then
φ(T ) ≥ α⋆ + β ⋆ (sT DG sT )2 ≥ α⋆ + β ⋆ κ ≥

κ′
κ′
· (α⋆ + β ⋆ κ) =
· λ(G, s, κ).
κ
κ

Finally, observe that if s = sU for some U ⊆ V, then (sTU DG sT )2 = K(U, T ). Note that strong
duality was used here.
Remark. We call this result a “cut improvement” result since it is the spectral analogue of the
flow-based “cut improvement” algorithms we mentioned when doing flow-based graph partitioning.
• These flow-based cut improvement algorithms were originally used as a post-processing algorithm to improve partitions found by other algorithms. For example, GGT, LR (Lang-Rao),
and AL (which we mentioned before).

• They provide guarantees of the form: for any cut C, C̄ that is ǫ-correlated with the input
cut, the cut output by the
 cut improvement algorithm has conductance ≤ some function of
the conductance of C, C̄ and ǫ.

• Theorem 43 shows that, while the cut value output by this spectral-based “improvement”
algorithm might not be improved, relative to the input, as they are often guaranteed to do
with flow-based cut-improvement algorithms, they do not decrease in quality too much, and
in addition one can make claims about the cut quality of “nearby” cuts.

• Although we don’t have time to discuss it, these two operations can be viewed as building
blocks or “primitives” that can be combined in various ways to develop algorithms for other
problems, e.g., finding minimum conductance cuts.

Lecture Notes on Spectral Graph Methods

21.2

201

Relationship between strongly and weakly local spectral methods

So far, we have described two different ways to think about local spectral algorithms.
• Operational. This approach provides an algorithm, and one can prove locally-biased Cheegerlike guarantees. The exact statement of these results is quite complex, but the running time
of these methods is extremely fast since they don’t even need to touch all the nodes of a
big graph.
• Optimization. This approach provides a well-defined optimization objective, and one can
prove locally-biased Cheeger-like guarantees. The exact statement of these results is much
simpler, but the running time is only moderately fast, since it involves computing eigenvectors
or linear equations on sparse graphs, and this involves at least touching all the nodes of a
big graph.
An obvious question here is the following.
• Shat is the precise relationship between these two approaches?
We’ll answer this question by considering the weakly-local LocalSpectral optimization problem (that
we’ll call MOV below) and the PPR-based local spectral algorithm due to ACL (that we’ll call ACL
below). What we’ll show is roughly the following.
• We’ll show roughly that if MOV optimizes an ℓ2 based penalty, then ACL optimizes an
ℓ1 -regularized version of that ℓ2 penalty.
That’s interesting since ℓ1 regularization is often introduced to enforce or encourage sparsity. Of
course, there is no ℓ1 regularization in the statement of the strongly local spectral methods like
ACL, but clearly they enforce some sort of sparsity, since they don’t even touch most of the
nodes of a large graph. Thus, this result can be interpreted as providing an implicit regularization
characterization of a fast approximation algorithm.

21.3

Setup for implicit ℓ1 regularization in strongly local spectral methods

Recall that L = D − A = B T CB, where B is the unweighted edge-incidence matrix. Then
X
kBxkC,1 =
C(ij) |xi − xj | = cut(S),
(ij)∈E

where S = {i : xi = 1}. In addition, we can obtain a spectral problem by changing k · k1 → k · k2
to get
X
kBxk2C,2 =
C(ij) (xi − xj )2
(ij)∈E

Let’s consider a specific (s, t)-cut problem that is inspired by the AL FlowImprove procedure. To
do so, fix a set of vertices (like we did when we did the semi-supervised eigenvector construction),
and define a new graph that we will call the “localized cut graph.” Basically, this new graph will
be the original graph augmented with two additional nodes, call them s and t, that are connected
by weights to the nodes of the original graph. Here is the definition.

202

M. W. Mahoney

Definition 55 (localized cut graph). Let G = (V, E) be a graph, let S be a set of vertices, possibly
empty, let S̄ be the complement set, and let α be a non-negative constant. Then the localized cut
graph is the weighted, undirected graph with adjacency matrix:


0
αdTS
0
A αdS̄ 
AS =  αdS
0
αdTS̄
0
where dS = DeS is a degree vector localized on the set S, A is the adjacency matrix of the original
graph G, and α ≥ 0 is a non-negative weight. Note that the first vertex is s and the last vertex is t.

We’ll use the α and S parameter to denote the matrices for the localized cut graph. For example,
the incidence matrix B(S) of the localized cut graph, which depends on the set S, is given by
the following.


e −IS 0
B(S) =  0 B 0  ,
0 −IS̄ e

where, recall, the variable IS are the columns of the identity matrix corresponding to vertices in S.
The edge-weights of the localized cut graph are given by the diagonal matrix C(α), which depends
on the value α.
Given this, recall that the 1-norm formulation of the LP for the min-s, t-cut problem, i.e., the
minimum weighted s, t cut in the flow graph, is given by the following.
min

kBxkC(α),1

s.t. xs = 1, xt = 0, x ≥ 0.
Here is a theorem that shows that PageRank implicitly solves a 2-norm variation of the 1-norm
formulation of the s, t-cut problem.
Theorem 44. Let B(S) be the incidence matrix for the localized cut graph, and C(α) be the edgeweight matrix. The PageRank vector z that solves
(αD + L)z = αv
with v = dS /vol(S) is a renormalized solution of the 2-norm cut computation:
min kB(S)xkC(α),2

(50)

s.t. xs = 1, xt = 0.

Specifically, if x(α, S) is the solution of Prob. (50), then


1
x(α, S) =  vol(S)z  .
0
Proof. The key idea is that the 2-norm problem corresponds with a quadratic objective, which
PageRank solves. The quadratic objective for the 2-norm approximate cut is:
kB(S)xk2C(α),2 = xT B(S)T C(α)B(S)x


αvol(S) −αdTS
0
= xT  −αdS L + αD −αdS̄  x.
0
−αdS̄ αvol(S̄)

Lecture Notes on Spectral Graph Methods

203

If we apply the constraints that xs = 1 and xt = 0 and let xG be the free set of variables, then we
arrive at the unconstrained objective:



αvol(S) −αdTS
0
1


1 xTG 0  −αdS L + αD −αdS̄   xG 
0
−αdS̄ αvol(S̄)
0
= xTG (L + αD)xG − 2αxTG dS + αvol(S).

Here, the solution xG solves the linear system
(αD + L)xG = αdS .
The vector xG = vol(S)z, where z is the solution of the PageRank problem defined in the theorem,
which concludes the proof.
Theorem 44 essentially says that for each PR problem, there is a related cut/flow problem that
“gives rise” to it. One can also establish the reverse relationship that extracts a cut/flow problem
from any PageRank problem.
To show this, first note that the proof of Theorem 44 works since the edges we added had weights
proportional to the degree of the node, and hence the increase to the degree of the nodes was
proportional to their current degree. This causes the diagonal of the Laplacian matrix of the
localized cut graph to become αD + D. This idea forms the basis of our subsequent analysis. For a
general PageRank problem, however, we require a slightly more general definition of the localized
cut graph, which we call a PageRank cut graph. Here is the definition.
Definition 56. Let G = (V, E) be a graph, and let s ≥ 0 be a vector such that d − s ≥ 0. Let s
connect to each node in G with weights given by the vector αs, and let t connect to each node in
G with weights given by α(d − s). Then the PageRank cut graph is the weighted, undirected graph
with adjacency matrix:


0
αsT
0
A(s) =  αs
A
α(d − s)  .
T
0 α(d − s)
0
We use B(s) to refer to the incidence matrix of this PageRank cut graph. Note that if s = dS , then
this is simply the original construction.
With this, we state the following theorem, which is a sort of converse to Theorem 44. The proof is
similar to that of Theorem 44 and so it is omitted.
Theorem 45. Consider any PageRank problem that fits the framework of
(I − βP T )x = (1 − β)v.
The PageRank vector z that solves
(αD + L)z = αv
is a renormalized solution of the 2-norm cut computation:
min kB(s)xkC(α),2

xs = 1, xt = 0

(51)

204

M. W. Mahoney

with s = v. Specifically, if x(α, S) is the solution of the 2-norm cut, then
 
1
x(α, s) =  z  .
0
Two things are worth noting about this result.
• A corollary of this result is the following: if s = e, then the solution of a 2-norm cut is a
reweighted, renormalized solution of PageRank with v = e/n. That is, as a corollary of this
approach, the standard PageRank problem with v = e/n gives rise to a cut problem where s
connects to each node with weight α and t connects to each node v with weight α(dv − 1).
• This also holds for the semi-supervised learning results we discussed. In particular, e.g., the
procedure of Zhou et al. for semi-supervised learning on graphs solves the following:
(I − βD −1/2 AD −1/2 )−1 Y.
(The other procedures solve a very similar problem.) This is exactly a PageRank equation for
a degree-based scaling of the labels, and thus the construction from Theorem 45 is directly
applicable.

21.4

Implicit ℓ1 regularization in strongly local spectral methods

In light of these results, let’s now move onto the ACL procedure. We will show a connection
between it and an ℓ1 regularized version of an ℓ2 objective, as established in Theorem 45. In
particular, we will show that the ACL procedure for approximating a PPR vector exactly computes
a hybrid 1-norm 2-norm variant of the min-cut problem. The balance between these two terms (the
ℓ2 term from Problem 51 and an additional ℓ1 term) has the effect of producing sparse PageRank
solutions that also have sparse truncated residuals, and it also provides an interesting connection
with ℓ1 -regularized ℓ2 -regression problems.
We start by reviewing the ACL method and describing it in such a way to make these connections
easier to establish.
Consider the problem (I − βAD −1 )x = (1 − β)v, where v = ei is localized onto a single node. In
addition to the PageRank parameter β, the procedure has two parameters: τ > 0 is a accuracy
parameter that determines when to stop, and 0 < ρ ≤ 1 is an additional approximation term
that we introduce. As τ → 0, the computed solution x goes to the PPR vector that is non-zero
everywhere. The value of ρ has been 1/2 in most previous implementations of the procedure; and
here we present a modified procedure that makes the effect of ρ explicit.
I. x(1) = 0, r (1) = (1 − β)ei , k = 1
II. while any rj > τ dj
III.

IV.

(where dj is the degree of node j)

x(k+1) = x(k) + (rj − τ dj ρ)ej


τ dj ρ
(k+1)
ri
= ri(k) + β(rj − τ dj ρ)/dj

 (k)
ri

i=j
i∼j

otherwise

Lecture Notes on Spectral Graph Methods
V.

205

k ←k+1

As we have noted previously, one of the important properties of this procedure is that the algorithm
maintains the invariant r = (1− β)v − (I − βAD −1 )x throughout. For any 0 ≤ ρ ≤ 1, this algorithm
converges because the sum of entries in the residual always decreases monotonically. At the solution
we will have
0 ≤ r ≤ τ d,
which provides an ∞-norm style worst-case approximation guarantee to the exact PageRank solution.
Consider the following theorem. In the same way that Theorem 45 establishes that a PageRank
vector can be interpreted as optimizing an ℓ2 objective involving the edge-incidence matrix, the
following theorem establishes that, in the case that ρ = 1, the ACL procedure to approximate
this vector can be interpreted as solving an ℓ1 -regularized ℓ2 objective. That is, in addition to
approximating the solution to the objective function that is optimized by the PPR, this algorithm
also exactly computes the solution to an ℓ1 regularized version of the same objective.
Theorem 46. Fix a subset of vertices S. Let x be the output from the ACL procedure with ρ = 1,
0 < β < 1, v = dS /vol(S), and τ fixed. Set α = 1−β
β , κ = τ vol(S)/β, and let zG be the solution on
graph vertices of the sparsity-regularized cut problem:
min
s.t.


1
2
2 kB(s)zkC(α),2

+ κkDzk1

(52)

zs = 1, zt = 0, z ≥ 0,



1
where z =  zG  as above. Then x = DzG /vol(S).
0
Proof. If we expand the objective function and apply the constraint zs = 1, zt = 0, then Prob. (52)
becomes:
min
s.t.

1 T
2 zG (αD

T d + α2 vol(S) + κdT z
+ L)zG − αzG
S
G

(53)

zG ≥ 0

Consider the optimality conditions of this quadratic problem (where s are the Lagrange multipliers):
0 = (αD + L)zG − αdS̄ + κd − s
s≥0

zG ≥ 0

T
zG
s = 0.

These are both necessary and sufficient because (αD + L) is positive definite. In addition, and for
the same reason, the solution is unique.
In the remainder of the proof, we demonstrate that vector x produced by the ACL method satisfies
these conditions. To do so, we first translate the optimality conditions to the equivalent PageRank
normalization:
0 = (I − βAD −1 )DzG /vol(S) − (1 − β)dS /vol(S) + βκ/vol(S)d − βs/vol(S)
s≥0

zG ≥ 0

T
zG
s = 0.

206

M. W. Mahoney

When the ACL procedure finishes with β, ρ, and τ as in the theorem, the vectors x and r satisfy:
r = (1 − β)v − (I − βAD −1 )x

x≥0

0 ≤ r ≤ τ d = βκ/vol(S)d.

Thus, if we set s such that βs/vol(S) = βκ/vol(S)d − r, then we satisfy the first condition with
x = DzG /vol(S). All of these transformations preserve x ≥ 0 and zG ≥ 0. Also, because τ d ≥ r,
T s = 0.
we also have s ≥ 0. What remains to be shown is zG
T s = 0 because the non-zero
Here, we show xT (τ d − r) = 0, which is equivalent to the condition zG
structure of the vectors is identical. Orthogonal non-zero structure suffices because zG s = 0 is
equivalent to either xi = 0 or τ di − ri = 0 (or both) for all i. If xi 6= 0, then at some point in
the execution, the vertex i was chosen at the step rj > τ dj . In that iteration, we set ri = τ di . If
any other step increments ri , we must revisit this step and set ri = τ di again. Then at a solution,
xi 6= 0 requires ri = τ di . For such a component, si = 0, using the definition above. For xi = 0, the
value of si is irrelevant, and thus, we have xT (τ d − r) = 0.

Remark. Finally, a comment about ρ, which is set to 1 in this theorem but equals 1/2 in most
prior uses of the ACL push method. The proof of Theorem 46 makes the role of ρ clear. If ρ < 1,
then the output from ACL is not equivalent to the solution of Prob. (52), i.e., the renormalized
T s = 0; but setting ρ < 1, however, will compute a solution much more
solution will not satisfy zG
rapidly. It is a nice open problem to get a clean statement of implicit regularization when ρ < 1.

Lecture Notes on Spectral Graph Methods

22

207

(04/14/2015): Some Statistical Inference Issues (1 of 3): Introduction and Overview

Reading for today.
• “Towards a theoretical foundation for Laplacian-based manifold methods,” in JCSS, by Belkin
and Niyogi

22.1

Overview of some statistical inference issues

So far, most of what we have been doing on spectral methods has focused on various sorts of
algorithms—often but not necessarily worst-case algorithms. That is, there has been a bias toward
algorithms that are more rather than less well-motivated statistically—but there hasn’t been a lot
statistical emphasis per se. Instead, most of the statistical arguments have been informal and by
analogy, e.g., if the data are nice, then one should obtain some sort of smoothness, and Laplacians
achieve that in a certain sense; or diffusions on graphs should look like diffusions on low-dimensional
spaces or a complete graph; or diffusions are robust analogues of eigenvectors, which we illustrated
in several ways; and so on.
Now, we will spend a few classes trying to make this statistical connection a little more precise. As
you can imagine, this is a large area, and we will only be able to scratch the surface, but we will try
to give an idea of the space, as well as some of the gotchas of naively applying existing statistical
or algorithmic methods here—so think of this as pointing to lots of interesting open questions to
do statistically-principled large-scale computing, rather than the final word on the topic.
From a statistical perspective, many of the issues that arise are somewhat different than much of
what we have been considering.
• Computation is much less important (but perhaps it should be much more so).
• Typically, one has some sort of model (usually explicit, but sometimes implicit, as we saw
with the statistical characterization of the implicit regularization of diffusion-based methods),
and one wants to compute something that is optimal for that model.
• In this case, one might want to show things like convergence or consistence (basically, that
what is being computed on the empirical data converges to the answer that is expected, as
the number of data points n → ∞).
For spectral methods, at a high level, there are basically two types of reference states or classes of
models that are commonly-used: one is with respect to some sort of very low-dimensional space;
and the other is with respect to some sort of random graph model.
• Low-dimensional spaces. In this simplest case, this is a line; more generally, this is a
low-dimensional linear subspace; and, more generally, this is a low-dimensional manifold.
Informally, one should think of low-dimensional manifolds in this context as basically lowdimensional spaces that are curved a bit; or, relatedly, that the data are low-dimensional,
but perhaps not in the original representation. This manifold perspective provides added

208

M. W. Mahoney
descriptive flexibility, and it permits one to take advantage of connections between the geometry of continuous spaces and graphs (which in very special cases are a discretization of those
continuous places).
• Random graphs. In the simples case, this is simply the Gnm or Gnp Erdos-Renyi (ER)
random graph. More generally, one is interested in finding clusters, and so one works with
the stochastic blockmodel (which can be thought of as a bunch of ER graphs pasted together).
Of course, there are many other extensions of basic random graph models, e.g., to include
degree variability, latent factors, multiple cluster types, etc.

These two places provide two simple reference states for statistical claims about spectral graph
methods; and the types of guarantees one obtains are somewhat different, depending on which of
these reference states is assumed. Interestingly, (and, perhaps, not surprisingly) these two places
have a direct connection with the two complementary places (line graphs and expanders) that
spectral methods implicitly embed the data.
In both of these cases, one looks for theorems of the form: “If the data are drawn from this place and
things are extremely nice (e.g., lots of data and not too much noise) then good things happen (e.g.,
finding the leading vector, recovering hypothesized clusters, etc.) if you run a spectral method.
We will cover several examples of this. A real challenge arises when you have realistic noise and
sparsity properties in the data, and this is a topic of ongoing research.
As just alluded to, another issue that arises is that one needs to specify not only the hypothesized
statistical model (some type of low-dimensional manifold or some type of random graph model
here) but also one needs to specify exactly what is the problem one wants to solve. Here are
several examples.
• One can ask to recover the objective function value of the objective you write down.
• One can ask to recover the leading nontrivial eigenvector of the data.
• One can ask to converge to the Laplacian of the hypothesized model.
• One can ask to find clusters that are present in the hypothesized model.
The first bullet above is most like what we have been discussing so far. In most cases, however, people want to use the solution to that objective for something else, and the other bullets
are examples of that. Typically in these cases one is asking for a lot more than the objective
function value, e.g., one wants to recover the “certificate” or actual solution vector achieving the
optimum, or some function of it like the clusters that are found by sweeping along it, and so one
needs stronger assumptions. Importantly, many of the convergence and statistical issues are quite
different, depending on the exact problem being considered.
• Today, we will assume that the data points are drawn from a low-dimensional manifold and
that from the empirical point cloud of data we construct an empirical graph Laplacian; and
we will ask how this empirical Laplacian relates to the Laplacian operator on the manifold.
• Next time, we will ask whether spectral clustering is consistent in the sense that it converges
to something meaningful and n → ∞, and we will provide sufficient conditions for this (and
we will see that the seemingly-minor details of the differences between unnormalized spectral
clustering and normalized spectral clustering lead to very different statistical results).

Lecture Notes on Spectral Graph Methods

209

• On the day after that, we will consider results for spectral clustering in the stochastic blockmodel, for both vanilla situations as well as for situations in which the data are
very sparse.

22.2

Introduction to manifold issues

Manifold-based ML is an area that has received a lot of attention recently, but for what we will
discuss today one should think back to the discussion we had of Laplacian Eigenmaps. At root,
this method defines a set of features that can then be used for various tasks such as data set
parametrization, clustering, classification, etc. Often the features are useful, but sometimes they
are not; here are several examples of when the features developed by LE and related methods are
often less than useful.
• Global eigenvectors are localized. In this case, “slowly-varying” functions (by the usual
precise definition) are not so slowly-varying (in a sense that most people would find intuitive).
• Global eigenvectors are not useful. This may arise if one is interested in a small local
part of the graph and if information of interest is not well-correlated with the leading or with
any eigenvector.
• Data are not meaningfully low-dimensional. Even if one believes that there is some
sort of hypothesized curved low-dimensional space, there may not be a small number of
eigenvectors that capture most of this information. (This does not necessarily mean that the
data are “high rank,” since it is possible that the spectrum decays, just very slowly.) This is
more common for very sparse and noisy data, which are of course very common.
Note that the locally-biased learning methods we described, e.g., the LocalSpectral procedure, the
PPR procedure, etc., was motivated by one common situation when the global methods such as
LE and related methods had challenges.
While it may be fine to have a “feature generation machine,” most people prefer some sort of
theoretical justification that says when a method works in some idealized situation. To that end,
many of the methods like LE assume that the data are drawn from some sort of low-dimensional
manifold. Today, we will talk about one statistical aspect of that having to do with converging to
the manifold.
To start, here is a simple version of the “manifold story” for a classification problem. Consider a
2-class classification problem with classes C1 and C2 , where the data elements are drawn from some
space X , whose elements are to be classified. A statistical or probabilistic model typically includes
the following two ingredients: a probability density p(x) on X ; and class densities {p (Ci |x ∈ X )},
for i ∈ {1, 2}. Importantly, if there are unlabeled data, then the unlabeled data don’t tell us
much about the conditional class distributions, as we can’t identify classes without labels, but the
unlabeled data can help us to improve our estimate of the probability distribution p(x). That is,
the unlabeled data tell us about p(x), and the labeled data tell us about {p (Ci |x ∈ X )}.
If we say that the data come from a low-dimensional manifold X , then a natural geometric object
to consider is the Laplace-Beltrami operator on X . In particular, let M ⊂ Rn be an n-dimensional
compact manifold isometrically embedded in Rk . (Think of this as an n-dimensional “surface” in
Rk .) The Riemannian structure on M induces a volume form that allows us to integrate functions

210

M. W. Mahoney

defined on M. The square-integrable functions form a Hilbert space L2 (M). Let C ∞ (M) be the
space of infinitely-differentiable functions on M. Then, the Laplace-Beltrami operator is a second
order differentiable operator ∆M : C ∞ (M) → C ∞ (M). We will define this in more detail below;
∂2
for now, just note that if the manifold is Rn , then the Laplace-Beltrami operator is ∆ = − ∂x
2.
1
There are two important properties of the Laplace-Beltrami operator.
• It provides a basis for L2 (M). In general, ∆ is a PSD self-adjoint operator (w.r.t.
the L2 inner product) on twice differentiable functions. In addition, if M is a compact
manifold, then ∆ has a discrete spectrum, the smallest eigenvalue of ∆ equals 0 and the
associated eigenfunction is the constant eigenfunction, and the eigenfunctions of ∆ provide
an orthonormal basis for the
space L2 (M). In that case, any function f ∈ L2 (M)
PHilbert
∞
can be written as f (x) =
i=1 ai ei (x), where ei are the eigenfunctions of ∆, i.e., where
∆ei = λi ei .

In this case, then the simplest model for the classification problem is that the class membership is a square-integrable function, call it m : M → {−1, +1}, in which case the classification
problem can be interpreted as interpolating a function
on the manifold. Then we can choose
P
the coefficients to get an optimal fit, m(x) = ni=1 ai ei , in the same way as we might approximate a signal with a Fourier series. (In fact, if M is a unit circle, call it S 1 , then
2 f (θ)
2 2
∆S1 f (θ) = − d dθ
2 , and the eigenfunctions are sinusoids with eigenvalues {1 , 2 , . . .}. and
we get the usual Fourier series.)

• It provides a smoothness functional. Recall that a simple measure of the degree of
smoothness for a function f on the unit circle S 1 is
Z
|f (θ)′ |2 dθ.
S(f ) =
S1

In particular, f is smooth iff this is close to zero. If we take this expression and integrate by
parts, then we get
Z
Z
′
f ∆f dθ = h∆f, f iL2 (S 1 ) .
f (θ)dθ =
S(f ) =
S1

S1

More generally, if f : M → R, then it follows that
Z
Z
f ∆f dµ = h∆f, f iL2(M) .
|∇f |2 dµ =
S(f ) =
M

M

So, in particular, the smoothness of the eigenfunction is controlled by the eigenvalue, i.e.,
S(ei ) = h∆ei , ei iL2 (M) = λi ,
P
and for arbitrary f that can be expressed as f = i αi ei , we have that
+
*
X
X
X
λi α2i .
S(f ) = h∆f, f i =
αi ∆ei ,
αi ei =
i

i

i

(So, in particular, approximating a function f by its first k eigenfunctions is a way to control
the smoothness of the eigenfunctions; and the linear subspace where the smoothness functions
is finite is a RKHS.)

Lecture Notes on Spectral Graph Methods

211

This has strong connections with a range of RKHS problems. (Recall that a RKHS is a Hilbert
space of functions where the evaluation functionals, the functionals that evaluate functions at a
point, are bounded linear functionals.) Since the Laplace-Beltrami operator on M can be used to
provide a basis for L2 (M), we can take various classes of functions that are defined on the manifold
and solve problems of the form
X
(yi − f (xi ))2 + λG(f ),
(54)
min
f ∈H

i

where H : M → R. In general, the first term is the empirical risk, and
R the second term
P is2 a
stabilizer orPregularization term. As anP
example, one could choose G(f ) = M h∇f, ∇f i = i αi λi
(since f = i αi ei (x)), and H = {f = i αi ei |G(f ) < ∞}, in which case one gets an optimization
problem that is quadratic in the α variables.
As an aside that is relevant to what we discussed last week with the heat kernel, let’s go through
the construction of a RKHS that is invariantly defined on the
P manifold M. To do so, let’s fix an
infinite sequence of non-negative numbers {µi |i ∈ Z+ } s.t.
i µi < ∞ (as we will consider in the
examples below). Then, define the following linear space of continuous functions
)
(
X
X α2
i
<∞ .
H= f=
αi fi |
µi
i

i

P
P
P αi βi
Then, we can define the inner product as: for f = i αi fi and
P g = i βi gi , we have hf, gi = i µi .
Then, H is a RKHS with the following kernel: K(p, q) = i µi ei (p)ei (q). Then, given this, we can
solve regularized optimization problems of the form given in Eqn. (54) above. In addition, we can
get other choices of kernels by using different choices of µ vectors. For example, if we let µi = e−tλi ,
where λi are the eigenvalues of ∆, then we get the heat kernel corresponding to heat diffusion on
the manifold; if we let µi = 0, for all i > i∗ , then we are solving an optimization problem in a finite
dimensional space; and so on.
All of this discussion has been for data drawn from an hypothesized manifold M. Since we are
interested in a smoothness measure for functions for a graph, then if we think of the graph as a
model for the manifold, then we want the value of a function not to change too much between
points. In that case, we get
X
SG (f ) =
Wij (fi − fj ) ,
i j

and it can be shown that
SG (f ) = f Lf T = hf, Lf iG =

n
X
i=1

λi hf, ei iG .

Of course, this is the discrete object with which we have been working all along. Viewed from
the manifold perspective, this corresponds to the discrete analogue of the integration by parts
we performed above. In addition, we can use all of this to consider questions having to do with
“regularization on manifolds and graphs,” as we have allude to in the past.
To make this connection somewhat more precise, recall that for a RKHS, there exists a kernel
K : X × X → R such that f (x) = hf (·), K(x, ·)iH . For us today, the domain X could be a manifold
M (in which case we are interested in kernels K : M × M → R), or it could be points from
Rn (say, on the nodes of the graph that was constructed from the empirical original data by a

212

M. W. Mahoney

nearest neighbor rule, in which case we are interested in kernels K : Rn × Rn → R). We haven’t
said anything precise yet about how these two relate, so now let’s turn to that and ask about
connections between kernels constructed from these two different places, as n → ∞.

22.3

Convergence of Laplacians, setup and background

Now let’s look at questions of convergence.
If H : M → R is a RKHS invariantly defined on M, then the key goal is to minimize regularized
risk functionals of the form
i
h
Eλ = min E (y − f (x))2 + λkf k2H .
f ∈H

In principle, we can do this—if we had an infinite amount of data available and the true manifold
is known. Instead, we minimize the empirical risk which is of the form
Êλ,n = min
f ∈H

1X
(yi − f (xi ))2 + λkf kH .
n

The big question is: how far is Êλ,n from Eλ .
The point here is the following: assuming the manifold is known or can be estimated from the
data, then making this connection is a relatively-straightforward application of Hoeffding bounds
and regularization/stability ideas. But:
• In theory, establishing convergence to the hypothesized manifold is challenging. We will get
to this below.
• In practice, testing the hypothesis that the data are drawn from a manifold in some meaningful
sense of the word is harder still. (For some reason, this question is not asked in this area.
It’s worth thinking about what would be test statistics to validate or invalidate the manifold
hypothesis, e.g., is that the best conductance clusters are not well balanced sufficient to
invalidate it?)
So, the goal here is to describe conditions under which the point cloud in X of the sample points
converges to the Laplace-Beltrami operator on the underlying hypothesized manifold M. From
this perspective, the primary data are points in X , that is assumed to be drawn from an underlying
manifold, with uniform or nonuniform density, and we want to make the claim that the Adjacency
Matrix or Laplacian Matrix of the empirical data converges to that of the manifold. (That is,
the data are not a graph, as will arise with the discussion of the stochastic block model.) In
particular, the graph is and empirical object, and if we view spectral graph algorithms as applying
to that empirical object then they are stochastically justified when they can relate to the underlying
processes generating the data.
What we will describe today is the following.
• For data drawn from a uniform distribution on a manifold M, the graph Laplacian converges
to the Laplace-Beltrami operator, as n → ∞ and the kernel bandwidth is chosen appropriately
(where the convergence is uniform over points on the manifold and for a class of functions).

Lecture Notes on Spectral Graph Methods

213

• The same argument applies for arbitrary probability distributions, except that one converges
to a weighted Laplacian; and in this case the weights can be removed to obtain convergence
to the normalized Laplacian. (Reweighting can be done in other ways to converge to other
quantities of interest, but we won’t discuss that in detail.)
Consider a compact smooth manifold M isometrically embedded in Rn . The embedding induces a
measure corresponding to volume form µ on the manifold (e.g., the volume form for a closed curve,
i.e., an embedding of the circle, measures the usual curve length in Rn ). The Laplace-Beltrami
operator ∆M is the key geometric object associated to a Riemannian manifold. Given ρ ∈ M, the
tangent space Tρ M can be identified with the affine space to tangent vectors to M at ρ. (This
vector space has a natural inner product induced by embedding M ⊂ Rn .) So, given a differentiable
function f : M → R, let ∇M f be the gradient vector on M (where ∇M f (p) points in the direction
of fastest ascent of f at ρ. Here is the definition.
Definition 57. The Laplace-Beltrami operator ∆M is the divergence of the gradient, i.e.,
∆M f = −div (∇M f ) .
Alternatively, ∆M can be defined as the unique operator s.t., for all two differentiable functions f
and h,
Z
Z
h∇M h(x), ∇M f (x)i dµ,
h(x)∆M f (x)dµ(x) =
M

M

where the inner product is on the tangent space and µ is the uniform measure.
In Rn , we have ∆f = −

P

∂2f
i ∂x2i .

More generally, on a k-dimensional manifold M, in a local

coordinate system (x1 , . . . , xn ), with a metric tensor gij , if g ij are the components of the inverse of
the metric tensor, then the Laplace-Beltrami operator applied to a function f is
!
X ∂
X
p
1
∂f
∆M f = p
det(g)
gij
∂xj
∂xi
det(g)
j

i

(If the manifold has nonuniform measure ν, given by dν(x) = P (x)dµ(x), for some function P (x)
and with dµ being the canonical measure corresponding to the volume form, then we have the more
1
general notion of a weighted manifold Laplacian: ∆M,µ = ∆P f = P (x)
div (P (x)∇M f ).)
The question is how to reconstruct ∆M , given a finite sample of data points from the manifold?
Here are the basic objects (in addition to ∆M ) that are used to answer this question.
• Empirical Graph Laplacian. Given
construct a weighted graph with weights


Ltn ij =

a sample of n points xi , . . . , xn from M, we can
2
Wij = e−kxi −xj k /4t , and then
−W
P ij
k Wik

if i 6= j
.
if i = j

Call Ltn the graph Laplacian matrix. We can think of Ltn as an operation of functions on the
n empirical data points:
X
X
2
2
Ltn f (xi ) = f (xi )
e−kxi −xj k /(4t) −
f (xj )e−kxi −xj k /(4t) ,
j

j

but this operator operates only on the empirical data, i.e., it says nothing about other points
from M or the ambient space in which M is embedded.

214

M. W. Mahoney
• Point Cloud Laplace operator. This formulation extends the previous results to any
function on the ambient space. Denote this by Ltn to get
Ltn f (x) = f (x)

1 X −kx−xj k2 /(4t) 1 X
2
e
−
f (xj )e−kx−xj k /(4t)
n
n
j

j

(So, in particular, when evaluated on the empirical data points, we have that Ltn f (xi ) =
t
1 T
n Ln f (xi ).) Call Ln the Laplacian associated to the point cloud x1 , . . . , xn .
• Functional approximation to the Laplace-Beltrami operator. Given a measure ν on
M, we can construct an operator
Z
Z
2
−kx−yk2 /(4t)
t
f (y)e−kx−yk /(4t) dν(y).
e
dν(y) −
L f (x) = f (x)
M

M

Observe that
on x1 , . . . , xn .

22.4

Ltn

t

is just a special form of L , corresponding to the Dirac measure supported

Convergence of Laplacians, main result and discussion

The main result they describe is to establish a connection between the graph Laplacian associated
to a point cloud (which is an extension of the graph Laplacian from the empirical data points to
the ambient space) and the Laplace-Beltrami operator on the underlying manifold M. Here is the
main results.
Theorem 47. Let x1 , . . . , xn be data points sampled from a uniform distribution on the manifold
M ⊂ Rn . Choose tn = n−1/(k+2+α) , for α > 0, and let f ∈ C ∞ (M). Then
lim

1

n→∞ tn (4πtn )k/2

Ltnn f (x) =

1
∆M f (x),
Vol (M)

where the limit is taken in probability and Vol (M) is the volume of the manifold with respect to the
canonical measure.
We are not going to go through the proof in detail, but we will outline some key ideas used in the
proof. Before doing that, here are some things to note.
• This theorem assert pointwise convergence of Ltn f (p) to ∆M f (p), for a fixed function f and
a fixed point p.
• Uniformity over all p ∈ M follows almost immediately from the compactness of M.
• Uniform convergence over a class of function, e.g., functions C k (M) with bounded kth derivative, follows with more effort.
• One can consider a more general probability distribution P on M according to which data
points are sampled—we will get back to an example of this below.
For the proof, the easier part is to show that Ltn → Lt , as n → ∞, if points are samples uniformly:
this uses some basic concentration results. The harder part is to connect Lt and ∆M : what must
be shown is that when t → 0, then Lt appropriately scaled converges to ∆M .

Lecture Notes on Spectral Graph Methods

215

Here are the basic proof ideas, which exploit heavily connections with the heat equation on M.
For simplicity, consider first Rn , where we have the following theorem.
Theorem 48 (Solution to heat equation on Rk ). Let f (x) be a sufficiently differentiable bounded
function. Then
Z
kx−yk2
e− 4t f (y)dy,
H t f = (4πt)−k/2
Rk

and
t

f (x) = lim H f (x) = (4πt)
t→0

−k/2

Z

e−

kx−yk2
4t

f (y)dy,

Rk

and the function u(x, t) = H t f satisfies the heat equation
∂
u(x, t) + ∆u(x, t) = 0
∂t
with initial condition u(x, 0) = f (x).
This result for the heat equation is the key result for approximating the Laplace operator.
∂
u(x, t)|t=0
∂t
∂
= − H t f (x)|t=0
∂t

1
= lim
f (x) − H t f (x) .
t→0 t

∆f (x) = −

By this last result, we have a scheme for approximating the Laplace operator. To do so, recall that
the heat kernel is the Gaussian that integrates to 1, and so


Z
Z
−kx−yk2
−kx−yk2
1
−k/2
−k/2
4t
4t
e
f (y)dy − f (x) (4πt)
dy.
∆f (x) = lim −
e
(4πt)
t→0
t
Rk
Rk
It can be shown that this can be approximated by the point cloud x1 , . . . , xn by computing the
empirical version as
!
−k/2
X −kx−xi k2 X −kx−xi k2
1
(4πt)
ˆ (x) =
∆f
f (x)
e 4t
−
e 4t f (xi )
t
n
i

=

1

t (4πt)

i

Lt f (x).
k/2 n

It is relatively straightforward to extend this to a convergence result for Rk . To extend it to a
convergence result for arbitrary manifolds M, two issues arise:
t (x, y). (It has
• With very few exceptions, we don’t know the exact form of the heat kernel HM
the nice form of a Gaussian for M = Rk .)

• Even asymptotic forms of the heat kernel requires knowing the geodesic distance between
points in the point cloud, but we can only observe distance in the ambient space.

216

M. W. Mahoney

See their paper for how they deal with these two issues; this involves methods from differential
geometry that are very nice but that are not directly relevant to what we are doing.
Next, what about sampling with respect to nonuniform probability distributions? Using the above
proof, we can establish that we converge to a weighted Laplacian. If this is not of interest, then
once can instead normalize differently and get one of two results.
• The weighted scaling factors can be removed by using a different normalization of the weights
of the point cloud. This different normalization basically amounts to considering the normalized Laplacian. See below.
• With yet a different normalization, we can recover the Laplace-Beltrami operator on the
manifold. The significance of this is that it is possible to separate geometric aspects of the
manifold from the probability distribution on it. This is of interest to harmonic analysts, and
it underlies extension of the Diffusion Maps beyond the Laplacian Eigenmaps.
As for the first point, if we have a compact Riemannian manifold M and a probability distribution
P : M → R+ according to which points are drawn in an i.i.d. fashion. Assume that a ≤ P (x) ≤ b,
for all x ∈ M. Then, define the point cloud Laplacian operator as
n

Ltn f (x) =

1X
W (xi , xj ) (f (x) − f (xi ))
n
i=1

kx−xi k

If W (x, xi ) = e 4t , then this corresponds to the operator we described above. In order to
normalized the weights, let
1
Gt (x, xi )
q
W (x, xi ) = q
,
t ˆ
ˆ
dt (x) dt (xi )
where

Gt (x, xi ) =
dˆt (x) =

kx−xi k2

1

e− 4t ,
k/2
(4πt)
1X
Gt (x, xj ), and
n
j6=i

dˆt (xi ) =

1 X
Gt (xi , xj ),
n−1
j6=i

where the latter two quantities are empirical estimates of the degree function dt (x), where
Z
Gt (x, y)P (y)Vol(y).
dt (x) =
M

Note that we get a degree function—which is a continuous function defined on M. This function
bears some resemblance to the diagonal degree matrix of a graph, and it can be thought of as
a multiplication operator, but it has very different properties than an integral operator like the
heat kernel. We will see this same function next time, and this will be important for when we get
consistency with normalized versus unnormalized spectral clustering.

Lecture Notes on Spectral Graph Methods

23

217

(04/16/2015): Some Statistical Inference Issues (2 of 3): Convergence and consistency questions

Reading for today.
• “Consistency of spectral clustering,” in Annals of Statistics, by von Luxburg, Belkin, and
Bousquet
Last time, we talked about whether the Laplacian constructed from point clouds converged to the
Laplace-Beltrami operator on the manifold from which the data were drawn, under the assumption
that the unseen hypothesized data points are drawn from a probability distribution that is supported
on a low-dimensional Riemannian manifold. While potentially interesting, that result is a little
unsatisfactory for a number of reasons, basically since one typically does not test the hypothesis
that the underlying manifold even exists, and since the result doesn’t imply anything statistical
about cluster quality or prediction quality or some other inferential goal. For example, if one is
going to use the Laplacian for spectral clustering, then probably a more interesting question is to
ask whether the actual clusters that are identified make any sense, e.g., do they converge, are they
consistent, etc. So, let’s consider these questions. Today and next time, we will do this in two
different ways.
• Today, we will address the question of the consistency of spectral clustering when there are
data points drawn from some space X and we have similarity/dissimilarity information about
the points. We will follow the paper “Consistency of spectral clustering,” by von Luxburg,
Belkin, and Bousquet.
• Next time, we will ask similar questions but for a slightly different data model, i.e., when the
data are from very simple random graph models. As we will see, some of the issues will be
similar to what we discuss today, but some of the issues will be different.
I’ll start today with some general discussion on: algorithmic versus statistical approaches; similarity
and dissimilarity functions; and embedding data in Hilbert versus Banach spaces. Although I
covered this in class briefly, for completeness I’ll go into more detail here.

23.1

Some general discussion on algorithmic versus statistical approaches

When discussing statistical issues, we need to say something about our model of the data generation
mechanism, and we will discuss one such model here. This is quite different than the algorithmic
perspective, and there are a few points that would be helpful to clarify.
To do so, let’s take a step back and ask: how are the data or training points generated? Here are
two possible answers.
• Deterministic setting. Here, someone just provides us with a fixed set of objects (consisting, e.g, of a set of vectors or a single graph) and we have to work with this particular set
of data. This setting is more like the algorithmic approach we have been adopting when we
prove worst-case bounds.
• Probabilistic setting. Here, we can consider the objects as a random sample generated
from some unknown probability distribution P . For example, this P could be on (Euclidean

218

M. W. Mahoney
or Hilbert or Banach or some other) space X . Alternatively, this P could be over random
graphs or stochastic blockmodels.

There are many differences between these two approaches. One is the question of what counts as
“full knowledge.” A related question has to do with the objective that is of interest.
• In the deterministic setting, the data at hand count as full knowledge, since they are all there
is. Thus, when one runs computations, one wants to make statements about the data at
hand, e.g., how close in quality is the output of an approximation algorithm to the output of
a more expensive exact computation.
• In the probabilistic setting, complete or full knowledge is to know P exactly, and the finite
sample contains only noisy information about P . Thus, when we run computations, we are
only secondarily interested in the data at hand, since we are more interested in P , or relatedly
in what we can say if we draw another noisy sample from P tomorrow.
Sometimes, people think of the deterministic setting as the probabilistic setting, in which the data
space equals the sample space and when one has sampled all the data. Sometimes this perspective
is useful, and sometimes it is not.
In either setting, one simple problem of potential interest (that we have been discussing) is clustering: given a training data (xi )i=1,...,n , where xi correspond to some features/patterns but for which
there are no labels available, the goal is to find some sort of meaningful clusters. Another problem
of potential interest is classification: given training points (xi , yi )i=1,...,n , where xi correspond to
some features/patterns and yi correspond to labels, the goal is to infer a rule to assign a correct y
to a new x. It is often said that, in some sense, in the supervised case, what we want to achieve is
well-understood, and we just need to specify how to achieve it; while in the latter case both what
we want to achieve as well as how we want to achieve it is not well-specified. This is a popular
view from statistics and ML; and, while it has some truth to it, it hides several things.
• In both cases, one specifies—implicitly or explicitly—an objective and tries to optimize it. In
particular, while the vague idea that we want to predict labels is reasonable, one obtains very
different objectives, and thus very different algorithmic and statistical properties, depending
on how sensitive one is to, e.g., false positives versus false negatives. Deciding on the precise
form of this can be as much of an art as deciding on an unsupervised clustering objective.
• The objective to be optimized could depend on just the data at hand, or it could depend on
some unseen hypothesized data (i.e., drawn from P ). In the supervised case, that might be
obvious; but even in the unsupervised case, one typically is not interested in the output per
se, but instead in using it for some downstream task (that is often not specified).
All that being said, it is clearly easier to validate the supervised case. But we have also seen
that the computations in the supervised case often boil down to computations that are identical
to computations that arise in the unsupervised case. For example, in both cases locally-biased
spectral ranking methods arise, but they arise for somewhat different reasons, and thus they are
used in somewhat different ways.
From the probabilistic perspective, due to randomness in the generation of the training set, it is
common to study ML algorithms from this statistical or probabilistic point of view and to model

Lecture Notes on Spectral Graph Methods

219

the data as coming from a probability space. For example, in the supervised case, the unseen data
are often modeled by a probability space of the form
((X × Y) , σ (BX × BY ) , P )
where X is the feature/pattern space and Y is the label space, BX and BY are σ-algebras on X
and Y, and P is a joint probability distribution on patterns and labels. (Don’t worry about the
σ-algebra and measure theoretic issues if you aren’t familiar with them, but note that P is the main
object of interest, and this is what we were talking about last time with labeled versus unlabeled
data.) The typical assumption in this case is that P is unknown, but that one can sample X × Y
from P . On the other hand, in the unsupervised case, there is no Y, and so in that case the unseen
data are more often modeled by a probability space of the form
(X , BX , P ) ,
in which case the data training points (xi )i=1,...,n are drawn from P .
From the probabilistic perspective, one is less interested in the objective function quality on the data
at hand, and instead one is often interested in finite-sample performance issues and/or asymptotic
convergence issues. For example, here are some questions of interest.
• Does the classification constructed by a given algorithm on a finite sample converge to a limit
classifier at n → ∞?
• If it converges, is the limit classifier the best possible; and if not, how suboptimal is it?
• How fast does convergence take place, as a function of increasing n?
• Can we estimate the difference between finite sample classifier and the optimal classifier, given
only the sample?
Today, we will look at the convergence of spectral clustering from this probabilistic perspective.
But first, let’s go into a little more detail about similarities and dissimilarities.

23.2

Some general discussion on similarities and dissimilarities

When applying all sorts of algorithms, and spectral algorithms in particular, MLers work with
some notion either of similarity or dissimilarity. For example, spectral clustering uses an adjacency
matrix, which is a sort of similarity function. Informally, a dissimilarity function is a notion that
is somewhat like a distance measure; and a similarity/affinity function measures similarities and is
sometimes thought about as a kernel matrix. Some of those intuitions map to what we have been
discussing, e.g., metrics and metric spaces, but in some cases there are differences.
Let’s start first with dissimilarity/distance functions. In ML, people are often a little less precise
than say in TCS; and—as used in ML—dissimilarity functions satisfy some or most or all of the
following, but typically at least the first two.
• (D1) d(x, x) = 0
• (D2) d(x, y) ≥ 0

220

M. W. Mahoney
• (D3) d(x, y) = d(y, x)
• (D4) d(x, y) = 0 ⇒ x = y
• (D5) d(x, y) + d(y, z) ≥ d(x, z)

Here are some things to note about dissimilarity and metric functions.
• Being more precise, a metric satisfies all of these conditions; and a semi-metric satisfies all
of these except for (D4).
• MLers are often interested in dissimilarity measures that do not satisfy (D3), e.g., the
Kullback-Leibler “distance.”
• There is also interest in cases where (D4) is not satisfied. In particular, the so-called cut
metric—which we used for flow-based graph partitioning—was a semi-metric.
• Condition (D4) says that if different points have distance equal to zero, then this implies
that they are really the same point. Clearly, if this is not satisfied, then one should expect
an algorithm should have difficulty discriminating points (in clustering, classification, etc.
problems) which have distance zero.
Here are some commonly used methods to transform non-metric dissimilarity functions into proper
metric functions.
˜ y) = |d(x, x0 ) − d(y, x0 )| is a
• If d is a distance function and x0 ∈ X is arbitrary, then d(x,
semi-metric on X .
• If (X , d) is a finite dissimilarity space with d symmetric and definite, then

d(x, y) + c
if x 6= y
˜
d=
,
0
if x = y
with c ≥ maxp,q,r∈X |d(p, q) + d(p, r) + d(r, q)|, is a metric.
• If D is a dissimilarity matrix, then there exists constants h and k such that the matrix with

1/2
elements d˜ij = d2ij + h
, for i 6= j, and also d¯ij = dij + k, for i 6= j, are Euclidean.

d
, for c ≥ 0 and r ≥ 1. If w : R → R is monotonically
• If d is a metric, so are d + c, d1/r , d+c
increasing function s.t. w(x) = 0 ⇐⇒ x = 0 and w(x + y) ≤ w(x) + w(y); then if d(·, ·) is a
metric, then w(d(·, ·)) is a metric.

Next, let’s go to similarity functions. As used in ML, similarity functions satisfy some subset of
the following.
• (S1) s(x, x) > 0
• (S2) s(x, y) = s(y, x)
• (S3) s(x, y) ≥ 0

Lecture Notes on Spectral Graph Methods
• (S4)

Pn

ij=1 ci cj s(xi , xj )

221

≥ 0, for all n ∈ N, ci ∈ R, xi ∈ X PSD.

Here are things to note about these similarity functions.
• The non-negativity is actually not satisfied by two examples of similarity functions that are
commonly used: correlation coefficients and scalar products
• One can transform a bounded similarity function to a nonnegative similarity function by
adding an offset: s(x, y) = s(x, y) + c for come c.
• If S is PSD, then it is a kernel. This is a rather strong requirement that is mainly satisfied
by scalar products in Hilbert spaces.
It is common to transform similarities to dissimilarities. Here are two ways to do that.
• If the similarity is a scalar product in a Euclidean space (i.e., PD), then one can compute the
metric
d(x, y)2 = hx − y, x − yi = hx, xi − 2 (x, yi + hy, yi .
• If the similarity function is normalized, i.e., 0 ≤ s(x, y) ≤ 1, and s(x, x) = 1, for all x, y, then
d = 1 − s is a distance.
It is also common to transform dissimilarities to similarities. Here are two ways to do that.
• If the distance is Euclidean, then one can compute a PD similarity
s(x, y) =


1
d(x, 0)2 + d(y, 0)2 − d(x, y)2 ,
2

where 0 ∈ X is an arbitrary origin.

• If d is a dissimilarity,
then a nonnegative decreasing function of d is a similarity, e.g., s(x, y) =

1
2
exp −d(x, y) /t , for t ∈ R, and also s(x, y) = 1−d(x,y)
.

These and related transformations are often used at the data preprocessing step, often in a somewhat ad hoc manner. Note, though, that the use of any one of them implies something about what
one thinks the data “looks like” as well as about how algorithms will perform on the data.

23.3

Some general discussion on embedding data in Hilbert and Banach spaces

Here, we discuss embedding data (in the form of similarity or dissimilarity functions) into Hilbert
and Banach spaces. To do so, we start with an informal definition (informal since the precise notion
of dissimilarity is a little vague, as discussed above).
Definition 58. A space (X , d) is a dissimilarity space or a metric space, depending on whether d
is a dissimilarity function or a metric function.

222

M. W. Mahoney

An important question for distance/metric functions, i.e., real metrics that satisfy the above conditions, is the following: when can a given metric space (X , d) be embedded isometrically in
Euclidean space H (or, slightly more generally, Hilbert space H). That is, the goal is to find a
mapping φ : X → H such that d(x, y) = kφ(x) − φ(y)k, for all x, y ∈ X . (While this was something we relaxed before, e.g., when we looked at flow-based algorithms and looked at relaxations
where there were distortions but they were not too too large, e.g., O(log n), asking for isometric
embeddings is more common in functional analysis.) To answer this question, note that distance
in Euclidean vector space satisfies (D1)–(D5), and so a necessary condition for the above is the
(D1)–(D5) be satisfied. The well-known Schoenberg theorem characterizes which metric spaces can
be isometrically embedded in Hilbert space.
Theorem 49. A metric space (X , d) can be embedded isometrically into Hilbert space iff −d2 is
conditionally positive definite, i.e., iff
−
for all ℓ ∈ N, xi , xj ∈ X , ci , cj ∈ R, with

ℓ
X

ij=1

P

ci cj d2 (xi , xj ) ≥ 0

i ci

= 0.

Informally, this says that Euclidean spaces and Hilbert spaces are not “big enough” for arbitrary
metric spaces. (We saw this before when we showed that constant degree expanders do not embed
well in Euclidean spaces.) More generally, though, isometric embeddings into certain Banach spaces
can be achieved for arbitrary metric spaces. (More on this later.) For completeness, we have the
following definition.
Definition 59. Let X be a vector space over C. Then X is a normed linear space if for all f ∈ X,
there exists a number, kf k ∈ R, called the norm of f s.t.: (1) kf k ≥ 0; (2) kf k = 0 iff f = 0; (3)
kcf k = |c|kf k, for all scalar c; (4) kf + gk ≤ kf k + kgk. A Banach space is a complete normed
linear space. A Hilbert space is a Banach space, whose norm is determined by an inner product.
This is a large area, most of which is off topic for us. If you are not familiar with it, just note that
RKHSs are particularly nice Hilbert spaces that are sufficiently heavily regularized that the nice
properties of Rn , for n < ∞, still hold; general infinite-dimensional Hilbert spaces are more general
and less well-behaved; and general Banach spaces are even more general and less well-behaved.
Since it is determined by an inner product, the norm for a Hilbert space is essentially an ℓ2 norm;
and so, if you are familiar with the ℓ1 or ℓ∞ norms and how they differ from the ℓ2 norm, then
that might help provide very rough intuition on how Banach spaces can be more general than
Hilbert spaces.

23.4

Overview of consistency of normalized and unnormalized Laplacian spectral methods

Today, we will look at the convergence of spectral clustering from this probabilistic perspective.
Following the von Luxburg, Belkin, and Bousquet paper, we will address the following two questions.
• Q1: Does spectral clustering converge to some limit clustering if more and more data points
are sampled and as n → ∞?

Lecture Notes on Spectral Graph Methods

223

• Q2: If it does converge, then is the limit clustering a useful partition of the input space from
which the data are drawn?
One reason for focusing on these questions is that it can be quite difficult to determine what is a
cluster and what is a good cluster, and so as a more modest goal one can ask for “consistency,”
i.e., that the clustering constructed on a finite sample drawn from some distribution converges to
a fixed limit clustering of the whole data space when n → ∞. Clearly, this notion is particularly
relevant in the probabilistic setting, since then we obtain a partitioning of the underlying space X
from which the data are drawn.
Informally, this will provide an “explanation” for why spectral clustering works. Importantly,
though, this consistency “explanation” will be very different than the “explanations” that have
been offered in the deterministic or algorithmic setting, where the data at hand represent full
knowledge. In particular, when just viewing the data at hand, we have provided the following
informal explanation of why spectral clustering works.
• Spectral clustering works since it wants to find clusters s.t. the probability of random walks
staying within a cluster is higher and the probability of going to the complement is smaller.
• Spectral clustering works since it approximates via Cheeger’s Inequality the intractable expansion/conductance objective.
In both of those cases, we are providing an explanation in terms of the data at hand; i.e., while we
might have an underlying space X in the back of our mind, they are statements about the data at
hand, or actually the graph constructed from the data at hand.
The answer to the above two questions (Q1 and Q2) will be basically the following.
• Spectral clustering with the normalized Laplacian is consistent under very general conditions.
For the normalized Laplacian, when it can be applied, then the corresponding clustering does
converge to a limit.
• Spectral clustering with the non-normalized Laplacian is not consistent, except under very
specific conditions. These conditions have to do with, e.g., variability in the degree distribution, and these conditions often do not hold in practice.
• In either case, if the method converges, then the limit does have intuitively appealing properties and splits the space X up into two pieces that are reasonable; but for the non-normalized
Laplacian one will obtain a trivial limit if the strong conditions are not satisfied.
As with last class, we won’t go through all the details, and instead the goal will be to show some
of the issues that arise and tools that are used if one wants to establish statistical results in this
area; and also to show you how things can “break down” in non-ideal situations.
To talk about convergence/consistency of spectral clustering, we need to make statements about
eigenvectors, and for this we need to use the spectral theory of bounded linear operators, i.e., methods from functional analysis. In particular, the information we will need will be somewhat different
than what we needed in the last class when we talked about the convergence of the Laplacian to
the hypothesized Laplace-Beltrami operator, but there will be some similarities. Today, we are
going to view the data points as coming from some Hilbert or Banach space, call in X , and from

224

M. W. Mahoney

these data points we will construct an empirical Laplacian. (Next time, we will consider graphs
that are directly constructed via random graph processes and stochastic block models.) The main
step today will be to establish the convergence of the eigenvalues and eigenvectors of random graph
Laplacian matrices for growing sample sizes. This boils down to questions of convergence of random
Laplacian matrices constructed from sample point sets.
(Note that although there has been a lot of work in random matrix theory on the convergence
of random matrices with i.i.d. entries or random matrices with fixed sample size, e.g., covariance
matrices, this work isn’t directly relevant here, basically since the random Laplacian matrix grows
with the sample size n and since the entries of the random Laplacian matrix are not independent.
Thus, more direct proof methods need to be used here.)
Assume we have a data space X = {x1 . . . . , xn } and a pairwise similarity k : X × X → R, which is
usually symmetric and nonnegative. For any fixed data set of n points, define the following:
• the Laplacian Ln = Dn − Kn ,
−1/2

• the normalized Laplacian L′n = Dn

−1/2

Ln Dn

, and

• the random walk Laplacian L′′n = Dn−1 Ln .
(Although it is different than what we used before, the notation of the von Luxburg, Belkin, and
Bousquet paper is what we will use here.) Note that here we assume that di > 0, for all i. We
are interested in computing the leading eigenvector or several of the leading eigenvectors of one of
these matrices and then clustering with them.
To see the kind of convergence result one could hope for, consider the second eigenvector (v1 , . . . , vn )T
of Ln , and let’s interpret is as a function fn on the discrete space Xn = {X1 , . . . , Xn } by defining
the function fn (Xi ) = vi . (This is the view we have been adopting all along.) Then, we can perform
clustering by performing a sweep cut, or we can cluster based on whether the value of fn is above or
below a certain threshold. Then, in the limit n → ∞, we would like fn → f , where f is a function
on the entire space X , such that we can threshold f to partition X .
To do this, we can do the following.
I. Choose this space to be C (X ), the space of continuous functions of X .
II. Construct a function d ∈ C (X ), a degree function, that is the “limit” as n → ∞ of the
discrete degree vector (d1 , . . . , dn ).
III. Construct linear operators U , U ′ , and U ′′ on C (X ) that are the limits of the discrete operators
Ln , L′n , and L′′n .
IV. Prove that certain eigenfunctions of the discrete operates “converge” to the eigenfunctions of
the limit operators.
V. Use the eigenfunctions of the limit operator to construct a partition for the entire space X .
We won’t get into details about the convergence properties here, but below we will highlight a
few interesting aspects of the limiting process. The main result they show is that in the case of
normalized spectral clustering, the limit behaves well, and things converge to a sensible partition of
the entire space; while in the case of unnormalized spectral clustering, the convergence properties
are much worse (for reasons that are interesting that we will describe).

Lecture Notes on Spectral Graph Methods

23.5

225

Details of consistency of normalized and unnormalized Laplacian spectral
methods

Here is an overview of the two main results in more detail.
Result 1. (Convergence of normalized spectral clustering.) Under mild assumptions, if the first r
eigenvalues of the limit operator U ′ satisfy λi 6= 1 and have multiplicity one, then
•
•
•
•

the same hold for the first r eigenvalues of L′n , as n → ∞;
the first r eigenvalues of L′n converge to the first r eigenvalues of U ′ ;
the corresponding eigenvectors converge; and
the clusters found from the first r eigenvectors on finite samples converge to a limit clustering
of the entire data space.

Result 2. (Convergence of unnormalized spectral clustering.) Under mild assumptions, if the first
r eigenvalues of the limit operator U do not lie in the range of the degree function d and have
multiplicity one, then
•
•
•
•

the same hold for the first r eigenvalues of Ln , as n → ∞;
the first r eigenvalues of Ln converge to the first r eigenvalues of U ;
the corresponding eigenvectors converge; and
the clusters found from the first r eigenvectors on finite samples converge to a limit clustering
of the entire data space.

Although both of these results have a similar structure (“if the inputs are nice, then one obtains
good clusters”), the “niceness” assumptions are very different: for normalized spectral clustering,
it is the rather innocuous assumption that λi 6= 1, while for unnormalized spectral clustering it is
the much stronger assumption that λi ∈ range(d). This assumption is necessary, as it is needed to
ensure that the eigenvalue λi is isolated in the spectrum of the limit operator. This is a requirement
to be able to apply perturbation theory to the convergence of eigenvectors. In particular, here is
another result.
Result 3. (The condition λ ∈
/ range(d) is necessary.)
• There exist similarity functions such that there exist no nonzero eigenvectors outside of
range(d).
• In this case, the sequence of second eigenvalues of n1 Ln converge to min d(x), and the corresponding eigenvectors do not yield a sensible clustering of the entire data space.
• For a wide class of similarity functions, there exist only finitely many eigenvalues r0 outside
of range(d), and the same problems arise if one clusters with r > r0 eigenfunctions.
• The condition λ ∈
/ range(d) refers to the limit and cannot be verified on a finite sample.
That is, unnormalized spectral clustering can fail completely, and one cannot detect it with a
finite sample.
The reason for the difference between the first results is the following.
• In the case of normalized spectral clustering, the limit operator U ′ has the form U ′ = I − T ,
where T is a compact linear operator. Thus, the spectrum of U ′ is well-behaved, and all the
eigenvalues λ 6= 1 are isolated and have finite multiplicity.
• In the case of unnormalized spectral clustering, the limit operator U has the form U = M −S,

226

M. W. Mahoney
where M is a multiplication operator, and S is a compact integral operator. Thus, the
spectrum of U is not as nice as that of U ′ , since it contains the interval range(d), and the
eigenvalues will be isolated only if λi 6= range(d).

Let’s get into more detail about how these differences arise. To do so, let’s make the following
assumptions about the data.
• The data space X is a compact metric space, B is the Borel σ-algebra on X , and P is a
probability measure on (X , B). We draw a sample of points (Xi )i∈N i.i.d. from P . The
similarity function k : X × X → R is symmetric, continuous, and there exists an ℓ > 0 such
that k(x, y) > ℓ, for all x, y ∈ X . (The assumption that f is bounded away from 0 is needed
due to the division in the normalized Laplacian.)
For f : X → R, we can denote the range of f by range(f ). Then, if X is connected and f is
continuous then range(f ) = [inf x f (x), supx f (x)]. Then we can define the following.
Definition 60. The restriction operator ρn : C (X ) → Rn denotes the random operator which
maps a function to its values on the first n data points, i.e.,
ρn (f ) = (f (X1 ), . . . , f (Xn ))T .
Here are some facts from spectral and perturbation theory of linear operators that are needed.
Let E be a real-valued Banach space, and let T : E → E be a bounded linear operator. Then, an
eigenvalue of T is defined to be a real or complex number λ such that
T f = λf, for some f ∈ E.
Note that λ is an eigenvalue of T iff the operator T − λ has a nontrivial kernel (recall that if
L : V → W then ker(L) = {v ∈ V : L(v) = 0}) or equivalently if T − λ is not injective (recall that
f : A → B is injective iff ∀a, b ∈ A we have that f (a) = f (b) ⇒ a = b, i.e., different elements of the
domain do not get mapped to the same element). Then, the resolvent of T is defined to be
ρ(T ) = {λ ∈ R : (λ − T )−1 exists and is bounded},
and the spectrum of T id defined to be
σ(T ) = R \ ρ(T ).
This holds very generally, and it is the way the spectrum is generalized in functional analysis.
(Note that if E is finite dimensional, then every non-invertible operator is not injective; and so
λ ∈ σ(T ) ⇒ λ is an eigenvalue of T . If E is infinite dimensional, this can fail; basically, one can
have operators that are injective but that have no bounded inverse, in which case the spectrum can
contain more than just eigenvalues.)
We can say that a point σiso ⊂ σ(T ) is isolated if there exists an open neighborhood ξ ⊂ C of σiso
such that σ(T ) ∩ ξ = (σiso ). If the spectrum σ(T ) of a bounded operator T in a Banach space
E consists of isolated parts, then for each isolated part of the spectrum, a spectral projection Piso
can be defined operationally as a path integral over the complex plane of a path Γ that encloses

Lecture Notes on Spectral Graph Methods

227

σiso and that separates it from the rest of σ(T ), i.e., for σiso ∈ σ(T ), the corresponding spectral
projection is
Z
1
(T − λI)−1 dλ,
Piso =
2πi Γ
where Γ is a closed Jordan curve in the complex plane separating σiso from the rest of the spectrum.
If λ is an isolated eigenvalue of σ(T ), then the dimension of the range of the spectral projection Pλ
is defined to be the algebraic multiplicity of λ, (for a finite dimensional Banach space, this is the
multiplicity of the root λ of the characteristic polynomial, as we saw before), and the geometric
multiplicity is the dimension of the eigenspace of λ.
One can split up the spectrum into two parts: the discrete spectrum σd (T) is the part of σ(T ) that
consists of isolated eigenvalues of T with finite algebraic multiplicity; and the essential spectrum
is σess (T ) = σ(T ) \ σd (T ). It is a fact that the essential spectrum cannot be changed by a finitedimensional or compact perturbation of an operator, i.e., for a bounded operator T and a compact
operator V , it holds that σess (T + V ) = σess (T ). The important point here is that one can define
spectral projections only for isolated parts of the spectrum of an operator and that these isolated
parts of the spectrum are the only parts to which perturbation theory can be applied.
Given this, one has perturbation results for compact operators. We aren’t going to state these
precisely, but the following is an informal statement.
• Let (E, k · kE ) be a Banach space, and (Tn )n and T bounded linear operators on E with
Tn → T . Let λ ∈ σ(T ) be an isolated eigenvalue with finite multiplicity m, and let ξ ⊂ C be
an open neighborhood of λ such that σ(T ) ∩ ξ = {λ}. Then,
– eigenvalues converge,
– spectral projections converge, and
– if λ is a simple eigenvalue, then the corresponding eigenvector converges.
We aren’t going to go through the details of their convergence argument, but we will discuss the
following issues.
The technical difficulty with proving convergence of normalized/unnormalized spectral clustering,
e.g., the convergence of (vn )n∈N or of (L′n )n∈N , is that for different sample sized n, the vectors vn
have different lengths and the matrices L′n have different dimensions, and so they “live” in different
spaces for different values of n. For this reason, one can’t apply the usual notions of convergence.
Instead, one must show that there exists functions f ∈ C (X ) such that kvn − ρn f k → 0, i.e., such
that the eigenvector vn and the restriction of f to the sample converge. Relatedly, one relates the
Laplacians to some other operator such that they are all defined on the same space. In particular,
one can define a sequence (Un ) of operators that are related to the matrices (Ln ); but each operator
(Un ) is defined on the space C(X ) of continuous functions on X , independent of n.
All this involves constructing various functions and operators on C (X ). There are basically two
types of operators, integral operators and multiplication operators, and they will enter in somewhat
different ways (that will be responsible for the difference in the convergence properties between
normalized and unnormalized spectral clustering). So, here are some basic facts about integral
operators and multiplication operators.
Definition 61. Let (X , B, µ) be a probability space, and let k ∈ L2 (X × X , B × B, µ × µ). Then,

228

M. W. Mahoney

the function S : L2 (X , B, µ) → L2 (X , B, µ) defined as
Z
k(x, y)f (y)dµ(y)
Sf (x) :
is an integral operator with kernel k.

X

If X is compact and k is continuous, then (among other things) the integral operator S is compact.
Definition 62. Let (X , B, µ) be a probability space, and let d ∈ L∞ (X , B, µ). Then a multiplication
operator Md : L2 (X , B, µ) → L2 (X , B, µ) is
Md f = f d.
This is a bounded linear operator; but if d is non-constant, then the operator Md is not compact.
Given the above two different types of operators, let’s introduce specific operators on C (X ) corresponding to matrices we are interested in. (In general, we will proceed by identifying vectors
(v1 , . . . , vn )T ∈ Rn with functions f ∈ C (X ) such that f (vi ) = vi and extending linear operators
on Rn to deal with such functions rather than vectors.)
Start with the unnormalized Laplacian:
P
Ln = Dn − Kn , where D = diag(di ), where di = ij K(xi , xj ).

We want to relate the degree vector (d1 , . . . , dn )T to a function on C (X ). To do so, define the true
and empirical degree functions:
Z
d(x) =
k(x, y)dP (y) ∈ C (X )
Z
dn (x) =
k(x, y)dPn (y) ∈ C (X )

(Note that dn → d as n → ∞ by a LLN.) By definition, dn (xi ) = n1 di , and so the empirical degree
function agrees with the degrees of the points Xi , up to the scaling n1 .

Next, we want to find an operator acting on C (X ) that behaves similarly to the matrix Dn on Rn .
Applying Dn to a vector f = (f1 , . . . , fn )T ∈ Rn gives (Dn f )i = di fi , i.e., each element is multiplied
by di . So, in particular, we can interpret n1 Dn as a multiplication operator. Thus, we can define
the true and empirical multiplication operators:
Md : C (X ) → C (X )

Mdn : C (X ) → C (X )

Md f (x) = d(x)f (x)
Mdn f (x) = dn (x)f (x)

Next, we will look at the matrix Kn . Applying it to a vector f ∈ Rn gives (Kn f )i =
Thus, we can define the empirical and true integral operator:
Z
Sn : C (X ) → C (X )
Sn f (x) = k(x, y)f (y)dPn (y)
Z
S : C (X ) → C (X )
Sn f (x) = k(x, y)f (y)dP (y)

P

j

K(xi , xj )fj .

With these definitions, we can define the empirical unnormalized graph Laplacian, Un : C (X ) →
C (X ), and the true unnormalized graph Laplacian, U : C (X ) → C (X ) as
Z
Un f (x) = Mdn f (x) − Sn f (x) = k(x, y) (f (x) − f (y)) dPn (y)
Z
U f (x) = Md f (x) − Sf (x) = k(x, y) (f (x) − f (y)) dP (y)

Lecture Notes on Spectral Graph Methods

229

For the normalized Laplacian, we can proceed as follows. Recall that v is an eigenvector of L′n with
eigenvalue v iff v is an eigenvector of Hn′ = D −1/2 Kn D −1/2 with eigenvalue 1 − λ. So, consider Hn′ ,
P K(x ,x )
defined as follows. The matrix Hn′ operates on a vector f = (f1 , . . . , fn )T as (Hn′ f )i = j √ i j .
di dj

Thus, we can define the normalized empirical and true similarity functions
p
hn (x, y) = k(x, y)/ dn (x)dn (y)
p
h(x, y) = k(x, y)/ d(x)d(y)

and introduce two integral operators

Tn : C (X ) → C (X )
T : C (X ) → C (X )

Z
Tn f (x) = hn (x, y)f (y)dPn (y)
Z
T f (x) = h(x, y)f (y)dP (y)

Note that for these operators the scaling factors n1 which are hidden in Pn and dn cancel each other.
Said another way, the matrix Hn′ already has n1 scaling factor—as opposed to the matrix Kn in the
unnormalized case. So, contrary to the unnormalized case, we do not have to scale matrices Hn′
and Hn with the n1 factor.
All of the above is machinery that enables us to transfer the problem of convergence of Laplacian
matrices to problems of convergence of sequences of operators on C (X ).
Given the above, they establish a lemma which, informally, says that under the general assumptions:
• the functions dn and d are continuous, bounded from below by ℓ > 0, and bounded from
above by kkk∞ ,
• all the operators are bounded,
• all the integral operators are compact,
• all the operator norms can be controlled.
The hard work is to show that the empirical quantities converge to the true quantities; this is done
with the perturbation result above (where, recall, the perturbation theory can be applied only to
isolated parts of the spectrum). In particular:
• In the normalized case, this is true if λ 6= 1 is an eigenvalue of U ′ that is of interest. The
reason is that U ′ = I − T ′ is a compact operator.
• In the unnormalized case, this is true if λ ∈
/ range(d) is an eigenvalue of U that is of interest.
The reason is that U = Md − S is not a compact operator, unless Md is a multiple of
the identity.
So, the key difference is the condition under which eigenvalues of the limit operator are isolated in
the spectrum: for the normalized case, this is true if λ 6= 1, while for the non normalized case, this
is true if λ ∈
/ range(d).
In addition to the “positive” results above, a “negative” result of the form given in the following
lemma can be established.
Lemma 27 (Clustering fails if λ ∈
/ range(d) is violated.). Assume that σ(U ) − {0} ∪ range(d) with
eigenvalue 0 having multiplicity 1, and that the probability distribution P on X has no point masses.

230

M. W. Mahoney

Then the sequence of second eigenvectors of n1 Ln converges to minx∈X d(x). The corresponding
eigenfunction will approximate the characteristic function of some x ∈ X , with d(x) = minx∈X d(x)
or a linear combination of such functions.
That is, in this case, the corresponding eigenfunction does not contain any useful information for
clustering (and one can’t even check if λ ∈ range(d) with a finite sample of data points).
While the analysis here has been somewhat abstract, the important point here is that this is not
a pathological situation: a very simple example of this failure is given in the paper; and this
phenomenon will arise whenever there is substantial degree heterogeneity, which is very common
in practice.

Lecture Notes on Spectral Graph Methods

24

231

(04/21/2015): Some Statistical Inference Issues (3 of 3): Stochastic blockmodels

Reading for today.
• “Spectral clustering and the high-dimensional stochastic blockmodel,” in The Annals of
Statistics, by Rohe, Chatterjee, and Yu
• “Regularized Spectral Clustering under the Degree-Corrected Stochastic Blockmodel,” in
NIPS, by Qin and Rohe
Today, we will finish up talking about statistical inference issues by discussing them in the context
of stochastic blockmodels. These are different models of data generation than we discussed in the
last few classes, and they illustrate somewhat different issues.

24.1

Introduction to stochastic block modeling

As opposed to working with expansion or conductance—or some other “edge counting” objective
like cut value, modularity, etc.—the stochastic block model (SBM) is an example of a so-called
probabilistic or generative model. Generative models are a popular way to encode assumptions
about the way that latent/unknown parameters interact to create edges (ij) Then, they assign a
probability value for each edges (ij) in a network. There are several advantages to this approach.
• It makes the assumptions about the world/data explicit. This is as opposed to encoding
them into an objective and/or approximation algorithm—we saw several examples of reverse
engineering the implicit properties of approximation algorithms.
• The parameters can sometimes be interpreted with respect to hypotheses about the network
structure.
• It allows us to use likelihood scores, to compare different parameterizations or different models.
• It allows us to estimate missing structures based on partial observations of graph structure.
There are also several disadvantages to this approach. The most obvious is the following.
• One must fit the model to the data, and fitting the model can be complicated and/or computationally expensive.
• As a result of this, various approximation algorithms are used to fit the parameters. This
in turn leads to the question of what is the effect of those approximations versus what is
the effect of the original hypothesized model? (I.e., we are back in the other case of reverse
engineering the implicit statistical properties underlying approximation algorithms, except
here it is in the approximation algorithm to estimate the parameters of a generative model.)
This problem is particularly acute for sparse and noisy data, as is common.
Like other generative models, SBMs define a probability distribution over graphs, P [G|Θ], where
Θ is a set of parameters that govern probabilities under the model. Given a specific Θ, we can then
draw or generate a graph G from the distribution by flipping appropriately-biased coins. Note that

232

M. W. Mahoney

inference is the reverse task: given a graph G, either just given to us or generated synthetically by a
model, we want to recover the model, i.e., we want to find the specific values of Θ that generated it.
The simpled version of a SBM is specified by the following.
• A positive integer k, a scalar value denoting the the number of blocks.
• A vector ~z ∈ Rn , where zi gives the group index of vertex i.
• A matrix M ∈ Rk×k , a stochastic block matrix, where Mij gives the probability that a vertex
of type i links to a vertex of type j.
Then, one generates edge (ij) with probability Mzi zj . That is, edges are not identically distributed,
but they are conditionally independent, i.e., conditioned on their types, all edges are independent,
and for a given pair of types (ij), edges are i.i.d.

Observe that the SBM has a relatively large number of parameters, k2 , even after we have chosen
the labeling on the vertices. This has plusses and minuses.
• Plus: it allows one the flexibility to model lots of possible structures and reproduce lots of
quantities of interest.
• Minus: it means that there is a lot of flexibility, thus making the possibility of overfitting
more likely.
Here are some simple examples of SBMs.
• If k = 1 and Mij = p, for all i, j, then we recover the vanilla ER model.
• Assortative networks, if Mii > Mij , for i 6= j.
• Disassortative networks, if Mii < Mij , for i 6= j.

24.2

Warming up with the simplest SBM

To illustrate some of the points we will make in a simple context, consider the ER model.
• If, say, p = 12 and the graph G has more than a handful of nodes, then it will be very easy
to estimate p, i.e., to estimate the parameter vector Θ of this simple SBM, basically since
measure concentration will occur very quickly and the empirical estimate of p we obtain
by counting the number of edges will be very close to its expected value, i.e., to p. More
generally, if n is large and p & log(n)
n , then measure will still concentrate, i.e., the empirical
and expected values of p will be close, and we will be able to estimate p well. (This is related
to the well-known observation that if p & log(n)
n , then Gnp and Gnm are very similar, for
appropriately chosen values of p and m.)
• If, on the other hand, say, p = n3 , then this is not true. In this regime, measure has not
concentrated for most statistics of interest: the graph is not even fully connected; the giant
component has nodes of degree almost O (log(n)); and the giant component has small sets

Lecture Notes on Spectral Graph Methods

233

1
of nodes of size Θ (log(n)) that have conductance O log(n)
. (Contrast all of these the a
3-regular random graph, which: is fully connected, is degree-homogeneous, and is a very
good expander.)


In these cases when measure concentration fails to occur, e.g., due to exogenously-specified degree
heterogeneity or due to extreme sparsity, then one will have difficulty with recovering parameters
of hypothesized models. More generally, similar problems arise, and the challenge will be to show
that one can reconstruct the model under as broad a range of parameters as possible.

24.3

A result for a spectral algorithm for the simplest nontrivial SBM

Let’s go into detail on the following simple SBM (which is the simplest aside from ER).
• Choose a partition of the vertics, call them V 1 and V 2 , and WLOG let V 1 = {1, . . . , n2 } and
V 2 = { n2 + 1, . . . , n}.
• Then, choose probabilities p > q and place edges between vertices i and j with probability

q
if i ∈ V 1 and j ∈ V 2 of i ∈ V 2 and j ∈ V 1
,
P [(ij) ∈ E] =
p
otherwise
In addition to being the “second simplest” SBM, this is also a simple example of a planted partition
model, which is commonly studied in TCS and related areas.
Here is a fact:


E number of edges crossing bw V 1 and V 2 = q|V 1 ||V 2 |.

In addition, if p is sufficiently larger than q, then every other partition has more edges. This is the
basis of recovering the model. Of course, if p is only slightly but not sufficiently larger than q, then
there might be fluctuational effects such that it is difficult to find this from the empirical graph.
This is analogous to having difficulty with recovering p from very sparse ER, as we discussed.
Within the SBM framework, the most important inferential task is recovering cluster membership of
nodes from a single observation of a graph (i.e., the two clusters in this simple planted partition form
of the SBM). There are a variety of procedures to do this, and here we will describe spectral methods.
In particular, we will follow a simple analysis motivated by McSherry’s analysis, as described by
Spielman, that will provide a “positive” result for sufficiently dense matrices where p and q are
sufficiently far apart. Then, we will discuss this model more generally, with an emphasis on how
to deal with very low-degree nodes that lead to measure concentration problems. In particular,
we will focus on a form of regularized spectral clustering, as done by Qin and Rohe in their
paper “Regularized spectral clustering under the degree-corrected stochastic blockmodel.” This
has connections with what we have done with the Laplacian over the last few weeks.
To start, let M be the population adjacency matrix, i.e., the hypothesized matrix, as described
above. That is,


p~1~1T q~1~1T
M=
q~1~1T p~1~1T
Then, let A be the empirical adjacency matrix, i.e., the actual matrix that is generated by flipping
coins and on which we will perform computations. This is generated as follows: let Aij = 1 w.p.

234

M. W. Mahoney

Mij and s.t. Aij = Aji . So, the basic goal is going to be to recover clusters in M by looking at
information in A.
Let’s look at the eigenvectors. First, since M ~1 = n2 (p + q)~1, we have
n
(p + q)
2
= ~1,

µ1 =
w1

where µ1 and w1 are the leading eigenvalue and eigenvector, respectively. Then, since the second
eigenvector (of M ) is constant on each cluster, we have that M w2 = µ2 w2 , where
µ2 =
w2 =

n
(p − q)
2
( 1
√ if i ∈ V 1
n

− √1n if i ∈ V 2

.

In that case, here is a simple algorithm for finding the planted bisection.
I. Compute v2 , the eigenvector of second largest eigenvalue of A.
II. Set S = {i : v2 (i) ≥ 0}
III. Guess that S is one side of the bisection and that S̄ is the other side.
We will show that under not unreasonable assumptions on p, q, and S, then by running this
algorithm one gets the hypothesized cluster mostly right.
Why is this?
The basic idea is that A is a perturbed version of M , and so by perturbation theory the eigenvectors
of A should look like the eigenvectors of M .
Let’s define R = A − M . We are going to view R as a random matrix that depends on the
noise/randomness in the coin flipping process. Since matrix perturbation theory bounds depend on
(among other things) the norm of the perturbation, the goal is to bound the probability that kRk2
is large. There are several methods from random matrix theory that give results of this general
form, and one or the other is appropriate, depending on the exact statement that one wants to
prove. For example, if you are familiar with Wigner’s semi-circle law, it is of this general form.
More recently, Furedi-Komlos got another version; as did Krivelevich and Vu; and Vu. Here we
state a result due to Vu.
4

Theorem 50. With probability tending to one, if p ≥ c logn(n) , for a constant c, then
√
kRk2 ≤ 3 pn.

The key question in theorems like this is the value of p. Here, one has that p ' log(n)
n , meaning that
one can get pretty sparse (relative to p = 1) but not extremely sparse (relative to p = n1 or p = n3 ).
If one wants stronger results (e.g., not just mis-classifying only a constant fraction of the vertices,
which we will do below, but instead that one predicts correctly for all but a small fraction of the

Lecture Notes on Spectral Graph Methods

235

vertices), then one needs p to be larger and the graph to be denser. As with the ER example, the
reason for this is that we need to establish concentration of appropriate estimators.
Let’s go onto perturbation theory for eigenvectors. Let α1 ≥ α2 ≥ · · · αn be the eigenvalues of A,
and let µ1 > µ2 > µ3 = · · · µn = 0 be the eigenvalues of M .
Here is a fact from matrix perturbation theory that we mentioned before: for all i,
|αi − µi | ≤ kA − M k2 = kRk2 .
The following two claims are easy to establish.
Claim 17. If kRk2 < n4 (p − q), then

n
3n
(p − q) < α2 <
(p − q)
4
4

Claim 18. If, in addition, q > p3 , then

3n
4 (p

− q) < α1 .

From these results, we have a separation, and so we can view α2 as a perturbation of µ2 . The
question is: can we view v2 as a perturbation of w2 ? The answer is Yes. Here is a statement of
this result.
Theorem 51. Let A, M be symmetric matrices, and let R = M − A. Let α1 ≥ · · · ≥ αn be
the eigenvectors of A, with v1 , · · · , vn the corresponding eigenvectors. Let µ1 ≥ · · · ≥ µn be the
eigenvectors of M , with w1 , · · · , wn the corresponding eigenvectors. Let θi be the angle between vi
and wi . Then,
2kRk2
minj6=i |αi − αj |
2kRk2
sin θi ≤
minj6=i |µi − µj |

sin θi ≤

Proof. WLOG, we can assume µi = 0, since the matrices M − µi I and A − αi I have the same
eigenvectors as M and A, and M − µi I has the ith eigenvalue being 0. Since the theorem is
vacuous if µi has multiplicities, we can assume unit multiplicity, and that wi is a unit vector in the
null space of M . Due to the assumption that µi = 0, we have that |αi | ≤ kRk2 .
P
Then, expand vi in an eigenbasis of M : vi = j cj wj , where cj = wjT vi . Let δ = minj |µj |. Then
observe that
X
X
X

c2j = δ2 1 − c2i = δ2 sin2 θi
kM vi k22 =
c2j µ2j ≥
c2j δ2 = δ2
j

j6=i

j6=i

and also that

kM vi k ≤ kAvi k + kRvi k = αi + kRvi k ≤ 2kRk2 .
So, from this it follows that sin θi ≤

2kRk2
δ

.

This is essentially a version of the Davis-Kahan result we saw before. Note that it says that the
amount by which eigenvectors are perturbed depends on how close are other eigenvalues, which is
what we would expect.
Next, we use this for partitioning the simple SBM. We want to show that not too many vertices
are mis-classified.

236

M. W. Mahoney
4

Theorem 52. Given the two-class SBM defined above, assume that p ≥ c logn(n) and that q > p/3.
If one runs the spectral algorithm described above, then at most a constant fraction of the vertices
are misclassified.
Proof. Consider the vector ~δ = v2 − w2 . For all i ∈ V that
p are misclassified by v2 , we have that
1
√
|δ(i)| ≥ n . So, if v2 misclassified k vertices, then kδk ≥ k/n. Since u and v are unit vectors, we
√
have the crude bound that kδk ≤ 2 sin θ2 .
Next, we can combine this with the perturbation theory result above. Since q > p/3, we have that
4
√
minj6=2 |µ2 − µi | = n2 (p − q); and since p ≥ c logn(n) , we have that kRk ≤ 3 pn. Then,
√
√
6 p
3 pn
=√
sin θ2 ≤ n
.
n(p − q)
2 (p − q)

So, the number k of mis-classified vertices satisfies

q

k
n

≤

√

√
6 p
,
n(p−q)

and thus k ≤

36p
.
(p−q)2

So, in particular, if p and q are both constant, then we expect to misclassify at most a constant
36p
n
fraction of the vertices. E.g., if p = 12 and q = p − √12n , then (p−q)
2 = 8 , and so only a constant
fraction of the vertices are misclassified.
This analysis is a very simple result, and it has been extended in various ways.
• The Ng et al. algorithm we discussed before computes k vectors and then does k means,
making similar gap assumptions.
• Extensions to have more than two blocks, blocks that are not the same size, etc.
• Extensions to include degree variability, as well as homophily and other empirically-observed
properties of networks.
The general form of the analysis we have described goes through to these cases, under the following
types of assumptions.
• The matrix is dense enough. Depending on the types of recovery guarantees that are hoped
for, this could mean that Ω(n) of the edges are present for each node, or perhaps Ω(polylog(n))
edges for each node.
• The degree heterogeneity is not too severe. Depending on the precise algorithm that is run,
this can manifest itself by placing an upper bound on the degree of the highest degree node
and/or placing a lower bound on the degree of the lowest degree node.
• The number of clusters is fixed, say as a function of n, and each of the clusters is not too
small, say a constant fraction of the nodes.
Importantly, none of these simplifying assumptions are true for most “real world” graphs. As such,
there has been a lot of recent work focusing on dealing with these issues and making algorithms
for SBMs work under broader assumptions. Next, we will consider one such extension.

Lecture Notes on Spectral Graph Methods

24.4

237

Regularized spectral clustering for SBMs

Here, we will consider a version of the degree-corrected SBM, and we will consider doing a form of
regularized spectral clustering (RSC) for it.
Recall the definition of the basic SBM.
Definition 63. Given nodes V = [n], let z : [n] → [k] be a partition of the n nodes into k blocks,
i.e., zi is the block membership of the ith node. Let B ∈ [0, 1]k×k . Then, under the SBD, we have
that the probability of an edge between i and j is
Pij = Bzi zj , for all i, j ∈ {1, . . . , n}.
In particular, this means that, given z, the edges are independent.
Many real-world graphs have substantial degree heterogeneity, and thus it is common to in corporate
this into generative models. Here is the extension of the SBM to the Degree-corrected stochastic
block model (DC-SBM), which introduces additional parameters θi , for i ∈ [n], to control the
node degree.
Definition 64. Given the same setup as for the SBM, specify also additional parameters θi , for
i ∈ [n]. Then, under the DC-SBM, the probability of an edge between i and j is
Pij = θi θj Bzi zj ,
where θi θj Bzi zj ∈ [0, 1], for all i, j ∈ [n].
Note: to make the DC-SBM identifiable (i.e., so that it is possible in principle to learn the true
model parameters, say given an infinite number of observations,
P which is clearly a condition that
is needed for inference),
P one can impose the constraint that i θi δzi ,r = 1, for each block r. (This
condition says that i θi = 1 within each block.) In this case Bst , for s 6= t, is the expected number
of links between block s and block t; and Bst , for s = t, is the expected number of links within
block s.
Let’s say that A ∈ {0, 1}n×n is the adjacency matrix; L = D −1/2 AD −1/2 . In addition, let A = E [A]
be the population matrix, under the DC-SBM. Then, one can express A as A = ΘZBZ T Θ, where
Θ ∈ Rn×n = diag(θi ), and where Z ∈ {0, 1}n×k is a membership matrix with Zit = 1 iff node i is
in block t, i.e., if zi = t.
We are going to be interested in very sparse matrices, for which the minimum node degree is very
small, in which case a vanilla algorithm will fail to recover the SBM blocks. Thus, we will need to
introduce a regularized version of the Laplacian. Here is the definition.
−1/2

Definition 65. Let τ > 0. The regularized graph Laplacian is Lτ = Dτ
Dτ = D + τ I, for τ > 0.

−1/2

ADτ

∈ Rn×n , with

This is defined for the empirical data; but given this, we can define the corresponding population quantities:
X
Dii =
Aij
j

Dτ

= D + τI

Lτ

= Dτ−1/2 ADτ−1/2

L = D −1/2 AD −1/2

238

M. W. Mahoney

Two things to note.
• Under the DC-SBM, if the model is identifiable, then one should be able to determine the
partition from A (which we don’t have direct access to, given the empirical data).
• One also wants to determine the partition from the empirical data A, under broader assumptions than before, in particular under smaller minimum degree.
Here is a description of the basic algorithm of Qin and Rohe. Basically, it is the Ng et al. algorithm
that we described before, except that we apply it to the regularized graph Laplacian, i.e., it involves
finding the leading eigenvectors of Lτ and then clustering in the low dimensional space.
Given as input an Adjacency Matrix A, the number of clusters k, and the regularizer τ ≥ 0.
I. Compute Lτ .
II. Compute the matrix Xτ = [X1τ , . . . , Xkτ ] ∈ Rn×k , the orthogonal matrix consisting of the k
largest eigenvectors of Lτ .
III. Compute the matrix Xτ∗ ∈ Rn×k by normalizing each row of Xτ toPhave unit length, i.e.,
τ,2
∗,τ
τ/
project each row of Xτ onto the unit sphere in Rk , i.e., Xij
= Xij
j Xij .
IV. Run k means on the rows of Xτ∗ to create k non-overlapping clusters V1 , . . . , Vk .

V. Output V1 , . . . , Vk ; node i is assigned to cluster r if the ith tow of Xτ∗ is assigned to V .
There are a number of empirical/theoretical tradeoffs in determining the best value for τ , but one
can think of τ as being the average node degree.
There are several things one can show here.
First, one can show that Lτ is close to Lτ .
Theorem 53. Let G be the random graph with P [edge bw ij] = Pij . Let δ = mini Dii be the
minimum expected degree of G. If δ + τ > O (log(n)), then with constant probability
r
log(n)
kLτ − Lτ k ≤ O(1)
.
δ+τ
Remark. Previous results required that the minimum degree δ ≥ O(log(n)), so this result generalizes these to allow δ to be much smaller, assuming the regularization parameter τ is large enough.
Importantly, typical real networks do not satisfy the condition that δ ≥ O(log(n)), and RSC is
most interesting when this condition fails. So, we can apply this result in here to graph with small
node degrees.
Remark. The form of Lτ is similar to many of the results we have discussed, and one can
imagine implementing RSC (and obtaining this theorem as well as those given below) by computing
approximations such as what we have discussed. So far as I know, that has not been done.
Second, one can bound the difference between the empirical and population eigenvectors. For this,
one needs an additional concept.

Lecture Notes on Spectral Graph Methods

239

• Given an n × k matrix A, the statistical leverage scores of A are the diagonal elements of the
projection matrix onto the span of A.
In particular, if the n × k matrix U is an orthogonal matrix for the column span of A, then the
leverage scores of A are the Euclidean norms of the rows of U . For a “tall” matrix A, the ith
leverage score has an interpretation in terms of the leverage or influence that the ith row of an A
has on the least-squares fit problem defined by A. In the following, we will use an extension of the
leverage scores, defined relative to the best rank-k approximation the the matrix.
Theorem 54. Let Xτ and Xτ be in Rn×k contain the top k eigenvectors of Lτ and Lτ , respectively.
Let
ξ = min{min{kXτi k2 , kXτi k2 }}.
i
q
∗
∗
≤ O(λk ) and
Let Xτ and Xτ be the row normalized versions of Xτ and Xτ . Assume that k log(n)
δ+τ
δ + τ > O(log(n)). Then, with constant probability,


1p
kXτ − Xτ OkF ≤ O
k log(n)δ + τ
λk


1 p
∗
∗
k log(n)δ + τ ,
kXτ − Xτ OkF ≤ O
ξλk

where O is a rotation matrix.

Note that the smallest leverage score enters the second expression but not the first expression. That
is, it does not enter the bounds on the empirical quantities, but it does enter into the bounds for
the population quantities.
We can use these results to derive misclassification rate for RSC. The basic idea for the misclassification rate is to run k-means on the rows of Xτ∗ and also on the rows of Xτ∗ . Then, one can say that
a node on the empirical data is clustered correctly if it is closer to the centroid of the corresponding
cluster on the population data. This basic idea needs to be modified to take into account the fact
that if any λi are equal, then only the subspace spanned by the eigenvectors is identifiable, so we
consider this up to a rotation O.
Definition 66. If Ci O is closer to Ci than any other Cj , then we say that the node is correctly
clustered; and we define the misclassified nodes to be

M = i : ∃j 6= i s.t. kCi OT − Ci k2 > kCi OT − Cj .
Third, one can bound the misclassification rate of the RCS classifier with the following theorem.
Theorem 55. With constant probability, the misclassification rate is
k log(n)
|M|
.
≤c 2
n
nξ (δ + τ )λ2k
Here too the smallest leverage score determines the overall quality.
Remark. This is the first result that explicitly relates leverage scores to the statistical performance
of a spectral clustering algorithm. This is a large topic, but to get a slightly better sense of it,

240
recall that the leverage scores of Lτ are kXτi k22 =

M. W. Mahoney
τ
P θτi
.
θ
j j δzj zi

So, in particular, if a node i has a

small expected degree, then θiτ is small and kXτi k2 is small. Since ξ appears in the denominator
of the above theorems, this leads to a worse bound for the statistical claims in these theorems. In
particular, the problem arises due to projecting Xτi onto the unit sphere, i.e., while large-leverage
nodes don’t cause a problem, errors for small-leverage rows can be amplified—this didn’t arise
when we were just making claims about the empirical data, e.g., the first claim of Theorem 54,
but when considering statistical performance, e.g., the second claim of Theorem 54 or the claim of
Theorem 55, for nodes with small leverage score it amplifies noisy measurements.

Lecture Notes on Spectral Graph Methods

25

241

(04/23/2015): Laplacian solvers (1 of 2)

Reading for today.
• “Effective Resistances, Statistical Leverage, and Applications to Linear Equation Solving,”
in arXiv, by Drineas and Mahoney
• “A fast solver for a class of linear systems,” in CACM, by Koutis, Miller, and Peng
• “Spectral Sparsification of Graphs: Theory and Algorithms,” in CACM, by Batson, Spielman,
Srivastava, and Teng
(Note: the lecture notes for this class and the next are taken from the lecture notes for the final
two classes of the class I taught on Randomized Linear Algebra in Fall 2013.)

25.1

Overview

We have seen problems that can be written in the form of a system of linear equations with
Laplacian constraint matrices, i.e.,
Lx = b.
For example, we saw this with the various semi-supervised learning methods as well as with the
MOV weakly-local spectral method. In some cases, this arises in slightly modified form, e.g., as an
augmented/modified graph and/or if there are additional projections (e.g., the Zhou et al paper
on “Learning with labeled and unlabeled data on a directed graph,” that is related to the other
semi-supervised methods we discussed, does this explicitly). Today and next time we will discuss
how to solve linear equations of this form.

25.2

Basic statement and outline

While perhaps not obvious, solving linear equations of this form is a useful algorithmic primitive—
like divide-and-conquer and other such primitives—much more generally, and thus there has been
a lot of work on it in recent years.
Here is a more precise statement of the use of this problem as a primitive.
Definition 67. The Laplacian Primitive concerns systems of linear equations defined by Laplacian
constraint matrices:
• INPUT: a Laplacian L ∈ Rn×n , a vector b ∈ Rn such that

Pn

i=1 bi

= 0, and a number ǫ > 0.

†
†
n
• OUTPUT: a vector x̃opt ∈ Rn such
√ that kx̃opt − L bkL ≤ ǫkL bkL , where for a vector z ∈ R
the L-norm is given by kzkL = z T Lz.

While we will focus on linear equations with Laplacian constraint matrices, most of the results in
this area hold for a slightly broader class of problems. In particular, they hold for any linear system
Ax = b, where A is an SDD (symmetric diagonally dominant) matrix (i.e., that the diagonal entry
of each row is larger, or not smaller, than the sum of the absolute values of the off-diagonal entries
in that row). The reason for this is that SDD systems are linear-time reducible to Laplacian linear
systems via a construction that only doubles the number of nonzero entries in the matrix.

242

M. W. Mahoney

As mentioned, the main reason for the interest in this topic is that, given a fast, e.g., nearly linear
time algorithm, for the Laplacian Primitive, defined above, one can obtain a fast algorithm for all
sorts of other basic graph problems. Here are several examples of such problems.
• Approximate Fiedler vectors.
• Electrical flows.
• Effective resistance computations.
• Semi-supervised learning for labeled data.
• Cover time of random walks.
• Max flow and min cut and other combinatorial problems.
Some of these problems we have discussed. While it might not be surprising that problems like
effective resistance computations and semi-supervised learning for labeled data can be solved with
this primitive, it should be surprising that max flow and min cut and other combinatorial problems
can be solved with this primitive. We won’t have time to discuss this in detail, but some of the
theoretically fastest algorithms for these problems are based on using this primitive.
Here is a statement of the basic result that led to interest in this area.
Theorem 56 (ST).
There is a randomized
algorithm for the Laplacian Primitive that runs in


O(1)
expected time O m log
(n) log (1/ǫ) , where n is the number of nodes in L, m is the number of
nonzero entries in L, and ǫ is the precision parameter.
Although the basic algorithm of ST had something like the 50th power in the exponent of the
logarithm, it was a substantial theoretical breakthrough, and since then it has been improved
by KMP to only a single log, leading to algorithms that are practical or almost practical. Also,
although we won’t discuss it in detail, many of the local and locally-biased spectral methods we
have discussed arose out of this line of work in an effort to develop and/or improve this basic result.
At a high level, the basic algorithm is as follows.
I. Compute a sketch of the input by sparsifying the input graph.
II. Use the sketch to construct a solution, e.g., by solving the subproblem with any black box
solver or by using the sketch as a preconditioner for an iterative algorithm on the original problem.
Thus, the basic idea of these methods is very simple; but to get the methods to work in the allotted
time, and in particular to work in nearly-linear time, is very complicated.
Today and next time, we will discuss these methods, including a simple but slow method in more
detail and a fast but complicated method in less detail.
• Today. We will describe a simple, non-iterative, but slow algorithm. This algorithm provides a very simple version of the two steps of the basic algorithm described above; and,
while slow, this algorithm highlights several basic ideas of the more sophisticated versions of
these methods.

Lecture Notes on Spectral Graph Methods

243

• Next time. We will describe a fast algorithm provides a much more sophisticated implementation of the two steps of this basic algorithm. Importantly, it makes nontrivial use
of combinatorial ideas and couples the linear algebra with combinatorial preconditioning in
interesting ways.

25.3

A simple slow algorithm that highlights the basic ideas

Here, we describe in more detail a very simple algorithm to solve Laplacian-based linear systems. It
will be good to understand before we get to the fast but more complicated versions of the algorithm.
Recall that L = D − W = B T W B is our Laplacian, where B is the m × n edge-incidence matrix,
and where W is an m × m edge weight matrix. In particular, note that m > n (assume the graph
is connected to avoid trivial cases), and so the matrix B is a tall matrix.
Here is a restatement of the above problem.
Definition 68. Given as input a Laplacian matrix L ∈ Rn×n , a vector b ∈ Rn , compute
argminx∈Rn kLx − bk2 .
The minimal ℓ2 norm xopt is given by xopt = L† b, where L† is the Moore-Penrose generalized inverse
of L.
We have reformulated this as a regression since it makes the proof below, which is based on RLA
(Randomized Linear Algebra) methods, cleaner.
The reader familiar with linear algebra might be concerned about the Moore-Penrose generalized
inverse since, e.g., it is typically not well-behaved with respect to perturbations in the data matrix.
Here, the situation is particularly simple: although L is rank-deficient, (1) it is invertible if we
work with vectors b ⊥ ~1, and (2) because this null space is particular simple, the pathologies that
typically arise with the Moore-Penrose generalized inverse do not arise here. So, it isn’t too far off
to think of this as the inverse.
Here is a simple algorithm to solve this problem. This algorithm takes as input L, b, and ǫ; and it
returns as output a vector x̃opt .
I. Form B and W , define Φ = W 1/2 B ∈ Rm×n , let UΦ ∈ Rm×n be an orthogonal matrix
spanning the column space of Φ, and let (UΦ )(i) denote the ith row of UΦ .
P
II. Let pi , for i ∈ [n] such that ni=1 pi = 1 be given by
pi ≥ β

k (UΦ )(i) k22
kUΦ k2F

=

β
k (UΦ )(i) k22
n

(55)

for some value of β ∈ (0, 1]. (Think of β = 1, which is a legitimate choice, but the additional
flexibility of allowing β ∈ (0, 1) will be important in the next class.)
A key aspect of this algorithm is that the sketch is formed by choosing elements of the Laplacian
with the probabilities in Eqn. (55); these quantities are known as the statistical leverage scores,
and they are of central interest in RLA. Here is a definition of these scores more generally.

244

M. W. Mahoney

Definition 69. Given a matrix A ∈ Rm×n , where m > n, the ith leverage score is

(PA )ii = UA UAT ii = k (UA )ii k22 ,

i.e., it is equal to the diagonal element of the projection matrix onto the column span of A.
Here is a definition of a seemingly-unrelated notion that we talked about before.
Definition 70. Given G = (V, E), a connected, weighted, undirected graph with n nodes, m edges,
and corresponding weights we ≥ 0, for all e ∈ E, let L = B T W B. Then, the effective resistance Re
across edge e ∈ E are given by the diagonal elements of the matrix R = BL†B.
Here is a lemma relating these two quantities.
Lemma 28. Let Φ = W 1/2 B denote the scaled edge-incidence matrix. If ℓi is the leverage score of
the ith row of Φ, then wℓii is the effective resistance of the ith edge.
Proof. Consider the matrix
P = W 1/2 B B T W B

†

B T W 1/2 ∈ Rm×m ,

and notice that P = W 1/2 RW 1/2 is a rescaled version of R = BL† B, whose diagonal elements are
the effective resistances. Since Φ = W 1/2 B, it follows that
†
P = Φ ΦT Φ ΦT .
Let UΦ be an orthogonal matrix spanning the columns of Φ. Then, P = UΦ UΦT , and so

Pii = UΦ UΦT ii = k (UΦ )(i) k22 ,
which establishes the lemma.

So, informally, we sparsify the graph by biasing our random sampling toward edges that are “important” or “influential” in the sense that they have large statistical leverage or effective resistance,
and then we use the sparsified graph to solve the subproblem.
Here is the main theorem for this algorithm.
Theorem 57. With constant probability, kxopt − x̃opt kL ≤ ǫkxopt kL .
Proof. The main idea of the proof is that we are forming a sketch of the Laplacian by randomly
sampling elements, which corresponds to randomly sampling rows of the edge-incidence matrix, and
that we need to ensure that the corresponding sketch of the edge-incidence matrix is a so-called
subspace-preserving embedding. If that holds, then the eigenvalues of the edge-incidence matrix
and it’s sketch are close, and thus the eigenvalues of the Laplacian are close, and thus the original
Laplacian and the sparsified Laplacian are “close,” in the sense that the quadratic form of one is
close to the quadratic form of the other.
Here are the details.
By definition,

kxopt − x̃opt k2L = (xopt − x̃opt )T L (xopt − x̃opt ) .

Lecture Notes on Spectral Graph Methods

245

Recall that L = B T W B, that xopt = L† b, and that x̃opt = L̃† b. So,
kxopt − x̃opt k2L = (xopt − x̃opt )T B T W B (xopt − x̃opt )
= kW 1/2 B (xopt − x̃opt ) k22

Let Φ ∈ Rm×n be defined as Φ = W 1/2 B, and let its SVD be Φ = UΦ ΣΦ VΦT . Then
L = ΦT Φ = VΦ Σ2Φ VΦT
and
T
xopt = L† b = VΦ Σ−2
Φ VΦ b.

In addition
L̃ = ΦT S T SΦ = (SΦ)T (SΦ)
and also
x̃opt = L̃† b = (SΦ)† (SΦ)T † b = SUΦ ΣΦ VΦT
By combining these expressions, we get that

†

SUΦ ΣΦ VΦT

T †

b

kxopt − x̃opt k2L = kΦ (xopt − x̃opt ) k22


  2
T
T †
T T†
bk2
V
−
SU
Σ
V
SU
Σ
V
= kUΦ ΣΦ VΦT VΦ Σ−2
Φ
Φ
Φ
Φ
Φ
Φ
Φ Φ

T †
T †
T
VΦ bk22
SUΦ ΣΦ VΦT
= kΣ−1
Φ VΦ b − ΣΦ SUΦ ΣΦ VΦ
Next, we note the following:

 √

E kUΦT S T SUΦ − Ik2 ≤ ǫ,

where of course the expectation can be removed by standard methods. This follows from a result
of Rudelson-Vershynin, and it can also be obtained as a matrix 
concentration bound. This is a
key result in RLA, and it holds since we are sampling O nǫ log nǫ rows from U according to the
leverage score sampling probabilities.
From standard matrix perturbation theory, it thus follows that

√
σi UΦT S T SUΦ − 1 = σi2 (SUΦ ) − 1 ≤ ǫ.
So, in particular, the matrix SUΦ has the same rank as the matrix UΦ . (This is a so-called
subspace embedding, which is a key result in RLA; next time we will interpret it in terms of
graphic inequalities that we discussed before.)
In the rest of the proof, let’s condition on this random event being true.
Since SUΦ is full rank, it follows that
†
(SUΦ ΣΦ )† = Σ−1
Φ (SUΦ ) .

So, we have that
†
T † −1 T 2
T
ΣΦ VΦ bk2
kxopt − x̃opt k2L = kΣ−1
Φ VΦ b − (SUΦ ) (SUΦ )
T
−2 T −1 T 2
= kΣ−1
Φ VΦ b − VΩ ΣΩ VΩ ΣΦ VΦ bk2 ,

246

M. W. Mahoney

where the second line follows if we define Ω = SUΦ and let its SVD be
Ω = SUΦ = UΩ ΣΩ VΩT .
T
T
Then, let Σ−1
Ω = I + E, for a diagonal error matrix E, and use that VΩ VΩ = VΩ VΩ = I to write
T
T −1 T 2
kxopt − x̃opt k2L = kΣ−1
Φ VΦ b − VΩ (I + E) VΩ ΣΦ VΦ bk2
T 2
= kVΩ EVΩT Σ−1
Φ VΦ bk2
T 2
= kEVΩT Σ−1
Φ VΦ bk2

T 2
≤ kEVΩT k22 kΣ−1
Φ VΦ bk2
T 2
= kEk22 kΣ−1
Φ VΦ bk2

But, since we want to bound kEk, note that
|Eii | = σi−2 (Ω) − 1 = σi−1 (SUΦ ) − 1 .
So,
kEk2 = max σi−2 (SUΦ ) − 1 ≤
i

√

ǫ.

So,
T 2
kxopt − x̃opt k2L ≤ ǫkΣ−1
Φ VΦ bk2 .

In addition, we can derive that
kxopt k2L = xTopt Lxopt

T 

W 1/2 Bxopt
=
W 1/2 Bxopt
= kΦxopt k22

T 2
= kUΦ ΣΦ VΦT VΦ Σ−2
Φ VΦ bk2
T 2
= kΣ−1
Φ VΦ bk2 .

So, it follows that
kxopt − x̃opt k2L ≤ ǫkxopt k2L ,
which establishes the main result.
Before concluding, here is where we stand. This is a very simple algorithm that highlights the basic
ideas of Laplacian-based solvers, but it is not fast. To make it fast, two things need to be done.
• We need to compute or approximate the leverage scores quickly. This step is very nontrivial. The original algorithm of ST (that had the log50 (n) term) involved using local random
walks (such as what we discussed before, and in fact the ACL algorithm was developed to
improve this step, relative to the original ST result) to construct well-balanced partitions
in nearly-linear time. Then, it was shown that one could use effective resistances; this was
discovered by SS independently of the RLA-based method outlined above, but it was also
noted that one could call the nearly linear time solver to approximate them. Then, it was
shown that one could relate it to spanning trees to construct combinatorial preconditioners.
If this step was done very carefully, then one obtains an algorithm that runs in nearly linear
time. In particular, though, one needs to go beyond the linear algebra to map closely to the
combinatorial properties of graphs, and in particular find low-stretch spanning trees.

Lecture Notes on Spectral Graph Methods

247

• Instead of solving the subproblem on the sketch, we need to use the sketch to create a
preconditioner for the original problem and then solve a preconditioned version of the original
problem. This step is relatively straightforward, although it involves applying an iterative
algorithm that is less common than popular CG-based methods.
We will go through both of these in more detail next time.

248

M. W. Mahoney

26

(04/28/2015): Laplacian solvers (2 of 2)

Reading for today.
• Same as last class.
Last time, we talked about a very simple solver for Laplacian-based systems of linear equations,
i.e., systems of linear equations of the form Ax = b, where the constraint matrix A is the Laplacian
of a graph. This is not fully-general—Laplacians are SPSD matrices of a particular form—but
equations of this form arise in many applications, certain other SPSD problems such as those based
on SDD matrices can be reduced to this, and there has been a lot of work recently on this topic
since it is a primitive for many other problems. The solver from last time is very simple, and it
highlights the key ideas used in fast solvers, but it is very slow. Today, we will describe how to take
those basic ideas and, by coupling them with certain graph theoretic tools in various ways, obtain
a “fast” nearly linear time solver for Laplacian-based systems of linear equations.
In particular, today will be based on the Batson-Spielman-Srivastava-Teng and the Koutis-MillerPeng articles.

26.1

Review from last time and general comments

Let’s start with a review of what we covered last time.
Here is a very simple algorithm. Given as input the Laplacian L of a graph G = (V, E) and a right
hand side vector b, do the following.
• Construct a sketch of G by sampling elements of G, i.e., rows of the edge-node incidence
matrix, with probability proportional to the leverage scores of that row, i.e., the effective
resistances of that edge.
• Use the sketch to construct a solution, e.g., by solving the subproblem with a black box or
using it as a preconditioner to solve the original problem with an iterative method.
The basic result we proved last time is the following.
Theorem 58. Given a graph G with Laplacian L, let xopt be the optimal solution of Lx = b; then
the above algorithm returns a vector x̃opt such that, with constant probability,
kxopt − x̃opt kL ≤ ǫkxopt kL .

(56)

The proof of this result boiled down to showing that, by sampling with respect to a judiciouslychosen set of nonuniform importance sampling probabilities, then one obtains a data-dependent
subspace embedding of the edge-incidence matrix. Technically, the main thing to establish was
that, if U is an m × n orthogonal matrix spanning the column space of the weighted edge-incidence
matrix, in which case I = In = U T U , then
kI − (SU )T (SU ) k2 ≤ ǫ,
where S is a sampling matrix that represents the effect of sampling elements from L.

(57)

Lecture Notes on Spectral Graph Methods

249

The sampling probabilities that are used to create the sketch are weighted versions of the statistical
leverage scores of the edge-incidence matrix, and thus they also are equal to the effective resistance
of the corresponding edge in the graph. Importantly, although we didn’t describe it in detail, the
theory that provides bounds of the form of Eqn. (57) is robust to the exact form of the importance
sampling probabilities, e.g., bounds of the same form hold if any other probabilities are used that
are “close” (in a sense that we will discuss) to the statistical leverage scores.
The running time of this simple strawman algorithm consists of two parts, both of which the fast
algorithms we will discuss today improve upon.
• Compute the leverage scores, exactly or approximately. A naive computation of the leverage
scores takes O(mn2 ) time, e.g., with a black box QR decomposition routine. Since they are
related to the effective resistances, one can—theoretically at least compute them with any one
of a variety of fast nearly linear time solvers (although one has a chicken-and-egg problem,
since the solver itself needs those quantities). Alternatively, since one does not need the exact
leverage scores, one could hope to approximate them in some way—below, we will discuss
how this can be done with low-stretch spanning trees.
• Solve the subproblem, exactly or approximately. A naive computation of the solution to the
subproblem can be done in O(n3 ) time with standard direct methods, or it can be done with
an iterative algorithm that requires a number of matrix-vector multiplications that depends on
the condition number of L (which in general could be large, e.g., Ω(n)) times m, the number
of nonzero elements of L. Below, we will see how this can be improved with sophisticated
versions of certain preconditioned iterative algorithms.
More generally, here are several issues that arise.
• Does one use exact or approximate leverage scores? Approximate leverage scores are sufficient
for the worst-case theory, and we will see that this can be accomplished by using LSSTs, i.e.,
combinatorial techniques.


elements from L to obtain
• How good a sketch is necessary? Last time, we sampled Θ n log(n)
2
ǫ
a 1 ± ǫ subspace embedding, i.e., to satisfy Eqn. (57), and this leads to an ǫ-approximate
solution of the form of Eqn (56). For an iterative method, this might be overkill, and it might
suffice to satisfy Eqn. (57) for, say, ǫ = 12 .
• What is the dependence on ǫ? Last time, we sampled and then solved the subproblem, and
thus the complexity with respect to ǫ is given by the usual random sampling results. In particular, since the complexity is a low-degree polynomial in 1ǫ , it will be essentially impossible to
obtain a high-precision solution, e.g., with ǫ = 10−16 , as is of interest in certain applications.
• What is the dependence on the condition number κ(L)? In general, the condition number
can be very large, and this will manifest itself in a large number of iterations (certainly in
worst case, but also actually quite commonly). By working with a preconditioned iterative
algorithm, one should aim for a condition number of the preconditioned problem that is
quite small, e.g., if not constant then log(n) or less. In general, there will be a tradeoff
between the quality of the preconditioner and the number of iterations needed to solve the
preconditioned problem.

250

M. W. Mahoney
• Should one solve the subproblem directly or use it to construct a preconditioned to the original
problem? Several of the results just outlined suggest that an appropriate iterative methods
should be used and this is what leads to the best results.

Remark. Although we are not going to describe it in detail, we should note that the LSSTs will
essentially allow us to approximate the large leverage scores, but they won’t have anything to say
about the small leverage scores. We saw (in a different context) when we were discussing statistical
inference issues that controlling the small leverage scores can be important (for proving statistical
claims about unseen data, but not for claims on the empirical data). Likely similar issues arise
here, and likely this issue can be mitigated by using implicitly regularized Laplacians, e.g., as as
implicitly computed by certain spectral ranking methods we discussed, but as far as I know no one
has explicitly addressed these questions.

26.2

Solving linear equations with direct and iterative methods

Let’s start with the second step of the above two-level algorithm, i.e., how to use the sketch from
the first step to construct an approximate solution, and in particular how to use it to construct a
preconditioner for an iterative algorithm. Then, later we will get back to the first step of how to
construct the sketch.
As you probably know, there are a wide range of methods to solve linear systems of the form
Ax = b, but they fall into two broad categories.
• Direct methods. These include Gaussian elimination, which runs in O(n3 ) time; and
Strassen-like algorithms, which run in O(nω ) time, where ω = 2.87 . . . 2.37. Both require
storing the full set of in general O(n2 ) entries. Faster algorithms exist if A is structured. For
example, if A is n × n PSD with m nonzero, then conjugate gradients, used as a direct solver,
takes O(mn) time, which if m = O(n) is just O(n2 ). That is, in this case, the time it takes
it proportional to the time it takes just to write down the inverse. Alternatively, if A is the
adjacency matrix of a path graph or any tree, then the running time is O(m); and so on.
• Iterative methods. These methods don’t compute an exact answer, but they do compute
an ǫ-approximate solution, where ǫ depends on the structural properties of A and the number
of iterations, and where ǫ can be made smaller with additional iterations. In general, iterations are performed by doing matrix-vector multiplications. Advantages of iterative methods
include that one only needs to store A, these algorithms are sometimes very simple, and
they are often faster than running a direct solver. Disadvantages include that one doesn’t
obtain an exact answer, it can be hard to predict the number of iterations, and the running
(A)
. Examtime depends on the eigenvalues of A, e.g., the condition number κ(A) = λλmax
min (A)
ples include the Richardson iteration, various Conjugate Gradient like algorithms, and the
Chebyshev iteration.
Since the running time of iterative algorithms depend on the properties of A, so-called preconditioning methods are a class of methods to transform a given input problem into another problem
such that the modified problem has the same or a related solution to the original problem; and
such that the modified problem can be solved with an iterative method more quickly.
(A)
,
For example, to solve Ax = b, with A ∈ Rn×n and with m = nnz(A), if we define κ(A) = λλmax
min (A)
where λmax and λmin are the maximum and minimum non-zero eigenvalues of A, to be the condition

Lecture Notes on Spectral Graph Methods

251

number of A, then CG runs in
O

p

κ(A) log (1/ǫ)



iterations (each of which involves a matrix-vector multiplication taking O(m) time) to compute
and ǫ-accurate solution to Ax = b. By an ǫ-accurate approximation, here we mean the same notion
that we used above, i.e., that
kx̃opt − A† bkA ≤ ǫkA† bkA ,
p
where the so-called A-norm is given by kykA = y T Ay. This A-norm is related to the usual
Euclidean norm as follows: kykA ≤ κ(A)kyk2 and kyk2 ≤ κ(A)kykA . While the A-norm is perhaps
unfamiliar, in the context of iterative algorithms it is not too dissimilar to the usual Euclidean
norm, in that, given an ǫ-approximation for the former, we can obtain an ǫ-approximation for the
latter with O (log (κ(A)/ǫ)) extra iterations.
In this context, preconditioning typically means solving
B −1 Ax = B −1 b,

where B is chosen such that κ B −1 A is small; and it is easy to solve problems of the form Bz = c.
The two extreme cases are B = I, in which case it is easy to compute and apply but doesn’t help
solve the original problem, and B = A−1 , which means that the iterative algorithm would converge
after zero steps but which is difficult to compute. The running time of the preconditioned problem
involves

p
κ (B −1 A) log (1/ǫ)
O

matrix vector multiplications. The quantity κ B −1 A is sometimes known as the relative condition
number of A with respect to B—in general, finding a B that makes it smaller takes more initial time
but leads to fewer iterations. (This was the basis for the comment above that there is a tradeoff in
choosing the quality of the preconditioner, and it is true more generally.)
These ideas apply more generally, but we consider applying them here to Laplacians. So, in
particular, given a graph G and its Laplacian LG , one way to precondition it is to look for a different
graph H such that LH ≈ LG . For example, one could use the sparsified graph that we computed
with the algorithm from last class. That sparsified graph is actually an ǫ-good preconditioned, but
it is too expensive to compute. To understand how we can go beyond the linear algebra and exploit
graph theoretic ideas to get good approximations to them more quickly, let’s discuss different ways
in which two graphs can be close to one another.

26.3

Different ways two graphs can be close

We have talked formally and informally about different ways graphs can be close, e.g., we used the
idea of similar Laplacian quadratic forms when talking about Cheeger’s Inequality and the quality
of spectral partitioning methods. We will be interested in that notion, but we will also be interested
in other notions, so let’s now discuss this topic in more detail.
• Cut similarity. One way to quantify the idea that two graphs are close is to say that they
are similar in terms of their cuts or partitions. The standard result in this area is due to BZ,
who developed the notion of cut similarity to develop fast algorithms for min cut and max
flow and other related combinatorial problems. This notion of similarity says that two graphs

252

M. W. Mahoney
are close if the weights of the cuts, i.e., the sum of edges or edge weights crossing a partition,
are close for all cuts. To
P define it, recall that, given a graph G = (V, E, W ) and a set S ⊂ V ,
we can define cutG = u∈S,v∈S̄ W(uv) . Here is the definition.
Definition 71. Given two graphs, G = (V, E, W ) and G̃ = (V, Ẽ, W̃ ), on the same vertex
set, we say that G and G̃ are σ-cut-similar if, for all S ⊆ V , we have that
1
cut (S) ≤ cutG (S) ≤ σcutG̃ (S).
σ G̃

As an example of a result in this area, the following theorem shows that every graph is cutsimilar to a graph with average degree O(log(n)) and that one can compute that cut-similar
graph quickly.
Theorem 59 (BK). For all ǫ > 0, every graph G = (V, E,W ) has a (1 + ǫ)-cut-similar graph
G̃ = (V, Ẽ, Ṽ ) such that Ẽ ⊆ E and |Ẽ|= O n log(n/ǫ2 ) . In addition, the graph G̃ can be
computed in O m log3 (n) + m log(n/ǫ2 ) time.

• Spectral similarity. ST introduced the idea of spectral similarity in the context of nearly
linear time solvers. One can view this in two complementary ways.
– As a generalization of cut similarity.
– As a special case of subspace embeddings, as used in RLA.
We will do the former here, but we will point out the latter at an appropriate point.
Given G = (V, E, W ), recall that L : Rn → R is a quadratic form associated with G such that
X
L(x) =
W(uv) (xu − xv )2 .
(uv)∈E

If S ⊂ V and if x is an indicator/characteristic vector for the set S, i.e., it equals 1 on nodes
u ∈ S, and it equals 0 on nodes v ∈ S, then for those indicator vectors x, we have that
L(x) = cutG (x). We can also ask about the values it takes for other vectors x. So, let’s define
the following.
Definition 72. Given two graphs, G = (V, E, W ) and G̃ = (V, Ẽ, W̃ ), on the same vertex
set, we say that G and G̃ are σ-spectrally similar if, for all x ∈ Rn , we have that
1
L (x) ≤ LG (x) ≤ σLG̃ (x).
σ G̃

That is, two graphs are spectrally similar if their Laplacian quadratic forms are close.
In addition to being a generalization of cut similarity, this also corresponds to a special case
of subspace embeddings, restricted from general matrices to edge-incidence matrices and their
associated Laplacians.
– To see this, recall that subspace embeddings preserve the geometry of the subspace and
that this is quantified by saying that all the singular values of the sampled/sketched
version of the edge-incidence matrix are close to 1, i.e., close to those of the edgeincidence matrix of the original un-sampled graph. Then, by considering the Laplacian,
rather than the edge-incidence matrix, the singular values of the original and sketched
Laplacian are also close, up to a quadratic of the approximation factor on the edgeincidence matrix.

Lecture Notes on Spectral Graph Methods

253

Here are several other things to note about spectral embeddings.
– Two graphs can be cut-similar but not spectrally-similar. For example, consider G to
be an n-vertex path and G̃ to be an n-vertex cycle. They are 2-cut similar but are only
n-spectrally similar.
– Spectral similarity is identical to the notion of relative condition number in NLA that
we mentioned above. Recall, given A and B, then A  B iff xT Ax ≤ xT Bx, for all
x ∈ Rn . Then, A and B, if they are Laplacians, are spectrally similar if σ1 B  A  σB.
In this case, they have similar eigenvalues, since: from the Courant-Fischer results, if
λ1 , . . . , λn are the eigenvalues of A and λ̃1 , . . . , λ̃n are the eigenvalues of B, then for all
i we have that σ1 λ̃i ≤ λi ≤ σ λ̃i .
– More generally, spectral similarity means that the two graphs will share many spectral
or linear algebraic properties, e.g., effective resistances, resistance distances, etc.
• Distance similarity. If one assigns a length to every edge e ∈ E, then these lengths induce
a shortest path distance between every u, v ∈ V . Thus, given a graph G = (V, E, W ), we can
let d : V × V → R+ be the shortest path distance. Given this, we can define the following
notion of similarity.
Definition 73. Given two graphs, G = (V, E, W ) and G̃ = (V, Ẽ, W̃ ), on the same vertex
set, we say that G and G̃ are σ-distance similar if, for all pairs of vertices u, v ∈ V , we have
that
1˜
˜ v).
d(u, v) ≤ d(u, v) ≤ σ d(u,
σ
Note that if G̃ is a subgraph if G, then dG (u, v) ≤ dG̃ (u, v), since shortest-path distances can
only increase. (Importantly, this does not necessarily hold if the edges of the subgraph are
re-weighted, as they were done in the simple algorithm from the last class, when the subgraph
is constructed; we will get back to this later.) In this case, a spanner is a subgraph such that
distances in the other direction are not changed too much.
Definition 74. Given a graph G = (V, E, W ), a t-spanner is a subgraph of G such that for
all u, v ∈ V , we have that dG̃ (u, v) ≤ tdG (u, v).
There has been a range of workin TCS on spanners (e.g., it is known that every graph has
a 2t + 1 spanner with O n1+1/ǫ edges) that isn’t directly relevant to what we are doing.
We will be most interested in spanners that are trees or nearly trees.

Definition 75. Given a graph G = (V, E, W ), a spanning tree is a tree includes all vertices
in G and is a subgraph of G.
There are various related notions that have been studied in different contexts: for example,
minimum spanning trees, random spanning trees, and low-stretch spanning trees (LSSTs).
Again, to understand some of the differences, think of a path versus a cycle. For today, we
will be interested in LSSTs. The most extreme form of a sparse spanner is a LSST, which
has only n − 1 edges but which approximates pairwise distances up to small, e.g., hopefully
polylog, factors.

254

M. W. Mahoney

26.4

Sparsified graphs

Here is an aside with some more details about sparsified graphs, which is of interest since this is
the first step of our Laplacian-based linear equation solver algorithm. Let’s define the following,
which is a slight variant of the above.
Definition 76. Given a graph G, a (σ, d)-spectral sparsifier of G is a graph G̃ such that
I. G̃ is σ-spectrally similar to G.
II. The edges of G̃ are reweighed versions of the edges of G.
III. G̃ has ≤ d|V | edges.
Fact. Expanders can be thought of as sparse versions of the complete graph; and, if edges are
weighted appropriately, they are spectral sparsifiers of the complete graph. This holds true more
generally for other graphs. Here are examples of such results.



spectral sparsifier. This was shown by
• SS showed that every graph has a 1 + ǫ, O( log(n)
2
ǫ
SS with an effective resistance argument; and it follows from what we discussed last time: last
time, we showed that sampling with respect to the leverage scores gives a subspace embedding,
which preserves the geometry of the subspace, which preserves the Laplacian quadratic form,
which implies the spectral sparsification claim.

√
,
d
-spectral sparsifier (which in general
• BSS showed that every n node graph G has a √d+1
d−1
is more expensive
to compute than running a nearly linear time solver). In particular, G has

a 1 + 2ǫ, ǫ42 -spectral sparsifier, for every ǫ ∈ (0, 1).

Finally, there are several ways to speed up the computation of graph sparsification algorithms,
relative to the strawman that we presented in the last class.

• Given the relationship between the leverage scores and the effective resistances and that the
effective resistances can be computed with a nearly linear time solver, one can use the ST or
KMP solver to speed up the computation of graph sparsifiers.
• One can use local spectral methods, e.g., diffusion-based methods from ST or the push algorithm of ACL, to compute well-balanced partitions in nearly linear time and from them
obtain spectral sparsifiers.
• Union of random spanning trees. It is known that, e.g., the union of two random spanning
trees is O(log(n))-cut similar to G; that the union of O log2 (n)/ǫ2 reweighed random spanning trees is a 1+ǫ-cut sparsifier; and so on. This suggests looking at spanning trees and other
related combinatorial quantities that can be quickly computed to speed up the computation
of graph sparsifiers. We turn to this next.

26.5

Back to Laplacian-based linear systems

KMP considered the use of combinatorial preconditions, an idea that traces back to Vaidya. They
coupled this with a fact that has been used extensively in RLA: that only approximate leverage
scores are actually needed in the sampling process to create a sparse sketch of L. In particular,

Lecture Notes on Spectral Graph Methods

255

they compute upper estimates of the leverage scores or effective resistance of each edge, and they
compute these estimates on a modified graph, in which each upper estimate is sufficiently good.
The modified graph is rather simple: take a LSST and increase its weights. Although the sampling
probabilities obtained from the LSST are strictly greater than the effective resistances, they are not
too much greater in aggregate. This, coupled with a rather complicated iterative preconditioning
scheme, coupled with careful accounting with careful data structures, will lead to a solver that runs
in O (m log(n) log(1/ǫ)) time, up to log log(n) factors. We will discuss each of these briefly in turn.
Use of approximate leverage scores. Recall from last class that an important step in the
algorithm was to use nonuniform importance sampling probabilities. In particular, if we sampled
edges from the edge-incidence matrix with probabilities {pi }m
i=1 , where each pi = ℓi , where ℓi is
the effective resistance or statistical leverage score of the weighted edge-incidence matrix, then we
showed that if we sampled r = O (n log(n)/ǫ) edges, then it follows that
kI − (SUΦ )T (SUΦ ) k2 ≤ ǫ,
from which we were able to obtain a good relative-error solution.
Using probabilities exactly equal to the leverage scores is overkill, and the same result holds if we
use any probabilities p′i that are “close” to pi in the following sense: if
p′i ≥ βℓi ,
P
′
for β ∈ (0, 1] and m
i=1 pi = 1, then the same result follows if we sample r = O (n log(n)/(βǫ))
edges, i.e., if we oversample by a factor of 1/β. The key point here is that it is essential not to
underestimate the high-leverage edges too much. It is, however, acceptable if we overestimate and
thus oversample some low-leverage edges, as long as we don’t do it too much.
m
In particular, let’s say that we have the leverage scores {ℓ1 }m
i=1 and overestimation factors {γi }i=1 ,
where each γi ≥ 1. From this, we can consider the probabilities

γi ℓ i
.
p′′i = Pm
i=1 γi ℓi

Pm
If
i=1 γi ℓi is not too large, say O (n log(n)) or some other factor that is only slightly larger
than n, then dividing by it (to normalize {γi ℓi }m
i=1 to unity to be a probability distribution) does
not decrease the probabilities for the high-leverage components too much,
P and so we can use the
probabilities p′′i with an extra amount of oversampling that equals β1 = m
i=1 γi ℓi .
Use of LSSTs as combinatorial preconditioners. Here, the idea is to use a LSST, i.e., use
a particular form of a “combinatorial preconditioning,” to replace ℓi = ℓ(uv) with the stretch of the
edge (uv) in the LSST. Vaidya was the first to suggest the use of spanning trees of L as building
blocks as the base for preconditioning matrix B. The idea is then that the linear system, if the
constraint matrix is the Laplacian of a tree, can be solved in O(n) time with Gaussian elimination.
(Adding a few edges back into the tree gives a preconditioned that is only better, and it is still
easy to solve.) Boman-Hendrickson used a LSST as a stand-along preconditioner. ST used a
preconditioner that is a LSST plus a small number of extra edges. KMP had additional extensions
that we describe here.
Two question arise with this approach.

256

M. W. Mahoney
• Q1: What is the appropriate base tree?
• Q2: Which off-tree edges should added into the preconditioner?

One idea is to use a tree that concentrates that maximum possible weight from the total weight
of the edges in L. This is what Vaidya did; and, while it led to good result, the results weren’t
good enough for what we are discussing here. (In particular, note that it doesn’t discriminate
between different trees in unweighted graphs, and it won’t provide a bias toward the middle edge of
a dumbbell graph.) Another idea is to use a tree that concentrates mass on high leverage/influence
edges, i.e., edges with the highest leverage in the edge-incidence matrix or effective resistance in
the corresponding Laplacian.
The key idea to make this work is that of stretch. To define this, recall that for every edge (u, v) ∈ E
in the original graph Laplacian L, there is a unique “detour” path between u and v in the tree T .
Definition 77. The stretch of the edge with respect to T equals the distortion caused by this detour.
In the unweighted case, this stretch is simply the length of the tree path, i.e., of the path between
nodes u and v that were connected by an edge in G in the tree T . Given this, we can define
the following.
Definition 78. The total stretch of a graph G and its Laplacian L with respect to a tree T is the
sum of the stretches of all off-tree edges. Then, a low-stretch spanning tree (LSST) T is a tree
such that the total stretch is low.
Informally, a LSST is one such that it provides a good “on average” detours for edges of the graph,
i.e., there can be a few pairs of nodes that are stretched a lot, but there can’t be too many such pairs.
There are many algorithms for LSSTs. For example, here is a result that is particularly relevant
for us.
Theorem 60. Every graph G has a spanning tree T with total stretch Õ (m log(n)), and this tree
can be found in Õ (m log(n)) time.
In particular, we can use the stretches of pairs of nodes in the tree T in place of the leverage scores
or effective resistances as importance sampling probabilities: they are larger than the leverage
scores, and there might be a few that much larger, but the total sum is not much larger than the
total sum of the leverage scores (which equals n − 1).
Paying careful attention to data structures, bookkeeping, and recursive preconditions.
Basically, to get everything to work in the allotted time, one needs the preconditioner B that is
extremely good approximation to L and that can be computed in linear time. What we did in the
last class was to compute a “one step” preconditioner, and likely any such “one step” preconditioned
won’t be substantially easier to compute that solving the equation; and so KMP consider recursion
in the construction of their preconditioner.
• In a recursive preconditioning method, the system in the preconditioned B is not solved
exactly but only approximately, via a recursive invocation of the same iterative method. So,
one must find a preconditioned for B, a preconditioned for it, and so on. This gives s multilevel

Lecture Notes on Spectral Graph Methods

257

hierarchy of progressively smaller graphs. To make the total work small, i.e., O(kn), for some
constant k, one needs the graphs in the hierarchy to get small sufficiently fast. It is sufficient
that the graph on the (i + 1)th level is smaller than the graph on the ith level by a factor
1
. However, one must converge within O(kn). So, one can use CG/Chebyshev, which
of 2k
need O(k) iterations to converge, when B is a k2 -approximation of L (as opposed to O(k2 )
iterations which are needed for something like a Richardson’s iteration).
So, a LSST is a good base; and a LSST also tells us which off-tree edges, i.e., which additional
edges from G that are not in T , should go into the preconditioner.

• This leads to an Õ m log2 (n) log(1/ǫ) algorithm.

If one keeps sampling based on the same tree and does some other more complicated and careful
stuff, then one obtains a hierarchical graph and is able to remove the the second log factor to yield
a potentially practical solver.
• This leads to an Õ (m log(n) log(1/ǫ)) algorithm.
See the BSST and KMP papers for all the details.

