What are Spherical Harmonics & Light Probes? What are Spherical Harmonics &amp; Light Probes? How useful are they in computer graphics? What exactly do they do? I've heard the word spherical harmonics &amp; Light Probes everywhere, from siggraph presentations to blog posts.

Recently Matt Pettineo posted a 6 part blog series on them but I still don't understand what they are.

Is it another way to improve ambient lighting?
Spherical harmonics

If you know what a Fourier transform is, you already almost know what spherical harmonics are: they're just a Fourier transform but on a spherical instead of a linear basis. That is, while a Fourier transform is a different way of representing a function $f(x)$, spherical harmonics are the analogous thing for polar functions $f(\theta, \phi)$.

If you don't know what a Fourier transform is, you kinda need to know before you can understand spherical harmonics. The Fourier transform lets you represent a signal as a series of sine and cosine waves, each with twice the frequency of the last. That is, you can represent the signal as its average, plus a sine wave whose wavelength is the same as the length of the signal, plus a sine wave twice that wavelength, and so on. Because the Fourier transform fixes you to these particular wavelengths, you only need to record the amplitude of each one.

We commonly use Fourier transforms to represent images, which are just 2D digital signals. It's useful because you can throw away some of the sine waves (or reduce the precision with which you store their amplitude) without significantly changing what the image looks like to human eyes. OTOH, throwing away pixels changes the look of the image a lot.

In a sampled signal like an image, if you use the same number of sine waves as there were samples (pixels) in the original image, you can reconstruct the image exactly, so once you start throwing away any frequencies, you're making the image take less storage.

Spherical harmonics are just like Fourier transforms, but instead of sine waves, they use a spherical function, so instead of linear functions (such as images), they can represent functions defined on the sphere (such as environment maps).

Light probes

Just like how a standard image records all the light reaching a certain point through the image plane, a light probe records all the light reaching a certain point from all directions. They first came out of movie effects. If you want to add a computer-generated object to a real-world scene, you need to be able to light the synthetic object with the real-world lighting. To do that, you need to know what light reaches the point in the scene where the synthetic object will be. (N.B. Although I say "lighting", you're recording an image of all the light, so it can be used for reflections as well.)

Because you can't have a camera with a spherical lens that records all the light reaching a single point from all directions, you record this by taking normal photographs of a spherical mirror, and then reprojecting the images onto a sphere.

Outside of movie effects, it's more common to use light probes generated from an artificial scene. Imagine you have some expensive algorithm to compute global illumination (GI) in a scene, and you also have some smaller objects moving around in this scene (such as a game level with players in it). You can't run the whole GI algorithm every time any object moves, so you run it once with the static scene, and save light probes taken at various points in the level. Then you can get a good approximation to the GI by lighting the player with whichever light probe they're closest to.

Using them together

Generally you want to filter out sharp edges in your global illumination anyway, so you want a way to represent them that's compact and easily lets you throw away high frequencies. That's what spherical harmonics are really good at! That's why you'll hear these two terms used together a lot.

You compute light probes with your expensive GI algorithm - typically in the level-design tool, or maybe once per second (instead of once per frame) if you want to include your dynamic objects to. You store those cheaply with spherical harmonics - 16 floats is enough for pretty high quality lighting, but not reflections. Then for each dynamic object you want to light, you pick the nearest light probe (or linearly interpolate several together) and use it as a uniform or constant input to your shader. It's also commonplace to use spherical harmonics to represent ambient occlusion data, and it's very cheap to convolve that with the light probe, though there's some complexity around rotating spherical harmonic functions.