How do render passes work? I got a lot of related questions about render passes in Cycles lingering for a while now and apparently I can't figure it out myself.

When it comes to render passes, this manual page with the image below is all the official reference I have found on this topic. This image surely helps, but it is heavily outdated by now. Missing passes:


Shadow, AO - How do they affect the final image?
Subsurface Direct/Indirect/Color - I thing it will behave like the other pass types, but where did this information go previously?
Volumetric - Volumetric information just doesn't show up as a render pass at all, but is still contained in the final image. Why is it missing?


Passes are categorized into lightning passes and data passes. To my understanding lightning passes are somehow encoded in the final image where data passes are only additional information for compositing. Additionally, image passes are noisy by nature and get less noisy with increased amount of samples, where data passes are not affected by samples at all except that they get anti-aliased with increasing samples amount. Am I correct? (related to this question)

Perhaps my biggest question in terms of understanding: How do these passes get generated? AFAIK, Cycles shoots one ray per pixel and sample from the camera where it bounces off from geometry until it hits a light source or gets killed by the Light paths settings. This results in one ray-traced color value per pixel and sample and the colors of every sample get more or less averaged together. Having one color per pixel calculated, where do the pass data come from?
"Cycles shoots one ray per pixel and sample from the camera where it
  bounces off from geometry until it hits a light source or gets killed
  by the Light paths settings"


This is what happens in principle. In practice there are all kinds of tricks and optimizations that make path-tracing much faster than the naive implementation described by you. For example shadow rays are shot from the surfaces in the direction of light sources, and a trick assures that the image converges towards the same image as without shadow rays, but much faster. These shadow rays allow us to collect the "direct" and "indirect" lighting information separately. They are always collected separately, the passes just expose this internal information to you. This shadow rays/direct sampling trick is called "next event estimation" in the CG literature, if you want to learn more about it.


  Shadow, AO - How do they affect the final image?


They don't. The information in the shadow pass is already contained in the other passes, and AO is just some kind of fake global illumination, which is not necessary, as path tracing provides physically based GI. 

About your other questions: I guess the answer is also in the Cycles implementation details. Note that (according to the Cycles roadmap) in the future Cycles will have AOVs (Arbitrary Output Variables) instead of hardcoded passes.