How are color images used as input values in nodes?
You're correct, it uses brightness. Or rather, luminance.
The color output contains three values, R G and B. When a color output (yellow socket) is plugged into a scalar input (gray socket), it is automatically converted as if using an RGB to BW node.

For example, this can be observed by plugging an image texture into a math node set to multiply by 1:



Before multiplying the values, the R G and B channels are turned into a single grayscale channel. Since we're multiplying by 1, the values are unchanged.

This is the same as the more "proper" method of using a RGB to BW node:



By default blender converts the R G B channels to a single channel while trying to preserve the luminance (see wikipedia).

To do this, the RGB to BW node does something like this:



Once the color channels are converted to a single channel, this can be used to set the mix factor for each pixel (using just the mix factor slider is equivalent to using a solid color image). See How does fac differ with manual control versus node control?.

If you want to use images with very specific values, then you'll want to pay close attention to color management. See: What exactly does the &quot;Non-color data&quot; option for image/environment textures do?