Does "open source hardware" only apply to controllers or to all components?
"Open Source" means quite literally that the source is open.

In the context of software that means the source code.

In the context of hardware that means the schematics, PCB designs, BOM, etc are all freely available.

What the hardware may be is of no concern, only that the designs are "open".

As far as components are concerned - you can think of a component as a single statement in a program, such as "if" or "else" or "while".  You put those together to make a program in the same way you put the components together to make a circuit.  It's not the components that are open source, but the finished design.

Similarly, "compound components" or "modules" could be open source, as these would equate to a "library" in programming parlance - a pre-written piece of code to do a specific job that you include into your program, but the source code is available for if you want it.