How do you decide on having more than two layers to have on a PCB?
This is a pretty broad topic, so I'll try to keep the answer general, instead of trying to answer all in detail.
First of all, some components force you use more than two layers. You simply cannot export all signals from a dense BGA chip on two layers. There's simply not enough room.
Then, some problems force you to use more than two layers for traces. Can't get the trace density on two layers; can't control impedance well enough (or with thin enough traces) on two layers; can't get the heat away quickly or the supply current enough without a dedicated power and ground plane... Most importantly, ground return paths are important, and if you need to break your ground plane because a single signal layer simply isn't able to support your circuitry (which easily happens, most schematics are not planar graphs...), then you've lost.
Then, the huge area of things that are trade-offs between complexity, cost, signal quality, ease of routing, manufacturing abilities, ... start, and you could (and will) find library shelves full of advanced PCB design methodology that explains what you can (and cannot) do on any given number of layers. For example, I don't know your salary, but not all boards are produced by the millions. If you need 10 boards of the same design, and layouting things takes you twice as long if you need to do it on two layers instead of having the freedom of four layers, well, it might simply be cheaper and better to do it on four, and move on to the next design.
Note that you say &quot;4 to 6 layers are used&quot;; but really, you can order (as far as I've personally seen) 18 layer boards (the comments show much higher layer counts, even!), so the world is slightly more complex than you think :)