When to use a special high frequency PCB material and not Fr-4? Rogers
FR-4 really only means the material is flame retardent; that may seem an oversimplification, but there are many varieties of FR-4 that are specified to 10GHz.

As a couple of examples, see Isola 370HR; this is a good general purpose laminate that I have successfully used with 5Gb/sec signalling rates. Above that, the variation of dielectric constant with frequency starts to make ISI troublesome over any reasonable distance.

For higher speeds, I have used 408HR as it has a very flat dielectric constant across frequency (to say nothing of a much lower loss tangent); there are numerous FR-4 type laminates with quite high performance for the simple reason you cite: Cost. I have used this particular laminate with signals (differential) of 10Gb/sec.

There are numerous suppliers, and all have high speed offerings.

Interestingly, many flexible materials have superior performance in terms of loss tangent.

Exotic materials have their place, but more usually they are found in microwave equipment.