Why are transistors difficult to use in special cases such as >100 MHz operation?
You premise is flawed. 

It is not that transistors are difficult above 100 MHz, it is that any specialized circuit will require appropriate engineering. Engineering high frequency circuits using any components requires particular attention to lead lengths due to the wavelengths of the frequencies evolved. Wavelength is inversely proportional to frequency. In fact for radio waves in space wavelength = (speed of light)/frequency.

If you think about it, virtually all FM radios, TVs, cellphones and internet wireless routers use transistors (inside of integrated circuits) these days rather than the tubes (valves) that used to be used. 100 MHz is in the FM radio band, and routers operate in the GHz range.

What does cause issues in designing high frequency circuits is that as wires between components become a significant fraction of the wavelength involved they start behaving more like transmission lines, antennas, inductors and capacitors. This is not a problem with audio circuits where the wavelength is enormous.

Other factors to consider include crosstalk (signals from one wire appearing on another where they are not wanted) and shielding.