How do data types influence hardware (CPU / GPU / TPU) performance?
The variation in performance gains you are seeing from reduction precision might be do to different frameworks using different types. Even after downcasting data types, some operations will automatically upcast types. You mention using Pandas and TensorFlow / Keras. Mixing these frameworks leads to unwanted recasting of data types. It is better to use a single framework to avoid recasting.
There is no software (i.e., language, library, or model) agnostic answer. Hardware utilization is software specific. In order to get the reduced precision gains, the same software stack should be throughout the entire modeling process. That software should be designed for the specific chip (i.e., CPU, GPU, or TPU).