We present an approach to diminish temperature induced color shifts of phosphor converted LEDs. In particular, we discuss the impact of the thermo-optic coefficients of the materials constituting the color conversion elements (CCEs) on the constancy of the CIE chromaticity coordinates. While silicones have a comparably large thermo-optic coefficient, phosphors, e.g., Ce:YAG have a much smaller one. Hence, increasing temperature will lead to an increase in the differences of the refractive indexes of phosphor and matrix. This will enhance light-scattering which for itself would give reason for a more yellowish emission. Thus, on the one hand the thermo-optic coefficient of the silicone matrix has to be considered for as a potential source of failure for color temperature deviation upon device operation. However, on the other hand adjusting the thermo-optic coefficient of the silicone matrix by an appropriate materials engineering to the temperature induced luminescence loss of a phosphor can be applied to counterbalance a shift of the chromaticity coordinates of the white light in this regard.