How to formulate reward of an rl agent with two objectives
In general it is not possible to simultaneously optimise two separate objective functions. Your approach of adding weights (your coefficients) to each objective, then summing the scaled objectives, is a standard way of resolving that. 

As your penalties are on different scales and in different units, it is your task as the engineer setting the objective to provide the conversion to a single scale. That is what the coefficients represent - you can even think of them as $points/Joule$ for energy and $points/(\Delta K)$ for temperature difference.

Sometimes analysis will show you that there is a natural combined scale. For instance, in business settings it may be possible to frame compromises as financial costs, e.g. your coefficients might be $\text{GBP}/Joule$ for energy and $\text{GBP}/(\Delta K)$ for temperature difference. Then you have one clear objective to minimise cost or maximise profit.

If that is not possible - a financial cost for exceeding temperature bounds may be hard if this is about human comfort in a building - deeper analysis might lead to thinking about longer-term outcomes. Perhaps your initial rewards are too focused on immediate numerical issues (that appear easy to collect, but don't represent your true goals), and a re-framing of the problem could work. For instance, perhaps it is more reasonable that both the temperature and energy costs stay within strict bounds over a year of varying external temperatures and system workload, with a scaling penalty depending on how badly these are exceeded.