How to calculate thermal conductivity of potting compound around complex shape
The material you suggest is electrically insulating and has reasonable thermal conductivity. There are thermally better materials available such as 3M TC-2810 which is about 2x as good. 

It's possible to predict the heat flow if you have access to appropriate thermal analysis software (often mechanical FEA programs will do this). An example would be MSC NASTRAN. However, cost and learning curve are both steep. 

You can easily measure the temperature rise of a built inductor by measuring the resistance delta of the coil, since the temperature coefficient of copper is known. You would turn the power off and make a measurement before the temperature has a chance to change much. That is a standard method used for transformer temperature rise measurement. You can also attempt to embed sensors in the epoxy, however it's difficult to get a measurement during operation if there is a lot of EMI floating around and the wires close to the high voltage may present other hazards.

Edit: 

You could fill the potting housing with salt water, with a copper plate at the position the heatsink would be in, and wrap the toroid in foil, immerse to the expected position and measure the resistance (use AC). Take the ratio of that to the (measured, with AC) volume resistivity of the salt water gives units of length. Multiply that by the thermal conductivity and you get W/°C.