How do I calculate correlated color temperature correctly?
The problem is that I don't know whether should I apply white balance process before calculating CCT. 


Definitely not. Afterwards, the light source would have a completely different color profile, depending on how you implement white balance, and hence, a different color temperature.


  If it should not, how do I get the conversion matrix that convert the current device-dependent RGB color space to CIE XYZ to calculate the algorithm correctly? 


Get the color profile (which might be much more complex than a simple matrix) of your device; usually, that means getting the ICC file for your camera/scanner/sensor, and applying that to convert your image from the device color space to something canonical.

If your camera/scanner/sensor doesn't have such a profile, you will need to calibrate it first.