How to get intersection of a specific object and the mouse in the BGE?
After some discussion with gandalf3 we found a solution:
The main problem was that the rayCast function will cast a ray between the centers of two objects: Those passed as objto and objfrom arguments. Thus the ray's hitpoint always was the position of objto, not the position over which the mouse hovered.

But rayCast also accepts 3D points as inputs. The trick is to use a Mouse sensor, set to 'Mouse Over Any', and connect it to the Python Controler which holds the script:


In the script one can use the rayTarget and raySource properties of the mouse over sensor as inputs for the rayCast funtion. The modified code looks as follows:

from bge import logic, render

render.showMouse(False)
#from bge

def main():
    scene = logic.getCurrentScene()
    cont = logic.getCurrentController()
    mouse_over = cont.sensors["mouse_over"]

    if mouse_over.positive:
        tracker = scene.objects["mainTarget"]
        crosshair = scene.objects["Cross_hair"]
        floor = scene.objects["Floor"]
        object, hitpoint, normal = crosshair.rayCast(mouse_over.rayTarget, mouse_over.raySource, 0.0, "isTrackerFloor", 0, 1, 0)
        tracker.worldPosition = hitpoint


One further note: The 'Edit Object' actuator which is set to 'Track To' seems to make the affected object point to the target with its local y-axis. Make sure the object with the actuator is rotated correctly in edit mode.