How to implement a custom ray tracing algorithm in Blender?
A very broad question. I know there is functionality in OSL for this but OSL equals Cycles. So if it should be possible in OSL it is possible with nodes. OSL does not have that much of references. The people who explored this field are Blender Sushi and the blog Small Blenderthings.

You do not exactly state what you are after so I can not help you that much further.