Ray tracing where illumination itself affects the geometry
You might look into Mitsuba, which is a physically based path tracing renderer that's designed to be easy to extend with plugins. It might be useful as a base for you to build on, implementing your own algorithms as plugins or by modifying its code.
It already contains facilities for measuring the amount of light falling on some geometry, in the form of the &quot;irradiance meter&quot; object. I'm not sure how much it can help you with geometry modification as it's not really designed for that (it just imports geometry from a file and renders it), but maybe you can find some other libraries to help with that part, depending on what kind of geometry operations you need to do.