OptiX CUDA out of memory error when rendering
The likely reason why the scene renders in CUDA but not OptiX is because OptiX exclusively uses the embedded video card memory to render (so there's less memory for the scene to use), where CUDA allows for host memory + CPU to be utilized, so you have more room to work with. I do know that OptiX isn't fully stable at the moment, Blender is still working on support for NVIDIA cards, so it could be that.
Things you could do to help make sure it renders:

Reduce your texture resolution. Bigger textures == a lot of memory eaten up
If you're using sub-surf modifiers like (especially adaptive subdivisions which are super costly) try to set them to lower # of subdivisions, or not use them at all.
Reduce the number of vertices in your scene

Personally, I've also tried:

Reducing the complexity of textures: Really intense procedural generated textures eat tons of memory too