Are carbon road bikes robust enough to use off-road?
The strength of a carbon fiber composite frame is not the problem. There are many CFC gravel, cyclocross and mountain bikes, so obviously CFC as a material can take the impacts. If you ride a road bike on a harsh, bumpy paved surfaces it does not break. If you want to be convinced of how tough CFC frames are, google for videos of the testing that manufactures put them through.

The primary features that make a gravel bike more suitable for bumpy and loose surfaces are stronger wheels, larger tires run at lower pressure, slacker steering geometry and longer chain-stays.