Can a 12V AGM battery be damaged by too light a load and relying exclusively on a 10.5V low voltage cutoff?
Taken to its extreme limit, you are suggesting that leaving a battery in storage, with no load, will harm the battery is some way that a heavy load would not cause harm. That is clearly not the case.

The key to your question is "the entire setup is not monitored carefully". If you let a battery discharge too far, whether from self-discharge or under heavy load, it may be damaged. Your problem is that you are using the voltage under load to estimate the remaining capacity, and that is a very poor method. It seems that the inverter manufacturer's intent is to protect the battery from damage, rather than trying to extract as much energy from the battery as possible.