Post Bitcoin mine reward: Lowering Difficulty an Option?
The difficulty is not constantly increasing. It is increasing in response to the hashrate going up. If the hashrate would be going down instead, the difficulty would automatically go down as well.

The goal of the difficulty adjustment algorithm is to keep the average time per block at 10 minutes.

If in response to dwindling fees and/or subsidy, mining would become less profitable, we will likely see a decrease in hashrate. That would result in a reduction to the difficulty, until mining is profitable again (to some).