Disable caps lock on backspace in iPhone
It happens because you hit caps before typing a letter, so what the iPhone is doing is rather deleting all the code associated with that letter, it deletes the letter leaving the code, so if you backspace again it removes all the code from that space and moves back another space.

If you backspace one step further it removes this formatting.

The settings you have changed do not relate to your issue.

The first is to capitalize proper nouns that grammatically require a capital (which I wish would happen on all applications) the second is to enable the locking of the caps key, which means if you tap it twice in quick succession it will lock in uppercase.