How do I remove overlapping faces for this case?
First of all you need to select the inner ring that you created with I. You can use Alt for it. Then deselect everything that is fine with the circle selection tool (C). Then press X and select Edge Collapse. Your problems should now be fixed. For that selection part I made a little picture here:

Just deselect everything which is outside the problem area.