What is this piece inside the spigot supply hole, and what is it used for?
It is indeed the spray hose diverter. I would not remove it, even if you don't normally use the spray hose. If for no other reason, you'll never be able to use the hose again. 

I am unsure if removing it would cause issue. I'm not brave enough to try. 

I have replaced the fittings two of my Delta faucets. One with a diverter. The gaskets were always my issue. You can buy a kit at big box stores that have a complete gasket set to replace them all.