Can I use tile sealant in place of grout?
This happens on every tile install at least a little. There's nothing wrong with some mortar squishing out. Now, if it's over the top of the tiles, that's another issue. If you think there's room for a caulking/grouted/sealant, then there's likely room for grout.

You can use a sealant in place of grout, but from my experience having done this before, the sealant attracts dirt which seems to stick to it vs. grout where you can just sweep dirt off your tiles. Also, the sealant is less durable to foot traffic and wear and it's a huge pain to put in.

Grout is just colored/sanded cement. It will adhere to your mortar that is already there. No one will be able to tell unless maybe you want white grout and your mortar is grey. I'd just use grout anyway and see how well it covers. And if you really think you need to remove some of the mortar from between tiles, you will make your life a lot easier by using a multi-tool with the proper vibrating blade attachment which is made for masonry. You can get a carbide masonry blade that will fit most of the common multi-tools. It's meant for grout removal, but it may help a lot for your situation as well if the mortar is sticking up really high.