Are there any benefits/caveats with using Entity Reference over Term Reference?
Swings and roundabouts really, it depends on your use case...

Behind the scenes there's essentially no difference - both field types relate one entity to another by way of an ID, so it really comes to down which type provides you with the 'frills' you need/want.

As you mentioned Taxonomy has the open tagging widget, which is nice, but then Entity Reference has Views integration (by way of Entity Reference View Widget) which means you can define as complex a view as you like for your selection widget. Also very nice.

And then there's Inline Entity Form which you also mentioned. That's a superb tool obviously, but only really useful if you need it in your project. 

What Taxonomy has going for it is that it's part of core, and doesn't require a bunch of other modules installed to work. But then in a normal Drupal build a lot of those modules will probably already be there. It also maintains a term index table for nodes which can help with performance, but obviously only if you're relating your terms to node types.

I think your choice is going to come down to what widgets/formatters you need for a particular situation to be honest, it's too tight to call :)