field_get_items gives me an array of node IDs. Can I filter these by type without using node_load()? I have an entity reference user field for more than one content type. I need to filter the node IDs by content type without using node_load(). Is it possible?

If not, how badly would it affect performance to send the user ID to a view and filter them there?
Unless you have a map of those ids -> types somewhere outside the DB to reference, no, it's not possible. You could go directly into the node table yourself, of course, and avoid node_load specifically, but there's no way to do it without a db hit.

It's impossible to say how performant a View would be without knowing how your site is built, server specs, traffic info, and so on. The best thing to do would be to try it and benchmark.