field_get_items gives me an array of node IDs. Can I filter these by type without using node_load()?
Unless you have a map of those ids -> types somewhere outside the DB to reference, no, it's not possible. You could go directly into the node table yourself, of course, and avoid node_load specifically, but there's no way to do it without a db hit.

It's impossible to say how performant a View would be without knowing how your site is built, server specs, traffic info, and so on. The best thing to do would be to try it and benchmark.