Load an OpenGL buffer from file
As for your questions:


Just use int, unless this is critical code where benchmarks have shown that integers are noticeably slower. Furthermore: your char will support 255 vertices which isn't a lot, I've loaded models with millions of vertices.
This level of error checking is possibly an indication that your code smells. You have a function that accepts many parameters, perhaps you can split it into multiple functions or a class type of structure.
Myself, I make everything const unless it cannot be const.
If the size is known at compile time, how about std::array instead of std::vector? As for std::unordered_set it seems too fancy for the job (e.g., a self balancing tree with 3 items, whoop! whoop!), but I have no strong arguments.
No opinion. Myself, I never use C++ exceptions - due to my lack of understanding / education. In other languages I never directly use the build-in Exception types, I always subclass my own - if anything it helps the library user understand if they are library errors or system errors.
You can place your new'ed memory in a std::unique_ptr or use the resize features of std::vector or std::string


Couple more random things:


file.close(); is optional, the std::ifstream will also close the resource (if it was open).
You're not testing for OpenGL related errors.
I don't like 'out' arguments, such as vertexCount. Just return a std::tuple with multiple arguments, or a struct.
Does your file contain binary data? You're reading it from disk, and directly upload it to the GPU without parsing/validation? If anything, make sure you've actually read the amount of bytes that tellg claims are available.
std::ifstream has a constructor, use it.