OleDbContext to keep OleDbConnection alive
It turns out that OleDb does not have connection pooling and
  opening/disposing it takes a lot of time.  


I have read somewhere, that as long as you keep one connection (for sure with the exact same connection string) open the connection pooling works, but don't count on it, just test it. You could at test initialize open this said connection and close it at test terminate.



Now let us look at your code.... well, there isn't much to say. It is easy to read, you are using braces {}, fields, methods and method parameters are well named.  

Just some little nitpicks:  


what happens if the constructor argument is an empty string ? 
you are testing excel files, why don't you ensure that the passed fileName has at least the correct file extension ?