Splitting a table with large columns for performance?
Generally this can make sense, BUT: you have first to make sure the data is in the table.

If you use large table fields (varbinary(max)) then the data is actually not stored in the table but in separate pages anyway. If the original table as as thin as you indicate (id, date upload, not a lot more fields) Then this is a fallacy, with one exception: EF will gladly load all the data on object access - which makes acess slow thanks to EF inefficiency.

You can avoid that by not using EF for accessing the file field. That does make a TON of sense because you may never want to load the whole file into memory anyway. You want to be able to access the file in a loop reading max x bytes - so you can stream them off.

Welcome to the limitations of EF and the even worse limitations of code first.