What are the downsides of using varbinary[max]?
[MAX] columns are to be considered like BLOBs (Binary Large Objects) and are physically stored on another "page".
This means that practically, they are like on another table, seamlessly linked to the one with the rest of the data.
One of the main limitations is that you can't index the [MAX] column.

I don't see any issue in you doing things as you suggest.