Specific disadvantages to storing infrequently accessed large files in a large database
InnoDB Architecture



This first thing that comes to mind is the InnoDB Buffer Pool. It holds three things


Data Pages from Accessed .ibd files
Index Pages from Accessed .ibd files
Insert Buffer for Changes to Secondary Indexes


Any query accessing infrequent data will basically remove the working dataset from the InnoDB Buffer Pool just to load the adhoc data. To get those data back, you have two options:


OPTION 1 : Reread the data and index pages from disk. You are certainly going to experience a cache storm  that will impact Server Load with Disk I/O, Memory Utilization, and possible swapping.
OPTION 2 : Unload Buffer Pool Before Accessing Old Data and Reload Buffer Pool Afterwards. This is a new feature for MySQL 5.6. Apparently, it was made for occasions like this. Here are the options for loading the InnoDB Buffer Pool at Startup, Shutdown and On-Demand :


innodb_buffer_pool_dump_now
innodb_buffer_pool_dump_at_shutdown
innodb_buffer_pool_load_now
innodb_buffer_pool_load_at_startup



EPILOGUE


OPTION 1 is something you reluctantly live with until all subsequent queries bring back your most recent working dataset 
OPTION 2 requires scripting adhoc reports to do the following


Backing up the InnoDB Buffer Pool
Run the adhoc queries on the infrequent data
Restoring the InnoDB Buffer Pool

OPTION 2 requires upgrading to MySQL 5.6
There are other ways MySQL and InnoDB are impacted. I discussed these in my recent post How to optimize MySQL for large BLOB updates