A lot of data in the table is only accessed for 4 days or so, how to handle it afterwards?
Yes, there is a cost to having lots of cold data in actively-queried tables.  One obvious case is that bigger tables have deeper indexes (i.e. more pages between the root and the leaves) so require more IO to read and write.

I'd suggest you use partitioning.  Keep the active partitions in the current table and swap the others out.  Ideally you'd then use a maintenance task to move the historical data to a separate DB for archive.  Make it clear to your external users what the time cut-off is and allow them to query either accordingly.