Practical limits of big-O performance scaling
While I'm not entirely sure what you're getting at, there are a number of frameworks that explicitly capture the cost changes that show up when we hit real resource limits. Three examples: 


the external memory framework that emphasizes the expense of going to disk (idealized by setting main memory reference costs to 0, and disk reference costs to some fixed constant)
The cache-oblivious model that assumes an unknown cache size and performance hits for going beyond.
The streaming computational model in which only you're only allowed sublinear working storage. 


Are these along the lines that you're thinking ?