Cut costly functions in inter-procedural abstract interpretation
Sure.  "Cut off and return Top" is sound.  It's valid.  It seems like a reasonable thing to do.

I don't know if it has been studied or used before in the literature.  I wouldn't be surprised if it has.  It feels to me like a variant on widening, even though it's not identical to the usual widening operator.   I've seen similar methods applied before (if you get stuck, give up and return Top) in other situations.

I don't think there's a single standard approach to this problem.  My impression is that there are multiple techniques. You can memoize analysis of a function (so you lazily generate a transfer function that summarizes the effect of the function, in terms of its arguments, and then reuse that information when you need it again in the future).  You can use widening operators.  You can use counter-example guided refinement to start with a coarse abstraction (which is efficient for static analysis), and then make the abstraction more precise as needed.  I imagine you could also do the converse: start with a fine-grained abstraction, and if it is taking too long, switch to a more coarse abstraction and restart the analysis with that coarser abstraction.  There are probably many other possible techniques, and the best one might depend on the specifics of your particular analysis.