maximum output of a LTI system within linear limits In book , 'fundamental of signal and system'by M.J ROBERT it is written that 


  Since no practical system can ever produce an unbounded response, strictly speaking, all practical systems are stable. The ordinary operational meaning of BIBO instability is a system described
  approximately by linear equations that would develop an unbounded response to a bounded excitation if the
  system remained linear. Any practical system will become nonlinear when its response reaches some large
  magnitude and can never produce a truly unbounded response. So a nuclear weapon is a BIBO-unstable system
  in the ordinary sense but a BIBO-stable system in the strict sense. Its energy release is not unbounded even
  though it is extremely large compared to most other artificial systems on earth


For a given LTI system, what parameter of system along with input which  determines the maximum magnitude of response after which it becomes non linear and how to calculate this maximum magnitude of response mathematically?
For a given LTI system, …, after which it becomes non linear


never. If you're modeling a system to be LTI, it's LTI: Linear (and time-invariant).

You can of course model a system to be non-linear in general, but linear for a range  of input. But then the answer to your question is "that's in your own model, and we can't tell you"!