Solving a recurrence
Assuming that the end goal is to get an asymptotic bound on the time or space complexity of the algorithm that this recurrence relation represents, then start with $n = 4k^2$ s.t. $k &gt; 1$ and notice the following:

$T(4k^2) = T(2k^2 + 2k) + 4k^2$

$T(2k^2 + 2k) = T( k^2 + k + \sqrt{ 2k(k + 1) } ) + 2k^2 + 2k$

$T(k^2 + k + \sqrt{ 2k(k + 1) }) = T(\ldots) + k^2 + k + \sqrt{ 2k(k + 1) }$

Substituting backward

$T(4k^2) = 7k^2  + 3k + \sqrt{ 2k^2 +2k } + T(\ldots)$

Substituting the original value of $n$

$T(n) = \frac{7}{4} n + \frac{3}{2} \sqrt{n} + \sqrt{\frac{1}{2}n + \sqrt{n}} + T(...) $

Each successive term is an order smaller than the one prior and as we look at the asymptotic behavior, those smaller terms will vanish and the linear term will dominate, thus $\mathcal{O}(n)$.