What is the exact meaning of unstable system in DSP?
Unstable typically means and unbounded output for a bounded input. In other words the output of, say , a filter can get infinitely large although the input is perfectly okay and of "normal" size.
A simple example would be the difference equation $y[n] =  x[n] + y[n-1]$. If we calculate the step response, i.e. $x[n] = u[n]$, we get y[0] = 1, y[1] = 2, y[2] = 3 ...
The output grows infinitely even though the input is a perfectly well behaved signal, bounded by 1.