How to Deduce a Linear System's Impulse Response from a Set of Input and Output Signals?
There seems to be an image which is gone now and hence I might be missing something.

In order to declare if the system is invariant you should see if a delay of the input yields only a delay in the output.
In you case, does the input $x_1[n-m]$ yields $y_1[n-m]$ and etc...

As written by @DilipSarwate, By looking at the response for $ {x}_{2} - {x}_{1} $ one could see the system is neither causal nor time invariant:

consider that linearity requires that the response to $ {x}_{2} − {x}_{1} $ be
$ T \left( {x}_{2} − {x}_{1} \right) = T \left( {x}_{2} \right) − T \left( {x}_{2} \right) = {y}_{2} − {y}_{1} $.
Now, $ {x}_{2} − {x}_{1} $ is a pulse of amplitude $ 2 $ at $ n = 2 $ and the response to it is nonzero at $ n = 0, 1, 2 $ and so the system is not causal, and since $ {x}_{2} − {x}_{1} $ is just $ {x}_{2} $ delayed by 2 units while the response is not $ {y}_{2} $ delayed by 2 time units, the system is time varying, not time invariant.


If the input signals are band limited and their bandwidth is less than your system you won't be able to restore the impulse response.
You will be able only to get the response in the frequencies the input has energy in.
This could be done by frequency analysis of the input and the output.
If your system is indeed LTI the connection between input and output is given by convolution with the impulse response.
Convolution is multiplication in the frequency domain, hence you could easily get the impulse response (Again, only at frequencies the input has energy in).

Yet since this system isn't time invariant it doesn't have a single impulse response. It has a response to an impulse at $ n = 0 $ but it might have a different one for an impulse at $ n = 1 $.
Update
This is a nice case to show the commutative property of convolution.
Since $ y \left[ n \right] = \left( h \ast x \right) \left[ n \right] = \left( x \ast h \right) \left[ n \right] $ you can switch the roles between the &quot;System&quot; and the &quot;Input Signal&quot;.
So this question is equivalent to Given known output and system response, how could one estimate the input of the system?
Well, this is a known problem called Deconvolution and in the case of no noise it is simple to solve.
As written above, one way to do so it write the problem in matrix form.