Is it impossible to determine the inverse Z-transform without any other information?
It's true that the algebraic expression for the $\mathcal{Z}$-transform is generally not sufficient for computing the corresponding time-domain sequence. The additional information we need is the region of convergence (ROC). The ROC together with the expression for the $\mathcal{Z}$-transform uniquely determines the corresponding time-domain sequence.

For the given example

$$H(z)=\frac{1}{1-az^{-1}}\tag{1}$$

it is clear that the expression $(1)$ can be obtained in two ways. First,

$$\begin{align}-\sum_{n=-\infty}^{-1}a^nz^{-n}&amp;=-\sum_{n=1}^{\infty}a^{-n}z^n\\&amp;=-\frac{\frac{z}{a}}{1-\frac{z}{a}}\\&amp;=\frac{1}{1-az^{-1}},\quad |z|&lt;|a|\end{align}$$

corresponding to the sequence $h[n]=-a^nu[-n-1]$, and, secondly,

$$\sum_{n=0}^{\infty}a^nz^{-n}=\frac{1}{1-az^{-1}},\quad |z|&gt;|a|$$

corresponding to $h[n]=a^nu[n]$. In both cases I've used the formula for the geometric series.

The only difference is the ROC, so it's crucial to know the ROC in addition to the expression for the $\mathcal{Z}$-transform. The ROC $|z|&gt;|a|$ indicates a right-sided sequence, whereas the ROC $|z|&lt;|a|$ corresponds to a left-sided sequence. For higher order systems, you can also have an annulus $|a|&lt;|z|&lt;|b|$ as ROC, corresponding to a two-sided sequence.

If the sequence is interpreted as the impulse response of a system, then the system is stable if the ROC contains the unit circle $|z|=1$. It is causal if the sequence is zero for $n&lt;0$.