Meaning of the determinant of an ABCD transmission matrix
That's an eigenvalue problem: The goal is not to calculate the determinant, the goal is to calculate the eigenvalues and -vectors of that matrix.
If you go back to your &quot;higher math 1&quot; or 2&quot; course, then you'll find that you always find the eigenvalues by subtracting \$\lambda\mathbf I\$ from a matrix, then writing down the determinant of that (which is a polynomial in \$\lambda\$), and finding the zeros of that.
Here, we're looking for eigenvalues that are some \$e^{-\gamma d}\$ to determine these parameters. That's why we directly subtract \$e^{-\gamma d}\mathbf I\$.
The intuition to finding the eigenvalue: For an eigenfunction of the system (and this we're expecting single tones  \$e^{j\pi f t}\$ to be eigenfunctions), the relationship between in- and output can be described as a single complex number. That complex number is literally what we're looking for.