How to invert an affine matrix with small values?
I found a solution to my specific problem. Instead of computing the determinant and hitting the precision wall, I use the Gauss-Jordan method step by step.

In my specific case of affine transformation matrices and the range of values I use, I don't hit any precision problem this way.