Separating a signal into its constituent components
You do a QR decomposition of $X^T$. Then the system reads as $$Y=AR^TQ^T.$$ Now multiply with $Q$ from the right, $$YQ=AR^T,$$ which already should give you hints on the reliability of the eventual solution, since ideally, the block of $YQ$ right of the leading $m\times n$ block should be zero. If that is true, then $$A=[YQ]_{[1:m,1:n]}[R^T]_{[1:n,1:n]}^{-1}$$.

Note that in the QR decomposition using Householder reflections, you can apply the reflections computed for $X$ simultaneously to $X$ and $Y$, if $$S=I-2 vv^T,$$ then $$X_{next}=XS=X-2(Xv)v^T\text{ and }Y_{next}=YS=Y-2(Yv)v^T$$ and $$X_{final}=R^T,\; Y_{final}=YQ$$.