Why does ammonium oxalate cause swelling of RBCs?
After a lot of search I think the following seems a satisfactory answer.

You are correct in thinking that outside the RBC, $\ce{NH_4^+}$ converts to $\ce{NH_3}$ and once inside the RBC, it again converts to $\ce{NH_4^+}$. But this does not explain how $\ce{NH_4^+}$ swells up the RBCs. For that, let us look at the process from the beginning.

As described by Hemker et al, 2003, an important part of this process is the enzyme carbonic anhydrase (an absolutely important constituent of RBCs). Hemker et al, while studying how the Rh complex in RBCs exports $\ce{NH_4^+}$ ions out of the cell (although they used $\ce{NH_4Cl}$ in their study, it doesn't matter since blood already contains enough $\ce{Cl^-}$) they came up with the following scenario: 


$\ce{NH_4^+}$ breaks into $\ce{NH_3}$ and $\ce{H^+}$ and enters inside the RBC. $\ce{H^+}$ combines with $\ce{OH^-}$ (coming from $\ce{HCO_3^}-$, we'll talk about this later) to give $\ce{H_2O}$.
Due to increased osmotic pressure (by $\ce{NH_3}$), $\ce{H_2O}$ enters inside the RBC where it is again broken into $\ce{H^+}$ and $\ce{OH^-}$. $\ce{NH_3}$ combines with $\ce{H^+}$ to give $\ce{NH_4^+}$ again (and we're done with $\ce{NH_4^+}$ here, though Hemker et al took this further).
The $\ce{OH^-}$ (from water) combines with $\ce{CO_2}$ (via carbonic anhydrase) to give $\ce{HCO_3^-}$. This $\ce{HCO_3^-}$, via Hamburger shift, is exchanged with a $\ce{Cl^-}$ while the $\ce{HCO_3^-}$ outside converts into $\ce{H^+}$, $\ce{OH^-}$ and $\ce{CO_2}$ (providing $\ce{OH^-}$ to $\ce{NH_4^+}$). See the diagram below for visual representation:




So, where does the swelling effect come in? It turns out, it is due to the increasing concentration of $\ce{NH_4^+}$ and $\ce{Cl^}-$ inside the RBC which keeps on generating higher and higher osmotic pressure. Hemker et al also found that Rhnull RBCs (lacking Rh complex) became victims to osmolysis much faster than the normal RBCs, suggesting that increased concentration of intracellular $\ce{NH_4^+}$ is the main cause of swelling and osmolysis of RBCs (in their research, they concluded that Rh factor indeed exports $\ce{NH_4^+}$ outside the RBCs).

Reference: Hemker, M. B., Cheroutre, G., Van Zwieten, R., Maaskant-van Wijk, P. A., Roos, D., Loos, J. A., Van Der Schoot, C. E. and Von Dem Borne, A. E. G. Kr. (2003), The Rh complex exports ammonium from human red blood cells. British Journal of Haematology, 122: 333–340. doi:10.1046/j.1365-2141.2003.04425.x