Why Fe(CO)5 exists, but Fe(NH3)5 doesn't?
While we cannot be sure that $\ce{Fe(NH3)5}$ does not exist, the bonding between ligand and metal offers a good explanation for why it is expected to be less stable than $\ce{Fe(CO)5}$.

In metal carbonyls like $\ce{Fe(CO)5}$, the bonding between CO and the metal consists of three single bonds, 1 $\sigma$ and 2 $\pi$ bonds. The $\sigma$ bond is formed by overlap of the carbon $sp$ hybrid orbital (which contains the lone pair) with an empty $d$ orbital of the metal. Electron density is donated from the orbital of CO into the metal orbital, so CO acts as a $\sigma$ donor. The two other $\pi$ bonds are formed by overlap of a filled metal $d$ orbital with a pair of $\pi^*$ molecular orbitals of CO. Electron density is donated from the metal orbital into the molecular orbitals of CO, so CO acts as a $\pi$ acceptor.

 

For the $\pi$ bonding to occur, the metal is required to have sufficient $d$ electrons, and thus preferably a lower oxidation state. This explains the relatively low oxidation state of iron (0) in $\ce{Fe(CO)5}$. The combination of one $\sigma$ and 2 $\pi$ bonds gives the M-CO bond a partial triple-bond character, while the C-O triple bond is weakened by the transfer of electron density into the $\pi^*$ orbitals (reference).



Because of its ability to serve as a $\sigma$ donor and $\pi$ acceptor simultaneously, CO is a stronger ligand than $\ce{NH3}$ which can only act as a $\sigma$ donor with its lone pair in the nitrogen $sp^3$ hybrid orbital. $\ce{NH3}$ lacks a $\pi$ system and therefore cannot act as a $\pi$ acceptor. Because $\pi$ backbonding is crucial for stabilization of a metal center in low oxidation state, ammonia is unlikely to form stable ammin complexes which are analogous to the corresponding metal carbonyls.