How can I build the molecular orbital diagram of the complex tetraammincopper(II) (square planar structure)?
In principle, it does not matter which central metal you choose and which ligands; if the geometry is identical, the resulting MOs will be identical, too. The problem is as you state that the complex you wish to determine does not have a square-planar geometry (nor is it a tetracoordinated copper) so whatever you calculate will be off. At first, I was thinking of using a nickel(II) analogue (which would be identical save a missing electron) until I realised that $\ce{ls{-}[Ni(NH3)4]^2+}$ likely doesn’t exist and $\ce{hs{-}[Ni(NH3)6]^2+}$ is, of course, octahedral. A little bit of additional searching led me towards the square-planar complex $\ce{[Ni(\textbf{L})](ClO4)2}$, where $\mathbf{L}$ is basically a 3,10-bissubstituted 1,3,5,8,10,12-hexaazacyclotetradecane.[1] We can pretend that this perfectly fits the $D_\mathrm{4h}$ point group (it doesn’t) and that the ‘ammine ligands’ are fully identical (at least they’re symmetry-equivalent) and coordinate exactly from the coordinate axes (well, almost).



The first step is to consider the symmetry of the metal orbitals. We have five of them, and we expect them to group up into one $\mathrm{e}$-representation (two degenerate orbitals) and another three non-degenerate orbitals. A table of the irreducible represenations can be found on meta (soon™) or we can use the one from TU Braunschweig. Thankfully, it already includes the representations of the d orbitals, but for comprehensiveness you can also determine it yourself by thinking ‘does the orbital stay the same ($+1$) or invert lobes ($-1$) if I apply that symmetry operation. Here is the answer for the d orbitals:

\begin{array}{cccccccccccc}\hline
\text{Orb.} &amp; E &amp; 2C_4 &amp; C_2 (z)&amp; 2 C_2' &amp; 2 C_2'' &amp; i &amp; 2 S_4 &amp; \sigma_\mathrm{h} &amp; 2 \sigma_\mathrm{v} &amp; 2 \sigma_\mathrm{d} &amp; \text{Rep.}\\ \hline
\mathrm{d}_{z^2} &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; \mathrm{A_{1g}}\\
\mathrm{d}_{x^2-y^2} &amp; 1 &amp; -1 &amp; 1 &amp; 1 &amp; -1 &amp; 1 &amp; -1 &amp; 1 &amp; 1 &amp; -1 &amp; \mathrm{B_{1g}} \\
\mathrm{d}_{xy} &amp; 1 &amp; -1 &amp; 1 &amp; -1 &amp; 1 &amp; 1 &amp; -1 &amp; 1 &amp; -1 &amp; 1 &amp; \mathrm{B_{2g}} \\
\mathrm{d}_{xz}/\mathrm{d}_{yz} &amp; 2 &amp; 0 &amp; -2 &amp; 0 &amp; 0 &amp; 2 &amp; 0 &amp; -2 &amp; 0 &amp; 0 &amp; \mathrm{E_g} \\ 
\mathrm{4s} &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; \mathrm{A_{1g}} \\
\mathrm{4p}_z &amp; 1 &amp; 1 &amp; 1 &amp; -1 &amp; -1 &amp; -1 &amp; -1 &amp; -1 &amp; 1 &amp; 1 &amp; \mathrm{A_{2u}} \\
\mathrm{4p}_x/\mathrm{4p}_y &amp; 2 &amp; 0 &amp; -2 &amp; 0 &amp; 0 &amp; -2 &amp; 0 &amp; 2 &amp; 0 &amp; 0 &amp; \mathrm{E_u} \\ \hline
\end{array}

$\mathrm{d}_{xz}$ and $\mathrm{d}_{yz}$ must be grouped together; only as a group they will be transformed onto each other by $C_4$ or $S_4$. Always if you transform an orbital into empty space with a symmetry operation, find out which orbital was there and include that into a degenerate group.

Thus, the d orbitals transform as $\mathrm{a_{1g} + b_{1g} + b_{2g} + e_g}$. I also included the 4s ($\mathrm{a_{1g}}$) and 4p ($\mathrm{a_{2u} + e_u}$) orbitals for completeness; we’ll need them.



Next, do the same thing for the ligands. Here, we have to make use of group orbitals, i.e. group all the four ammine ligands together. Once we have the represenation of the group orbitals, we will need to reduce that to their corresponding irreducible represenations. Thankfully, we don’t have to consider whether the orbitals are pure or hybrid; we just need to consider &#x3c3;-type orbitals and &#x3c0;-type orbitals. See the table below.

\begin{array}{cccccccccccc}\hline
\text{Orb.} &amp; E &amp; 2C_4 &amp; C_2 (z)&amp; 2 C_2' &amp; 2 C_2'' &amp; i &amp; 2 S_4 &amp; \sigma_\mathrm{h} &amp; 2 \sigma_\mathrm{v} &amp; 2 \sigma_\mathrm{d} &amp; \text{Rep.}\\ \hline
4 \sigma &amp; 4 &amp; 0 &amp; 0 &amp; 2 &amp; 0 &amp; 0 &amp; 0 &amp; 4 &amp; 2 &amp; 0 &amp; ? \\
8 \pi &amp; 8 &amp; 0 &amp; 0 &amp; -4 &amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; ? \\ \hline
\end{array}

Here is where your ‘rule 5’ — or more accurately, the reduction formula — comes into play. It allows us to determine which irreducible representations a reducible one is made up of. (For smaller point groups you can often just derive it logically, but I wouldn’t do that for a group this size.) The general reduction formula is given in equation $(1)$ ($n$ is the number of times a given irreducible representation $\chi$, whose character with respect to a symmetry operation is $\chi_i$, is part of the reducible representation $\xi$. $h$ is the order of the group.) and the results for the &#x3c3; and &#x3c0; type orbitals is given in the table below.

$$n = \frac1h \sum_i \chi_i \xi_i $$

\begin{array}{ccccccccccc}\hline
\text{Group} &amp; \mathrm{A_{1g}} &amp; \mathrm{A_{2g}} &amp; \mathrm{B_{1g}} &amp; \mathrm{B_{2g}} &amp; \mathrm{E_g} &amp; \mathrm{A_{1u}} &amp; \mathrm{A_{2u}} &amp; \mathrm{B_{1u}} &amp; \mathrm{B_{2u}} &amp; \mathrm{E_u} \\ \hline
4\sigma &amp; 1 &amp; 0 &amp; 1 &amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; 0 &amp; 1 \\
8\pi &amp; 0 &amp; 1 &amp; 0 &amp; 1 &amp; 1 &amp; 0 &amp; 1 &amp; 0 &amp; 1 &amp; 1 \\
\hline \end{array}

By doing this, we see that the &#x3c3;-type orbitals transform as $\mathrm{a_{1g} + b_{1g} + e_u}$ while the &#x3c0;-type orbitals transform as $\mathrm{a_{2g} + b_{2g} + e_g + a_{2u} + b_{2u} + e_u}$. (That is all assuming I didn’t miscalculate.)

The next step is to start drawing an orbital diagram. The five d-orbitals should initially be on one level corresponding to the ionisation energy of nickel(II). The virtual 4s and 4p orbitals somewhat higher. The &#x3c0;-type ligand group orbital should be on a level corresponding to the ionisation energy of ammonia, the &#x3c3;-type ones slightly below. I don’t have values here so I’m going to ballpark the relative energies draw a merely schematical diagram for simplicity. Any orbitals that have the same irreducible representation will mix with each other. If there are only two, these will form a bonding/antibonding pair with the antibonding orbital being relatively higher than the bonding one low. If there are more than two, they will form some kind of ladder with intermediate levels preserving the overall orbital count. Scheme 1 shows what I would end up with.


Scheme 1: Approximate orbital scheme for a fully square-planar, ideal $D_\mathrm{4h}$ $\ce{[Ni(\mathbf{L})]^2+}$ $\mathrm{d^8}$ ion.

Note that there are a number of estimates in there. For example, both $\mathrm{b_{2g}}$ and $\mathrm{e_g}$ are made up solely of d-&#x3c0; interactions; I can’t guess a priori which one would have a higher or lesser energy. Likewise for the relative orderings of other orbitals. The only thing clear is that any interactions stemming from the &#x3c3; orbitals should be stronger than those stemming from &#x3c0; orbitals due to &#x3c3; typically displaying better overlap.

To get back to your hypothetical $\mathrm{d^9}$ system, just add the missing electron into the $\mathrm{b_{1g}}$ orbital.



Reference:

[1]: M. P. Suh, B. Y. Shim, T.-S. Yoon, Inorg. Chem. 1994, 33, 5509. DOI: 10.1021/ic00102a026.