How can I realize a complex communication system in a real time application?
The output of qammod in your program is a sequence of amplitudes. If $a_k$ is the real part of the sequence, and $b_k$ the imaginary part, then the QAM baseband signal is: 
\begin{align*}
s_{BB}(t)&amp;=\sum_k a_k p(t-kT_p) + j\sum b_k p(t-kT_p) \\
&amp;= A(t) + jB(t).
\end{align*} In this equation, $T_p$ is the pulse (baud) rate, and $p(t)$ is the pulse shape.

In order to convert this to a real, physical passband signal with carrier frequency $f_c$, you upconvert the baseband signal to frequency $f_c$: $$s_{PB}(t)=\text{Re}(s_{BB}(t)e^{j2\pi f_ct}).$$ While the previous equation is mathematically nice, in practice this is implemented as: $$s_{PB}(t)=A(t)\cos(2\pi f_ct)+B(t)\sin(2\pi f_ct)$$ with possible scale factors to keep the signal energy constant.

Once you have $s_{PB}(t)$, all that remains is amplify it, possibly bandpass filter it to eliminate out-of-band emissions, and connect an antenna.

Note that Matlab works in the discrete-time domain and the signals I've described are continuous time. At some point in your system, you will need to do a digital to analog conversion. Exactly where that will happen depends on many factors. I recommend that you take a look at some common RF architectures, such as heterodyne, homodyne, zero-IF and so on. They can give you ideas on how to implement the analog part of your system. One good reference is "RF System Design of Transceivers for Wireless Communications" by Q. Gu. Another very good book is Steven Tretter's "Communication System Design Using DSP Algorithms: With Laboratory Experiments".