How is quantum function $f(x) = a^x \mod N$ constructed in Shor's algorithm? I can't see how I can apply quantum function $a^x\mod N$ to initial states. Explanations of Shor's algorithm often skips this part, and I do not see what unitary matrix this algorithm would correspond to.

reference: https://en.wikipedia.org/?title=Shor%27s_algorithm
This paper by Pavlidis and Gizopoulos walks through one possible construction of the circuit you're looking for and also provides references to other literature on the subject:

"Fast Quantum Modular Exponentiation Architecture for Shor's Factorization Algorithm". Archimedes Pavlidis, Dimitris Gizopoulos.  http://arxiv.org/abs/1207.0511