Is there a limit on transition function compositions in NFAs?
You can define the transition function of an automaton to be whatever you want, as long as it's a function of the appropriate type, i.e., $Q\times\Sigma\to Q$ fora DFA and $Q\times\Sigma\to 2^Q$ for an NFA.

However, the composition you've attempted doesn't actually make sense.  Since $f$&nbsp;is the transition function of an NFA, it has type $Q\times\Sigma\to 2^Q$. As such, you can't write $f(f(q_i,\sigma))$, since the argument of the outer application of&nbsp;$f$ is some subset of&nbsp;$Q$, whereas there are supposed to be two arguments: an element of&nbsp;$Q$ and a symbol from the alphabet.  So, for a double-composition, you probably need to write something like
$$\bigcup\{f(q,\sigma)\mid q\in f(q_i,\sigma)\}\,.$$
You can try to work out the triple-composition yourself.

But are you sure you want to do that? You're defining an automaton that says "Every time I read the character&nbsp;$\sigma$, I will do whatever the original automaton would have done if it read three&nbsp;$\sigma$s." As I said, you can make your automaton do whatever you want, but I'd be interested to know the context that makes this a sensible thing to do.