What set of primitive operations are assumed to be constant time in complexity analyses?
It depends on the "model of computation".  The model of computation specifies things like what can be done in a single step and what operations are available.  There are different models of computation, each of which can sometimes lead to a slightly different running time.

If the model of computation makes an important difference and the paper is being careful, it will usually mention which model of computation it is working in.  Sometimes we don't care about the difference (or the difference is negligible) and then the model of computation might not be mentioned.

So, your model of computation could specify that pointers and integers (large enough to hold numbers in the range $[0,N]$) fit in a single word and can be accessed in $O(1)$ time.  Or, it might specify that this takes $O(\log N)$ time and space.  A circuit-centric model of computation might count the number of gates in a circuit needed to do the computation.  There are many others.

One example of a model of computation is the RAM model.