Diffie-Hellman protocol: Clarifications
Your assumption is correct, that is called the Discrete Logarithm Problem.

The complexity of the problem is linear to the number of elements of the group but exponential to the number of bits of the group's size (These days you usually use a 2048 bit prime for a DH exchange, meaning that there are ~ 2^2048 elements in the group)