How small are we talking about when defining the small public/private key exponent
Exponentiation by squaring takes time approximately proportional to the number of bits in the exponent. 3 has been used but allows attacks on some weak padding schemes. 65537 ($=2^{16}+1$) is the one used widely in practice.

Cf. Impacts of not using RSA exponent of 65537