Are GPUs inefficient for calculations on big numbers? Is RSA and EC cypto generally done on CPU only? Looking specifically at RSA and EC algorithms which imply doing operations on integers &gt;= 256 bits (&gt;&gt; 64 bits), I have noticed (from my limited experience) that 99% of the software for crunching big numbers is designed for CPU. For example, there appears to be no actively maintained CUDA package for big-number operations (arbitrary length or even 256-512 bit numbers for EC calculations).
Is there a reason for big number calculations (and thus RSA and EC calculations) to remain on CPU? Is it sub-optimal to run such computations on GPU (I understand that GPUs are not optimized for big number calculations because there is no such requirement in graphics, but can't it be efficiently done anyway)? Will this change in the future or will RSA &amp; EC remain on CPU?
If you run RSA speed tests in OpenSSL you'll see that many operations are pretty fast, with the possible exception of RSA key pair generation. However, for that reason key pair generation is generally not used very often - RSA is generally not used to provide forward security in transport protocols, for instance.
CPU's have been upgraded as well, and they can perform up to 512 bit operations on large numbers nowadays - see e.g. Intel's AVX-512. Although those still come with some drawbacks I would expect that those disappear as the technology matures.
Using GPU instructions has the following drawbacks:

for top performance you generally need vendor specific code;
not all (security minded) servers will have fast GPU support;
the CPU instructions used for the asymmetric primitives are generally not the bottleneck - bulk encryption would take much more CPU power.

Furthermore, latency introduced by the PCIe bus makes the GPU most applicable for parallel operations, and the amount of parallelism for e.g. performing private key operations is usually limited.
Of course, I would imagine that e.g. a double ratchet that uses a lot of (EC)DH operations could still benefit from CUDA. But it would be a lot of work to create, to secure and to maintain.
Note: that said, I didn't perform or lookup any benchmarks for GPU's - I do run OpenSSL speed tests now and then - if just to test the new hardware that I buy.