ECDSA vs RSA: Performance on Android platform and surprising results
BouncyCastle has a really bad ECC implementation. It uses affine coordinates which incur a huge performance hit (factor 20 or so) since it computes a field inversion after every single step. Good implementations use Jacobi coordinates (or a similar approach) where denominators are kept and there is only one field inversion at the end.

It's also potentially vulnerable to side channel attacks since it branches on secret data (only relevant to signing, not verification).

As another ECC benchmark you could look at the the ref10 branch of str4d/ed25519-java. This uses a different elliptic curve, Ed25519, but the main difference is that the implementation has been optimized more than the one you tested. But this is still new and experimental code, so be careful.