Regression For Elliptical Curve Public Key Generation Possible?
I don't know of any neural network that can do cryptography well, so you would have to do a little experimenting yourself. The main thing that sticks out to me is that doing operations in the elliptical curve requires the modulus operator since it works in finite field, and I think neural networks have a hard time learning the modulus operator in general. So I would focus on that first. Some things to help the network learn the modulus operator:

I would try to increase the hidden layers to a number 4x-10x bigger than the input dimension, which maps it to a higher dimension to hopefully learn more complex behavior.
I would use less layers, maybe only 2-4 hidden layers, to speed up the development time.
Most importantly, I would train with a LOT of data points (25% of the total possible finite field possibilities). I don't think the neural network can learn unless the number of data is this high.
For reference, someone got a network to learn the modulus operator using these points.

For rapid iterating, I would test with a much smaller finite field. E.g. use a 8 bit security and see if the neural network can do well with that first before moving on to the full 256-bit security key (or whatever your end goal is).
Taking this a step further, I would first test to see if the neural network can even perform a point addition in the elliptical curve well, because if it doesn't then it definitely can't do point multiplication which is needed to compute the public key.