Unable to call FaceTime only to a particular caller ID
FaceTime requires a very strong network connection, so I believe your issue has less to do with the device, and more to do with the type of connections you trying to use.

FaceTime works best with a solid wifi connection on both ends, or at least LTE. If either device is trying to use 3G or 4G, you will struggle to establish or maintain a decent connection.

To test this, I would ensure you are both on a good wifi connection and try it again to see how it works. If not, then you have deeper compatibility issues.

P.S. I would recommend upgrading your iPhone 4 to iOS 7.1. It has improved performance over 7.0.x and will help alleviate further compatibility issues.