Calculate forces between two flat coils
It is not so easy to calculate forces with mathematical expressions based on this setup. You would have to integrate the B-field in the coil. You would need to calculate the energy gradient in a 3D setup to get the forces between the coils. 

But it is very easy to simulate this setup with a finite element field solver tool. I can recommend FEMM for this. 

Here is a tutorial that got me up and running. I must admit that the drawing editor is not easy to get used to. One important thing to know is that you can delete objects with with the right mouse button.to.