How to calculate the movement of the object passing near other object in space?
Since you're interested in the equations of motion, I would solve this problem by using Lagrangian mechanics. Essentially, find the kinetic and potential energies for these two bodies, A and B. 

Construct the Lagrangian:

$$L = T - V$$ 

where T is the kinetic energy, and V is the potential energy. Then use the Euler-Lagrange Equation to achieve the equations of motion (I would add it here, but I'm not sure of the specifics of your problem).

Two-body motion can always be constrained to a plane, so you may have to throw a constraint in there through the use of a Lagrange multiplier.