How to obtain the unknown values $a_i,b_j$ given an unordered list of $a_i-b_j\mod N$?
Update: The description below is for a different problem (in which you have all pairwise distances in a set rather than pairwise distances between two distinct sets). I'll leave it up anyway since it is closely related.

This problem is called the beltway problem, and is a special case of the general $d$-torus embedding problem. It is also closely related to the turnpike problem, in which the distances differences are absolute (not modulo some number). 

It is not known whether the beltway problem admits a poly-time algorithm. There are various pseudo-poly-time algorithms for related questions. The best reference (alas an old one) is the paper by Lemke, Skiena and Smith.