Transforming data points to match other data points
For this task, you need to decide 3 things:
(1) What are the allowed transformations.
For example do you allow an arbirtrary affine transformation (rotation, scaling, translation). Then you need a matrix $f \in \mathbb{R}^{3 \times 3}$ (using homogeneous coordinates) that maps the points from the left image to the right one. Each entry of the matrix is a variable you want to determine. You could also restrict to specific affine maps (e.g. only rotations) by setting certain entries in $\mathbf{A}$ fixed.
In principle you can use  any  map $f_{w}: \mathbb{R}^{2 \times 2} \rightarrow \mathbb{R}^{2 \times 2}$, that depends on some variables $w$.
(2) You need to think about a measure (or &quot;loss&quot;), which evaluates how similar the two sets are. For arbitrary sets, there is the  Hausdorff distance, which ignores any temporal information. If you consider your data as two curves over time, there is the Frechet distance.
You can have a look here to see that its a difficult problem to  using the Haussdorf distance in general. In fact,  computing only the Hausdorff distance, without any optimization is already  not trivial.
However, for polygonal curves, you can compute Hausdorff distance, Frechet distance, and discrete Frechet distance efficiently.
Finally, in your specific case, you have for each point of $A$ a corresponding point in $B$, you can measure the distance for each correspondence (this gives you simply the mean squared errors in $\mathbb{R}^{2}$).
(3) Depending on the choice of (1) and (2), you need to employ a corresponding optimizer.
If you want to minimize the pairwise distance, but you do not have the correspondences available, there is the Iterative closest point.
If you simple compute the distance between corresponding points, you have a non-linear optimization problem. You can use solvers such as Levenberg–Marquardt.
As @bogovicj points out, the topic is called point-set registration.