Decidability of a problem concerning polynomials
I'm not knowledgeable in that field either, but I think I can provide a non-constructive answer.
The first-order theory of real closed fields is decidable. Your problem can be stated as a system of algebraic equations and inequations over the real algebraic numbers. Consider $2 (\deg P + \deg Q)$ variables $x_1,\ldots,x_{\deg P}, y_1,\ldots,y_{\deg P}, x'_1,\ldots,x'_{\deg P}, y'_1,\ldots,y'_{\deg P}$. You want to know whether the following system is satisfiable:
$$\begin{align*}
  P(x_j+i\,y_j) &amp;= 0 &amp; \text{for \(1 \le j \le \deg P\)} \\
  Q(x'_k+i\,y'_k) &amp;= 0 &amp; \text{for \(1 \le k \le \deg Q\)} \\
  x_j^2 + y_j^k &amp;\le x_1^2 + x_2^2 &amp; \text{for \(2 \le j \le \deg P\)} \\
  {x'}_j^2 + {y'}_j^k &amp;\le {x'}_1^2 + {x'}_2^2 &amp; \text{for \(2 \le k \le \deg Q\)} \\
  x_1^2 + y_1^2 = {x'}_1^2 + {y'}_1^2 \\
\end{align*}$$
The first two equation families express that the $x_j+i\,y_j$ and $x'_k+i\,y'_k$ are roots of the polynomials, the next two inequation families express that $x_1+i\,y_1$ and $x'_1+i\,y'_1$ have the largest absolute value, and the last inequation compares these largest absolute values.
It is decidable whether this system is satisfiable: your problem is decidable. However, this statement is probably not the most efficient way to go about it.
A more useful answer probably involves the theory of Gröbner bases. If you're trying to solve that problem for yourself, I think reading the first few chapters of any computational algebra book will give you the requisite background. If you're just aiming to solve your underlying problem, there is probably an off-the-shelf algorithm you can implement.