How to calculate complexity of an algorithm which calls a subroutine?
This sentence: "In a single run of algorithm $A$, algorithm $B$ is called at most $10n^2$ times" is vague. If it means in each iteration of $A$, then as each time the calling $B$ takes $10n^2\times O(n^3) = O(n^5)$, we can say $A$ will be run in $n^4 \times O(n^5) = O(n^9)$.

On the other hand, if it means in all round of $A$, it calls $10n^2$ times $B$, as you said the time complexity would be $O(n^5)$.