Proving $\{ a^n b^m \mid n \leq m^2 \}$ is not context-free using pumping lemma
When $vwx = a^j b^k$, there are two possibilities:

$v$ contains both $a$s and $b$s, or $x$ contains both $a$s and $b$s. In this case $uv^2wx^2y \notin a^*b^*$, and in particular $uv^2wx^2y \notin L$.
$v = a^s$ and $x = b^t$, where $s,t&gt;0$ and $s+t \leq p$ (since $|vwx| \leq p$). In this case, $uv^0wx^0y = a^{p^2-s}b^{p-t}$, and so $p^2 - s \leq (p-t)^2 \leq (p-1)^2$, which translates to $s \geq 2p-1$, which is impossible, since $s \leq p-1$.