Is this language regular or not?
Take $k=1$. No need to go further, any more $a$ can be absorbed by $u$ without risk.

So you need $u$ to have at least 1 $a$ and only $a$'s or $b$'s. Therefore, $L_1$ is described by $a(a|b)^*a(a|b)^*$ and is regular.

For the side question (at most $k$ $a$'s, let's call that language $L_2$), you can reason in a similar fashion to get $u$ as small as possible. For simplicity let's consider $L_3$ the intersection of $L_2$ with the one denoted by $a^*ba^*$. You have $L_3=\{a^k b a^j, k \geq j\}$. Intersection with its reverse yields $\{a^kba^k\}$ which is not regular, therefore $L_2$ is not regular either since regular languages are closed under intersection and reversal.