Policy Gradients - gradient Log probabilities favor less likely actions?
Yes, you are right that the term $\frac{1}{\pi_\theta(a_t|s_t)}$ itself is favoring the less likely actions. But don't forget how $\nabla \pi_\theta(a_t|s_t)$ is calculated, and you will see that $\nabla \pi_\theta(a_t|s_t)$ is over-emphasizing on the first action and $\frac{1}{\pi_\theta(a_t|s_t)}$ is correcting it.

Recall that the derivative for y =softmax(x) is $\frac{\partial y_i}{\partial x_j} = y_i(\delta_{ij}-y_i)$. Notice that $y_i$ is multiplied in the front, i.e. over-emphasizing the more likely $y_i$. By multiplying $\frac{1}{y_i}$ in the front, the over-emphasizing effect is being corrected.

In your example, when ${\pi_\theta(a_t|s_t)}=[0.5,  0.1,  0.1,  0.1,  0.1,  0.1]$, we can write down that 
$$\nabla \pi_\theta(a_t|s_t)=\left(\begin{array}{cccccc} 
0.5\times0.5 &amp; -0.5\times0.1 &amp; -0.5\times0.1 &amp; -0.5\times0.1 &amp; -0.5\times0.1 &amp;  -0.5\times0.1\\
 -0.5\times0.1 &amp; 0.1\times0.9 &amp;  -0.1\times0.1 &amp; -0.1\times0.1 &amp; -0.1\times0.1 &amp; -0.1\times0.1 \\
 -0.5\times0.1 &amp; -0.1\times0.1 &amp; 0.1\times0.9  &amp; -0.1\times0.1 &amp; -0.1\times0.1 &amp; -0.1\times0.1 \\ 
-0.5\times0.1 &amp;  -0.1\times0.1 &amp; -0.1\times0.1 &amp; 0.1\times0.9 &amp;-0.1\times0.1 &amp; -0.1\times0.1 \\
 -0.5\times0.1  &amp;  -0.1\times0.1 &amp; -0.1\times0.1 &amp; -0.1\times0.1 &amp; 0.1\times0.9 &amp; -0.1\times0.1 \\
 -0.5\times0.1  &amp;  -0.1\times0.1 &amp; -0.1\times0.1 &amp; -0.1\times0.1 &amp; -0.1\times0.1 &amp; 0.1\times0.9\\
\end{array}\right)$$
And by multiplying it with $\frac{1}{\pi_\theta(a_t|s_t)}=[\frac{1}{0.5},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1}]$, you effectively get
$$\nabla \log\pi_\theta(a_t|s_t)=\frac{1}{\pi_\theta(a_t|s_t)}\nabla \pi_\theta(a_t|s_t)=
\left(\begin{array}{cccccc} 
0.5 &amp; -0.5, &amp;-0.5, &amp;-0.5, &amp;-0.5, &amp;-0.5, \\
-0.1 &amp; 0.9 &amp; -0.1 &amp; -0.1 &amp; -0.1 &amp; -0.1 &amp;  \\
-0.1 &amp; -0.1 &amp; 0.9 &amp; -0.1 &amp; -0.1 &amp; -0.1 &amp;  \\
-0.1 &amp; -0.1 &amp; -0.1 &amp; 0.9 &amp; -0.1 &amp; -0.1 &amp;  \\
-0.1 &amp; -0.1 &amp; -0.1 &amp; -0.1 &amp; 0.9 &amp; -0.1 &amp;  \\
-0.1 &amp; -0.1 &amp; -0.1 &amp; -0.1 &amp; -0.1 &amp; 0.9 &amp;  \\
\end{array}\right)$$

In case you are still thinking that because $0.9&gt;0.5$ in the above result, we are slightly favoring the less likely action (although not 5 times more favorable), I would say that's the desired behavior. Because when we already have a relatively high score for action 1 (pre-activation 0.5) and now increase the score a little bit, we don't want the output probability to increase that much because otherwise the probability would more easily explode out of the range of $[0,1]$.

This kind of saturating gradient effect is very common for anything using the log derivative trick, not only in RL. You can also think of the example in the original REINFORCE algorithm
paper (on page 6) where a Bernoulli random variable
$$g_i(y_i,p_i)=\left\{
\begin{array}{c l} 
     1-p_i &amp; \text{if } y_i=0\\
     p_i &amp; \text{if } y_i=1
\end{array}\right.$$ 

has a log derivative as 

$$\frac{\partial \log g_i}{\partial p_i}=\left\{
\begin{array}{c l} 
     -\frac{1}{1-p_i} &amp; \text{if } y_i=0\\
     \frac{1}{p_i} &amp; \text{if } y_i=1
\end{array}\right.$$ 
Here when $p_i$ is already large and $y_i=1$, the derivative would be relatively small.

Finally, regarding your question what if we only got such a large large reward because we took the first action? Don't worry, here you are estimating the expected reward (and its gradient) of your current policy $\pi$, not exploiting this policy, so you just want to have a fair (unbiased) estimate, not a best possible reward.