Is the following construction of a secure PRG is also a secure PRG?
Indeed, your intuition is perfectly correct and the distinguisher $D$ you're trying to build is basically $D'$ itself.
To see that this is the case, simply divide the experiment for $G'$ into two cases: either the chosen input is palindrome or it is not.
The first case occurs with negligible probability, so we can focus in the second case, but now it's easy since this case corresponds to the original experiment for $G$.

If you want to be formal, just keep in mind that
\begin{multline*}
\mathbb{P}\left[D'(G'(x)) = 1\right] = \mathbb{P}\left[D'(G'(x)) = 1|x\text{ is palindrome}\right]\cdot\overbrace{\mathbb{P}\left[x\text{ is palindrome}\right]}^{\delta}
\\ + \mathbb{P}\left[D'(G'(x)) = 1|x\text{ is not palindrome}\right]\cdot\underbrace{\mathbb{P}\left[x\text{ is not palindrome}\right]}_{1-\delta}
\end{multline*}
and 
$$\mathbb{P}\left[D'(r) = 1\right] = 
\delta\cdot\mathbb{P}\left[D'(r) = 1\right] + (1-\delta)\cdot\mathbb{P}\left[D'(r) = 1\right].$$
Now, when you take the difference
$$\begin{align*}
\left|\mathbb{P}\left[D'(G'(x)) = 1\right] - \mathbb{P}\left[D'(r) = 1\right]\right|
\end{align*}$$
try to associate terms appropriately and apply some straightforward inequalities to show this is negligible.