All your base palindromic belong to us
Pyth, 13 bytes
mlf_ITjLdSdSQ

The brevity of this is mostly due to the Invaluable &quot;Invariant&quot; command.
msf_ITjLdSdSQ       implicit: Q=input
m         d         map lambda d over
           SQ       Inclusive range 1 to Q
      jLdSd         Convert d to all the bases between 1 and d
  f                  filter lambda T:
   _IT                 is invariant under reverse
 l                  number that are invariant under reverse

If True is an acceptable output for 1, msm_IjdkSdSQ (12 bytes) works.
Try it here.