Help me understand dielectrics
There is a big difference between an electric potential and an electric field, so watch out for those terms. The potential difference between two plates depends upon the electric field between them.

I need to pause for a moment here. Note that I didn't say the electric field in each plate of metal. I said the electric field between the plates. The reason is that in a conductor in equilibrium, in which there is a veritable sea of mobile electrons, the net electric field is zero everywhere inside the metal. You might imagine this as a kind of Faraday cage of sorts, I suppose. It is the work required to move a particle from one plate to the other, through the intervening electric field, that counts here.

When an insulating material is inserted into this electric field, the molecules may polarize. This process is as complex as materials themselves are. But when we are talking about increasing capacitance, we are talking about polarized insulators. The easier it is to polarize the insulator, the higher the capacitance will be.

With a polarized insulator, the polarized molecules are little electric dipoles. The net electric field inside the polarized insulating material is the sum of the field due to the capacitor's plates and the field due to the induced dipoles in the insulator. And the insulator dipoles set themselves up to oppose the electric field due to the capacitor's plates. (An argument from a path integral along the outside of the insulator can demonstrate this must be the case.) So the total field is therefore smaller than the field due to the capacitor by itself. (This net field is still in the direction set up by the capacitor, as the opposing dipole field is smaller.)

The dielectric constant \$K\$ is defined this way:

$$\overrightarrow{E}_{insulator}=\frac{\overrightarrow{E}_{applied}}{K}$$

This constant must always be more than 1 (with dipoles setting up opposing fields) because polarization always weakens the applied field. And as I said earlier, the easier it is to polarize the molecules the bigger \$K\$ gets.

So, a polarizing insulator:


decreases the potential difference across the insulator
decreases the electric field inside the insulator


Now it is a small step to see why capacitance increases. Suppose you have a certain charge difference between two plates with air in between them, set up by establishing a voltage across the plates. This same charge, though, placed on a capacitor with a polarizing insulator between the plates would find that the insulator sets up an opposing (smaller, but opposing) electric field because of the aligned dipoles. This would imply a decreased potential difference -- lower voltage -- for that same charge. But:

$$ C = \frac{Q}{V} $$

If there is a smaller potential difference but for the same charge, then the capacitance must be larger. It just follows.

There are other ways to look at this. One of my own favorites is to simply imagine that each dipole acts like a "shortcut" across vacuum space they fill. When you account for all these shortcuts, you wind up with a smaller effective distance between the plates. And this means a higher capacitance. (This analogy also works just as well for magnetic materials, where the aligning domains act like magnetic shortcuts reducing the effective magnetic path length. Same idea, really. But very different process. But it works just as well for relative permeability there.)

Hope that helps a little.