If I charge a electrolytic capacitor outside of a circuit and let it sit
A real-word capacitor is not ideal. In this case the non-infinite resistance parallel to the ideal capacitor does its work: it let's the positve and negative charge cancel each other, generating heat in that resistor.

In practice you won't notice the heat, but the capacitor does heat up a tiny little bit.