Capacitor not charging?
Your circuit should look like this:







simulate this circuit &ndash; Schematic created using CircuitLab

Also, if the value of your capacitor is small, when you disconnect the power source, the led might go out very quickly. Are you sure your capacitor is 104F?

edit: as G36 points out, you probably meant it was marked 104, which means 100nF (except for with aluminium capacitors). To see any afterglow, you'll probably need at least 1000uF