Why did permanent marker come off of glass after rubbing it with my fingers?
Glass is a very smooth surface and therefore things adhere to it less effectively such as the ink in your permanent marker. Maybe your marker is not of a very good quality or maybe you didnt leave it long enough for the ink to stabilize on the surface.

Why dont you conduct an experiment to try your various hypotheses by varying one factor while keeping the rest constant? For exaple:

1) Try different brands of markers on the same glass surface leaving them to dry for the same time and then rubbing them in the same way.
2) Use the same marker but leave it to dry for different amount of time and then rub it in the same way each time.
3) Use the same marker on different surfaces and rub in the same way. 
4) Try with different amounts of dilute saline solution to simulate sweat or cooking oil to simulate skin oil etc.
etc.

Or even better: Try at glass surfaces that have been etched with different sandpaper grits.

Your experiment is reproducible though, i tried as well and small amounts come off easily.