Alpha wave frequency variance in EEG of single subject over period of a few hours?
I noticed that a strong background signal in the alpha-wave range was existent at ~10 Hz, but then after a while of doing other things and coming back to it the strong background signal at 10 Hz disappeared but a strong background signal at a smidgen over 8 Hz was present instead.


Alpha-waves (7-13 Hz) are related to a dozen of things. Here is an easy first experiment for you:


Sit still, stare at a wall, eyes open. Remove sources of noise such as TV, radio, etc.
Repeat the same, but now hold your eyes closed.
Record the signals in both conditions for about one minute.
Compare the power spectrums between the two conditions: you will see a big difference in the alpha power, especially at the occipital electrodes.
This is actually a standard test for resting-state analysis.


Back to the actual question: What you noticed is probably about drowsiness. When one gets sleepy, the alpha power starts to increase. You can try this by watching a boring movie.


  Can alpha wave frequencies vary within a single subject over the course of a single day? If they can, by how much, and under what conditions?


Yes, because of the varying cognitive load, thoughts, sleepiness, emotions, etc. The brain signals vary a lot.

Also notice that all sorts of artefacts (e.g. movement, cardiac, eye-blinks, biting of teeth) have a dramatic effect on the measured signal. If you blink your eyes it produces a signal that has 100x higher amplitude compared to the signal that comes from your brain. You cannot really say much what causes any differences in the EEG unless you would do research-level recordings with research-level equipment.