Serotonin levels (in the form of 5-HT) have been shown to decrease during REM and NREM sleep, leading some researchers to believe that it is also involved in memory consolidation during sleep.
Another theory suggests that the stress brought on by sleep deprivation affects memory consolidation by changing the concentration of corticosteroids in the body.