Understanding the "Waterfall Illusion"
The motion after-effect (MAE) is believed to be primarily due to adaptation of direction sensitive cells in the middle temporal area (MT) (Fig. 1.). The directional cells in this area of the cortex are selectively sensitive to motion in one direction. Hence, looking at a waterfall will selectively cause adaptation in those cells tuned to the direction of the waterfall. When one looks away, the MT cells responsive to opposite directions will become dominant, as their counterparts are momentarily shut down and one will see the waterfall running in opposite direction.

Adaptation thus apparently weakens the opponent input to the adapted MT cells, enabling an enhanced response to motion balanced stimuli. However, the possibility of adaptation occurring in MT neurons themselves cannot be ruled out (Mather et al., 2008).

The MT area is one of the principal visual cortical areas devoted to motion sensing and hence a likely culprit for MAE. However, fMRI studies have indicated that a lot more visual areas in the brain may be involved, either directly, or indirectly, including V1, V2, V3, VP, V3A, V4, and also MT (Mather et al., 2008).   


Fig. 1. Two visual streams dominate vision: the dorsal 'where" pathway (containing the direction-sensitive MT area), and the ventral 'what' pathway. source: InTech Open

Reference
-  Mather et al., Trends Cog Sci (2008); 12(12): 481-7