What does 'contextual' mean in 'contextual bandits'?
A contextual bandit algorithm not only adapts to the user-click feedback as the algorithm progresses, it also utilizes pre-existing information about the user's (and similar users) browsing patterns to select which content to display.

So, rather than starting with no prediction (cold start) with what the user will click (traditional bandit and also traditional A/B testing), it takes other data into account (warm start) to help predict which content to display during the bandit test.

See: http://www.research.rutgers.edu/~lihong/pub/Li10Contextual.pdf