Reference request: Introduction to reinforcement learning with hand calculation examples
Poole and Mackworth's Artificial Intelligence: Foundations of Computational Agents, fully available online, has one such example for Q-learning.

Sutton &amp; Barto's Reinforcement Learning: An Introduction, also online, has a few tiny iterative examples here and there.

Isbell &amp; Littman's udacity course on RL has examples, quizzes, and programming exercises in Java.