How does forum determine new replies for each user?
Drupal has a history table to record when any user viewed any node.

So it knows when you last viewed the node, and which comments have newer timestamps. So it marks those comments with newer timestamps as new or updated.

UPDATE: There is also a constant NODE_NEW_LIMIT set in the node.module file which defaults to 30 days ago. That means nodes changed before 30 days are always marked as read.