Season 5 of Outlander is based on The Fiery Cross, the fifth book in Diana Gabaldon's Outlander series (book spoilers ahead!). Though Roger and Brianna do eventually go back through the stones, it's not until after the birth of their second child, Amanda, in the sixth book, A Breath of Snow and Ashes.