Do you always need to keep track of what cards have been drawn in a turn?
If both players can't verify which cards were drawn that turn any unverified cards cannot be returned to the top of the library with Sylvan Library. If you think you might be playing a Library with Flash be sure to keep track of which cards you have drawn that turn.

From the Magic Judge Rules Blog:


  Here’s the Official ruling that has been handed down for Competitive REL events where Sylvan Library is legal: “Players will not have an option to return a card that can’t be proven to have been drawn this turn.”