1. A method for generating playback instructions for playback of a recorded computer session, the method comprising the steps of: (a) receiving, by a protocol engine executing on a first device, a recorded stream comprising a plurality of packets representing display data generated by an application program executed on a second device; (b) determining, by the protocol engine, for a packet in the recorded stream, to render the contents of the packet in a human-perceptible manner; and (c) storing, by the protocol engine, the determination in a playback data structure.