1. A client device for reconstructing a viewport from a rendered rendering surface of a predicted user viewpoint, comprising: a user input device that receives a stream of user input; a display device; a processor; and a storage device holding instructions for a client application program, executable by the processor to: receive, from a server device over a network, a rendered rendering surface of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user of the client device prior to the point in time, wherein the rendered rendering surface has a wider field of view than a viewport of the client application program and includes a plurality of non-displayed rendered pixel data items in regions not displayed in the viewport of the client application program centered at the predicted user viewpoint; receive, from the user input device, a subsequent user navigation input near the point in time in the stream of user input; determine an actual user viewpoint based on the subsequent user navigation input; determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint; reconstruct a viewport for the actual user viewpoint from the rendered rendering surface, wherein the viewport for the actual user viewpoint includes a portion of the plurality of non-displayed rendered pixel data items; and display the viewport on the display device.