1. A method comprising: iteratively upscaling an input image using a scale factor in which each iteration of the upscaling is performed using overlapping patches formed using a patch sampling step size, the patch sampling step size being an amount a patch is moved to form another patch to be sampled, the iteratively upscaling including: upscaling the input image by the scale factor to generate an upscaled input image; smoothing the input image to generate a smoothed input image; and generating an upscaled image iteration comprising, for each patch in the upscaled input image: identifying which of a plurality of patches in the smoothed input image is most similar to the patch in the upscaled input image by: identifying an in-place patch in the smoothed input image at a corresponding location that is proportional to a difference in image size between the upscaled input image and the smoothed input image; computing an image distance of the patch in the upscaled input image to the plurality of patches in the smoothed input image, the plurality of patches included within a neighborhood of the in-place patch; and locating the most similar patch from the plurality of patches based on the computed image distances; computing a prediction for the patch in the upscaled input image using the most similar patch from the smoothed input image or the in-place patch based on comparing a ratio of the computed image distances to a threshold, and a corresponding patch from the input image; and forming a prediction of a patch for the upscaled image iteration using the prediction for the patch in the upscaled image and the patch in the upscaled image; adapting the patch sampling step size such that the patch sampling step size for a first said iteration is different than a patch sampling step size for a second said iteration; and adaptively limiting a number of iterations of said iteratively upscaling by setting a minimum number of iterations and a maximum number of iterations, different from the minimum number of iterations, and enforcing the number of iterations to be within the minimum number of iterations and the maximum number of iterations.