1. A spatial domain video enhancement/up-scaling system comprising: a first stage including a K×K transform circuit configured to receive temporal domain input video data and transform the temporal domain input video data to spatial domain video data having spatial domain coefficients; a gain adjust circuit configured to vary elements of a K×K enhancement matrix; a multiplier configured to multiply each spatial domain coefficient by corresponding elements of the K×K enhancement matrix to obtain enhanced spatial domain coefficients and deposit them in a K×K corner of a zero padded 2K×2K inverse transform matrix; and a 2K×2K inverse transform circuit configured to scale the enhanced spatial domain coefficients and convert them back to the temporal domain.