Affine transformation on a pixel grid
What you need is an interpolation method. 

The method you described is called nearest-neighbor, because you pick the pixel that is nearest to the place you actually wanted.

Other methods include:


bi-linear interpolation (Select 4 nearest points, interpolate by x and y according to distance) (see here)
barycentric interpolation (Select 3 nearest points, interpolate in the system of coordinates of the triangle)
bi-cubic interpolation (see here)