Image rescaling algorithm Suppose I have to use bilinear interpolation to rescale image. I more or less understand how enlargement works in case of integer scale factor. 

But what to do with non-integer scale and especially when you need to downsample an image?
Explanation
With a non-linear scale, you apply different weights for each pixel (or whatever unit you are using). You can use Euclidean distance towards the closest pixels in both directions to determine the weights.
Example
Normalization
For example, in the image below, the red image is the downsampled image (3x3) and the black image is the original image (4x4). The dots represent the sample points (center of the pixels).
You can align the two images by treating each pixel coordinate as a set of two values (x and y) between 0 and 1 (inclusive). Pixels should not be treated as square, so you should treat each pixel as a point rather than a square (when aligning the two images).

In the above image, the four corners would all carry the weight of the original image's four corners, for example. The points in the middle would carry different weights depending on the difference between the four neighboring points.
With numbers:
This is a hypothetical area to sample in larger images (not related to the image above). The distances are between original pixels are normalized from 0 to 1 to simplify the math.
The black dots represent the sample points (pixels) of the original image. The larger red dots represent the sample points of the downsampled image. The red dot in the top left is closer to the black dot in the top left than the other three surrounding dots, so that pixel will have a heavier weight for determining the color of the downsampled pixel.

Horizontal distance from:

Right 2 original (green) pixels = 0.1 pixels
Left 2 original (green) pixels = 0.9 pixels

Vertical distance from:

Top 2 original (green) pixels = 0.1 pixels
Bottom 2 original (green) pixels = 0.9 pixels

Weight for final pixel:
$$weight(0, 0) = (1.0 - 0.1)(1.0 - 0.1) = 0.81$$
$$weight(0, 1) = (1.0 - 0.1)(1.0 - 0.9) = 0.09$$
$$weight(1, 0) = (1.0 - 0.9)(1.0 - 0.1) = 0.09$$
$$weight(1, 1) = (1.0 - 0.9)(1.0 - 0.9) = 0.01$$
Subtract from 1 both the horizontal and vertical distance (because closer pixels should carry more weight). The scale factor in this example is 0.75 in each direction.
Scaling Up vs. Scaling Down
Scaling up in the same  process as scaling down. By normalizing both the original image and the new image to between (0, 0) and (1, 1), you can do either with the same process. As an exercise, in the first figure, if the red points were from the original image and the black points were from the enlarged image, you should be able to see that the process is the same.