Which image transformations are considered continuous transformations?
I think "continuous" is meant in the strict mathematical sense here, i.e. no abrupt changes in the coordinate mapping.

Most non-rigid image alignment algorithms (try to) produce continuous mappings. Distortion filters in popular photo-editing applications (like Photoshop's Warp effect) are usually continuous mappings, too.

ADD: Here are two examples of transformations that are continuous but not affine or perspective:

How to flatten image label on a food jar

Flatten a sudoku grid