1. A method of deciding an optimum orthogonal transform mode before quantization in a motion picture coding system in which coding of interlaced TV signals proceeds in a frame structure and an orthogonal transform is made in the unit of a macroblock, comprising the steps of: dividing said macroblock into a plurality of frame mode subblocks; dividing said macroblock into a plurality of field mode subblocks; calculating variances of the respective frame mode subblocks; calculating variances of the respective field mode subblocks; calculating a frame mode variance of said macroblock by calculating a sum of values of a monotonously increasing and substantially saturating nonlinear function using said calculated variances of frame mode subblocks as a variable; calculating a field mode variance of said macroblock by calculating a sum of values of a monotonously increasing and substantially saturating nonlinear function using said calculated variances of field mode subblocks as a variable; and comparing said frame mode variance of said macroblock with said field mode variance of said macroblock to select one of the modes which is smaller in sum as the orthogonal transform mode.