Pentaprism scanning system has been widely used in the measurement of large flat and wavefront, based on its property that the deviated beam will have no motion in the pitch direction. But the manufacturing and position errors of pentaprisms will bring error to the measurement and so a good error analysis method is indispensable. In this paper, we propose a new method of building mathematic models of pentaprism and through which the size and angle errors of a pentaprism can be put into the model as parameters. 4 size parameters are selected to determine the size and 11 angle parameters are selected to determine the angles of a pentaprism. Yaw, Roll and Pitch are used to describe the position error of a pentaprism and an autocollimator. A pentaprism scanning system of wavefront test is simulated by ray tracing using matlab. We design a method of separating the constant from the measurement results which will improve the measurement accuracy and analyze the system error by Monte Carlo method. This method is simple, rapid, accurate and convenient for computer programming.