1. An apparatus of controlling a robot, the apparatus comprising: an image obtaining unit configured to obtain a three-dimensional image of a user; a driving unit configured to drive an arm of the robot that is composed of a plurality of segments; and a control unit configured to generate a user model that corresponds to a motion of the joint of the user based on the three-dimensional image, to generate a target model having the plurality of segments of which the length varies based on the user model, and to allow the arm of the robot to be driven based on the target model.