1. A method for building a model to control a first device, comprising: receiving, as input: demonstration data corresponding to human operation of a second device used to perform a demonstration, wherein the second device is a camera and wherein the demonstration data includes a first video captured by the camera under control of the human operator; and environmental sensory data associated with the demonstration data; determining device settings of the second device from the demonstration data, wherein the device settings are pan-tilt-zoom settings of the camera associated with the first video and wherein determining the settings includes: estimating a calibration matrix of each frame of the first video using a pinhole model; and applying optimization to estimate parameters of a modified pinhole model and to estimate per-frame pan-tilt-zoom settings; extracting, from the environmental sensory data, feature vectors describing at least locations of objects in the environment; training, based on the determined device settings and the extracted feature vectors, a regressor which takes additional feature vectors as inputs, and outputs planned device settings for operating the first device; and instructing the first device to attain the planned device settings output by the trained regressor.