Researchers of different fields have been involved in human behavior analysis during the last years. The successful recognition of human activities from video analysis is still a challenging problem. Within this context, applications targeting elderly care are of considerable interest both for public and industrial bodies, especially considering the aging society we are living in. Ambient intelligence (AmI) technologies, intended as the possibility of automatically detecting and reacting to the status of the environment and of the persons, is probably the major enabling factor. AmI technologies require suitable networks of sensors and actuators, as well as adequate processing and communication technologies. In this paper we propose an innovative solution based on a real time analysis of video with application in the field of elderly care. The system performs anomaly detection and proposes the automatic reconfiguration of the camera network for better monitoring of the ongoing event. The developed framework is tested on a publicly available dataset and has also been deployed and evaluated in a real environment.