The paper presents a general agent-based system identification framework as potential solution for data-driven models of building systems that can be developed and integrated with improved efficiency, flexibility and scalability, compared to centralized approaches. The proposed method introduces building sub-system agents, which are optimized independently, by solving locally a maximum likelihood estimation problem. Several models are considered for the sub-system agents and a systematic selection approach is established considering the root mean square error, the parameter sensitivity to output trajectory and the parameter correlation. The final model is integrated from selected models for each agent. Two different approaches are developed for the integration; the negotiated-shared parameter model, which is a distributed method, and the free-shared parameter model based on a decentralized method. The results from a case-study for a high performance building indicate that the model prediction accuracy of the new approach is fairly good for implementation in predictive control.