Defect management (DM) for quality inspection (QI) is a major strategy employed by general contractors to enhance construction management of building projects. However, there are significant issues in construction DM in standard practice that affects quality inspection, including protracted procedures, data entry redundancies, confusion, and inefficient information management. Recognition of these construction DM issues, this paper proposes a new and practical approach that applies Building Information Modeling (BIM) technology to quality inspection and defect management. Specifically, BIM digitally contains precise geometry and relevant data needed to support building structures to describe 3D object-oriented CAD. Using BIM technology, this study proposes a BIM-based Defect Management (BIMDM) system by on-site quality managers during the construction phase. The intended approach integrates web and BIM technologies in the BIMDM system to illustrate and analyze defect information at the jobsite in real time. The anticipated result is the effectively managed status and results of the corrective works performed. The BIMDM system is applied in a selected case study of a building project in Taiwan to verify the proposed approach and demonstrate the effectiveness of defect management practice. Utilizing the BIMDM system, on-site quality managers are better able to track and manage defects with BIM models through accurate records and photos. The combined results of the study demonstrate that a BIMDM-like system can be an effective visual defect management platform when integrating BIM and web technologies. The advantage of the BIMDM system lies not only in improving defect management efficiency for on-site quality engineers and managers, but also in facilitating easy quality inspection while identifying and communicating in the 3D BIM environment. As such, authors expect that effective use of the proposed BIMDM would significantly assist on-site quality engineers and managers to systematically handle defect management work using BIM technologies in future construction projects.