CO2-based transcritical Rankine cycle (CTRC) is a promising technology for the waste heat recovery of an engine considering its safety and environment friendly characteristics, which also matchs the high temperature of the exhaust gas and satisfies the miniaturization demand of recovery systems. But the traditional CTRC system with a basic configuration (B-CTRC) has a poor thermodynamic performance. This paper introduces an improved CTRC system containing both a preheater and regenerator (PR-CTRC), for recovering waste heat in exhaust gas and engine coolant of an engine, and compares its performance with that of the B-CTRC system and also with that of the traditional excellent Organic Rankine Cycle (ORC) systems using R123 as a working fluid. The utilization rate of waste heat, total cooling load, net power output, thermal efficiency, exergy loss, exergy efficiency and component size have been investigated. Results show that, the net power output of the PR-CTRC could reach up to 9.0 kW for a 43.8 kW engine, which increases by 150% compared with that of the B-CTRC (3.6 kW). The PR-CTRC also improves the thermal efficiency and exergy efficiency of the B-CTRC, with increases of 184% and 227%, respectively. Compared with the ORC system, the PR-CTRC shows the significant advantage of highly recycling the exhaust gas and engine coolant simultaneously due to the special property of supercritical CO2's specific heat capacity. The supercritical property of CO2 also generates a better heat transfer and flowing performances. Meanwhile, the PR-CTRC possesses a smaller SP (0.010-0.020 m) than that of R123 systems (0.055-0.070 m). Therefore, the PR-CTRC system is suitable for the waste heat recovery of an engine, especially for recovering both high-grade and low-grade waste heat. (C) 2016 Elsevier Ltd. All rights reserved.