Collision detection, which refers to the computational problem of finding the relative placement or configuration of two or more objects, is an essential component of many applications in computer graphics and robotics. In image-guided robotic surgery, real-time collision detection is critical for preserving healthy anatomical structures during the surgical procedure. However, the computational complexity of the problem usually results in algorithms that operate at low speed. In this paper, we present a fast and accurate algorithm for collision detection between Oriented-Bounding-Boxes (OBBs) that is suitable for real-time implementation. Our proposed Sweep and Prune algorithm can perform a preliminary filtering to reduce the number of objects that need to be tested by the classical Separating Axis Test algorithm, while the OBB pairs of interest are preserved. These OBB pairs are re-checked by the Separating Axis Test algorithm to obtain accurate overlapping status between them. To accelerate the execution, our Sweep and Prune algorithm is tailor-made for the proposed method. Meanwhile, a high-performance scalable hardware architecture is proposed by analyzing the intrinsic parallelism of our algorithm and is implemented on FPGA platform. Results show that our hardware design on the FPGA platform can achieve around 8x higher running speed than the software design on a CPU platform. As a result, the proposed algorithm can achieve a collision frame rate of 1 kHz and fulfill the requirement for the medical surgery scenario of Robot-Assisted Laparoscopy.