Clusters with the CPU-MIC heterogeneous architecture are becoming more popular in recent years. However, it is not easy to achieve good performance on such machines. The key challenge has been the asymmetry within clusters, arising from different kinds of execution units as well as different communication latencies. To improve the performance of large-scale structural analysis on CPU-MIC heterogeneous clusters, a multi-layer and multi-grain collaborative parallel computing approach is proposed in the paper. The proposed method combines the parallel algorithm and the hardware architecture of CPU-MIC heterogeneous clusters together. Through mapping computing tasks to various hardware layers, it both resolves the load balance problem between CPU and MIC devices and significantly reduces the communication overheads of the system. Numerical experiments conducted on Tianhe-2 supercomputer show that the proposed method obtained better performance compared with the traditional approach. Scalability investigation showed that the proposed method had good scalability with respect to problem sizes. The findings of this paper are of help to the parallel porting and performance optimization of other applications on CPU-MIC heterogeneous clusters.