Permanent deformation is known as one of the most critical distresses observed in asphalt pavements, and is known to depend on asphalt binder, aggregates and voids, which are the components of asphalt mixture. This study is an effort to investigate the effect of asphalt binder modification type and aggregate structure on rutting resistance through experimentation and finite-element simulation. Asphalt binders with different levels of non-recoverable compliance (Jnr) and elastic recovery (%R) based on the Multiple Stress Creep Recovery test were selected. Asphalt mixtures with different aggregate gradations were prepared for testing and image processing for numerical simulation. A flow number test was conducted on the mixture samples to obtain the asphalt mixture rutting performance. A two-dimensional (2D) image analysis was also conducted on the mixture samples to characterise their internal aggregate structures. In addition, a recently developed image-based multi-scale finite-element model was used to predict the permanent deformation of the asphalt mixtures. In this approach, four interconnected scales were modelled, namely asphalt binder, mastic, mortar and mixture scales through homogenisation and upscaling techniques to transfer the material properties from lower scale to higher scale. A novel approach for taking into account contact mechanics between aggregates as function of proximity is used at all scales. The simulation results show that by changing the aggregate gradation (packing), and the viscous component of asphalt binder (Jnr), the rutting resistance of asphalt mixtures can be significantly improved. However in comparison the elastic component of asphalt binder (% recovery) is not found to be a significant factor especially in a well-packed aggregate structure. The results raise questions about the focus by many agencies on the requirements of elastic recovery of binders as a tool to select modified binders for better high-temperature performance.