In recent years, image encryption algorithms have been developed rapidly in order to ensure the security of image transmission. With the assistance of our previous work, this paper proposes a novel chaotic image encryption algorithm based on self-adaptive model and feedback mechanism to enhance the security and improve the efficiency. Different from other existing methods where the permutation is performed by the self-adaptive model, the initial values of iteration are generated in a novel way to make the distribution of initial values more uniform. Unlike the other schemes which is on the strength of the feedback mechanism in the stage of diffusion, the piecewise linear chaotic map is first introduced to produce the intermediate values for the sake of resisting the differential attack. The security and efficiency analysis has been performed. We measure our scheme through comprehensive simulations, considering key sensitivity, key space, encryption speed, and resistance to common attacks, especially differential attack.