The phase retrieval (PR) problem of recovering an image from its Fourier magnitudes is an important issue. Several PR algorithms have been proposed to address this problem. Recent efforts of exploiting sparsity were developed to improve the performance of PR algorithms, such as the reconstruction quality, robustness to noise, and convergence behavior. In this paper, we propose a novel sparsity-based algorithm, which can adaptively learn an orthogonal sparsifying transform, and reconstruct the image simultaneously from the Fourier magnitudes. However, the estimated images at the early iterations are extremely bad. Training samples from these images cannot provide much useful information for sparsifying transform learning. To avoid unnecessary updating, an orthogonal sparsifying transform learning method based on transfer learning is proposed. Through transfer learning, we transfer the fixed sparsifying transform to an adaptive one. We apply this new sparsifying transform learning method to PR, and exploit the alternating directions method of multipliers (ADMM) technique to solve the formulated problem. Since the learnt sparsifying transform is adaptive to data, it favors better sparsity. Using this learnt sparsifying transform for image reconstruction can improve the reconstruction quality at low oversampling ratios. Experimental results show that the proposed PR algorithm can improve nearly 6 dB compared with the recently proposed PR-TIHP-l(1) algorithm in terms of the average PSNR (Peak Signal to Noise Ratio) at oversampling ratio 2.47, 2.53, 2.59. Moreover, our algorithm is robust to noise and has better convergence behavior heuristically. (C) 2016 Elsevier Inc. All rights reserved.