In this study, we introduce a new framework for optimizing and learning force fields to match experimental data while minimizing the derivation from the original ensemble. The framework is based on two techniques: ensemble reweighting and potential contrasting. The ensemble reweighting method is used to assign weights to conformations generated with an existing force field to match experimental data while minimizing the perturbation to the original ensemble. The potential contrasting method is utilized to optimize and learn force fields to reproduce the reweighted ensemble. By reproducing the reweighted ensemble, force fields are optimized to align with experiments while having minimal perturbation to the original ensemble. We demonstrate the effectiveness of the proposed framework by optimizing a CG force field of IDPs using experimental R g data. We show that the framework is capable of both refining existing force field parameters and learning new functional expressions to match experimental data.