In this work, we present a comparative study on the performance of different small-molecule force fields at reproducing conformational ensembles of semi-peptidic and non-peptidic macrocycles in solution, as judged by the comparison to experimental upper distance bounds from NOESY experiments. The performance of four different force fields (GAFF2, OPLS/AA, OpenFF 2, and XFF) was evaluated using a set of 11 publicly available and six in-house macrocycles. Overall, we find that OpenFF 2.0 performs well. However, we also observe that the same force field can perform differently on the same compound depending on the solvent. This implies that the solvent parameters are crucial to obtain the correct balance between relatively open and more folded (closed) conformations.