We have benchmarked protein-compound binding models using two molecular representations for compounds and two prediction algorithms under four cross-validation strategies. One of our main findings is the existence of a database-specific bias that challenges the generalization of machine learning models between databases. Performance estimates derived from classical random cross-validation are overly optimistic, despite being widely used in literature. Instead, we recommend a clustering-based cross-validation since it addresses the chemical series bias while providing more reliable performance estimates. For molecular representation, fingerprints have led to better models than the SMILES identification string.