You're right, here's a quick calculation: Say you bench press a 100kg mass and it travels 1m (apparently you have really long arms). Lifting that weight 1m expends (100kg)(1m)(~10m/s²) = 1000J. Converting to calories and then kilocalories (food calories), this gives you something one the order of 1kcal. Bench pressing 100kg burns about 1kcal.