Caritoux was born in Carpentras in Provence at the foot of Mont Ventoux. He first started cycle racing as a junior with the local club in Carpentras, he developed into a strong climber riding in the hilly Vaucluse region and often using Mont Ventoux in his training rides. As an amateur Caritoux won the 1982 edition of the Tour du Vaucluse Open beating future Tour de France winner Laurent Fignon into second place, he clinched that victory by winning the 15 kilometre mountain time trial up Mont Ventoux between Bédoin and Chalet Reynard. That win persuaded Jean de Gribaldy manager of the Sem-France Loire professional squad to sign Caritoux for the 1983 season.