What happened between Kaoru-tan's father and Hajime-kun?
I guess I should post this as answer. As a father, you would want your daughter to have a good husband who has a proper job so she won't be having a hard time to word in order to sustain their needs. So Kaoru's father thinks that Hajime is not worthy of being her husband and that makes him very disappointed of Hajime.