I was sharing an iCloud account, would he know about my photos?
If you deleted the photos, then went into the Recently Deleted folder and permanently deleted them from your iPhone, followed by disabling iCloud Photo Library as you stated, then you should be fine.