Can I use chapter specific characters in a custom space marine chapter?
Yes, since your chapter has the Adeptus Astartes keyword, units with that or imperial keyword can be included BUT these HQ units will function as written, their abilities won't change to reference your custom chapter.

Using Lysander as an example, Icon of Obstinacy will ONLY affect Imperial Fists units, For the Khan! from Kor'sarro will ONLY affect White Scars. When talking about casual play, house rules can change this, you can agree with your opponent to allow these abilities to work on your custom faction, but not in tournament.