How do i determine max voltage for a custom BLDC
Motors use technology with highly technical concepts. 

But the basics are heat rise comes from thermal resistance*Pd and ,
Pd = I^2R losses and other losses,
while Voltage and number of pole pairs per Rev determines the RPM,
 with no load often rated as kRPM/V.    

Torque is max. with current at start with rated V at 100%  and has zero torque at no-load full speed. Max rated power is often measured around 82% no-load RPM for linear loads but depends on load, water cooling etc.

I think you should study 1st , buy and compare specs with measured results before you try to design so you can understand what it takes to design.