Not able to reach minimum phase using Hilbert transform
More taps. You don't have anywhere near enough taps for a filter that steep. Start large with 8192 or so cut to desired accuracy, if needed
Due to the low number of tabs you are seeing the effect of "circular" hilbert transform. See for example: http://andrewduncan.net/air/
How do you know you have zeros outside the unit circle? Calculating the roots of a high order polynomial is a numerically difficult problem and using a Matlab function like "roots()" will given you often inaccurate or plainly wrong results.