LT3663 regulating low
You're probably running into the LT3663's maximum duty-cycle limit as well as other internal losses.
The datasheet specifies the duty-cycle limit as: typical 92%, minimum 80%.
So if you hope for the typical value and assume 100% efficiency everywhere else, the best you could get is 13.8V
But since efficiency is never 100%, you have a diode drop, and you can't bet on getting the typical value, you shouldn't expect as much as 13.8V out.
Taking into account a lower max duty-cycle, some diode drop and other losses (transistor switch, inductor, current-sense, etc), your 10.4V is not surprising.

EDIT:
I'm surprised somone didn't spot this sooner, but you should really have a look at the top of page 2 of the datasheet - particularly where it lists the Absolute Maximum Ratings for the Vout pin (amongst others) as 6V !!!
That IC is apparently not intended to be used for anything much above 5V output - even the adjustable version.
A check on Linear's product selector grid, with VOut Max column enabled shows 5.5V as the maximum output voltage.
None of the application circuits show anything above 5V either.
I hope for your sake that LT has a pin-compatible substitute ....