Boost Regulator LDO bucking voltage
This is going to hurt, but I am guessing you forgot to pull the EN pin high.  (or you did pull it high but there is a broken connection somewhere)

Here's the chip block diagram from the datasheet:


With the EN pin low, the boost converter is disabled but the LDO portion not affected (at least, if the block diagram is honest). Then DC conduction through the inductor and the switching diode (between the SW and BSTOUT pins) will produce a voltage on BSTOUT about 0.5 to 0.7 V below the input voltage (like you saw). This would normally put the LDO into dropout, but when you get this voltage sufficiently above 5 V, the LDO will work normally. Again, this is exactly the behavior you observed.

If you just left the EN pin floating, there is nothing in the datasheet that says which way the pin will default. So you will need to actively pull it high to enable the chip.