N Mosfet not turned off fully
Q1 will leak current to the output and, if you look at its data sheet it might be as high as 10 uA (\$I_{DSS}\$). Additionally, U4 is also a contributor - if you read its data sheet, the leakage is also going to be several uA.

With insufficient load on the output to drag the output down when disabled, you will see a voltage. How much is dependent on your load.

You should also be aware that the absolute maximum input voltage for the LTC4411 is 6 volts and, you are likely pushing it beyond reasonable expectations with your 6 volt input supply.