Is there anything that doesn't mix with water OR oil?
In short, yes, many highly fluorinated liquids are miscible with neither water nor organic solvents.
If you look at my answer to this question, you can see that I made a mixture of water, hexanes, and a fluorinated solvent called HT-110.
Likewise, the HT-110 will not dissolve most compounds, apart from fluorinated ones like Teflon AF.
Many compounds will have appreciable solubility in more than one layer and if they are in contact, there can be transfer between them.

Polarity plays a large role in whether two liquids are miscible.
Basically, for two liquids to mix, it must be energetically favourable to disrupt the intermolecular forces of the individual liquids in favour of the new intermolecular forces between the components of the mixture.
For your example of water and oil, water on its own has significant hydrogen bonding, but since the oil can't participate in hydrogen bonding, there would be a lot less hydrogen bonding in a homogeneous mixture of oil and water—the oil molecules essentially block water molecules from accessing each other.
Because the mixture would greatly reduce the intermolecular forces holding the liquids together, it is more energetically favourable to remain separate.
In contrast, a mixture of water and ethanol is energetically favourable because ethanol can hydrogen bond and is similarly polar, so the disrupted water-water intermolecular interactions are replaced by similar water-ethanol interactions.

In a little more detail, a mixture is favourable when its Gibbs energy of mixing is negative: $$\Delta G_{mix} = \Delta H_{mix}-T\Delta S_{mix}$$

In an ideal mixture of two components A and B, A-B interactions are the same strength as A-A and B-B, so the enthalpy of mixing, ∆H, is 0.
If A and B have similar properties, A-B interactions tend to be similar and ∆H will be close to 0.
If A-B interactions are weaker (e.g. water and oil), ∆H will be positive.
∆S is almost always positive for common mixtures because each component has a larger volume to spread into than if the components were separate. (the exception is when A-B interactions are much stronger than A-A and B-B interactions which prevents A and B from moving randomly through the volume)
This means that mixtures can still form if the enthalpy of mixing is positive, if the entropic contribution can overcome it.
This explains why some mixtures are possible at elevated temperatures but not room temperature.