Is there a candidate for a natural problem in $P/poly - P$?
I do not know if this satisfies your requirements, but Bill Gasarch’s blog post in July 2010 asks about languages in SPARSE∩NP which are not thought to be in P, giving an example from Ramsey Theory.  Any such languages belong to (P/poly)∩NP.

Related to this, For any language L∈NP, the language TL={1n: L contains some string of length n} is in TALLY∩NP ⊆ SPARSE∩NP ⊆ (P/poly)∩NP.  Depending on the choice of the language L, TL may not have an obvious reason to belong to P.