Oxidation number of phosphorus in phosphine
Here is a way to get the answer. If you place a reagent with oxidation number of H = -1 (hydride) it would produce dihydrogen gas. For example: $\ce{KH + H2O-&gt; KOH + H2}$.

If you place a molecule with oxidation number of $\ce{H} = +1$ dihydrogen gas is NOT formed. $\ce{NH3 + H2O -&gt; NH4+ + OH-}$. Phosphine doesn't form dihydrogen gas on contact with water so $\ce{H}$ is in oxidation state +1 and $\ce{P}$ is in oxidation state -3. Reaction with strong acids leads to $\ce{PH4+}$ which has similar structure to the ammonium ion. For these reasons even for $\ce{As}$ in $\ce{AsH3}$ the oxidation state of $\ce{As = -3}$.

The problem with electronegativity is that we are trying to assign a single number, while we actually need a function. We take average of electronegativity of $\ce{H}$ in $\ce{HF, NaH, H2O, CH4, NH4+}$ and come up with a single number. This works well if you need to guesstimate the behavior, but it is not accurate. It doesn't even account for softness/hardness of a partner. 

Analysis "will $\ce{PH3 + H2O}$ form $\ce{H2}$?" (no) vs "will $\ce{PH3 + P2O3}$ form $\ce{P + H2O}$?" (yes) is more accurate way to assign formal charges.