Why isn't phosphorus or nitrogen a limiting nutrient for animals?
Phosphorus

Your suggestion that if we are meeting our calorific requirement we will be getting enough is true for phosphorus.
Most foods contain lots of phosphorus. The maximum dietary requirement occurs during adolescent growth, estimated at 1250 mg per day. Assuming a calorie intake of 2500 kcal we can calculate a 2500 kcal equivalent phosphorus content for various foods:

skimmed milk contains 7,400 mg phosphorus per 2500 kcal

roasted chicken breast contains 7,500 mg phosphorus per 2500 kcal

cooked white rice contains 3840 mg per 2500 kcal

(Calculations are based upon values obtained via this site.)

Nitrogen

Our requirement for nitrogen is met by our protein intake: inadequate protein intake manifests as kwashiorkor which is essentially due to a dietary deficiency of essential amino acids. In other words, the only way to achieve a nitrogen-deficient diet is to not eat protein, and this would not be alleviated by any inorganic source of nitrogen, even if we could consume enough of such a N source.