Can pumps fail due to age?
Can a pump that is not being actively used fail?

On the one hand - nothing lasts forever.
On the other hand - you don't need your cache of pumps to last forever. You only need your cache to last:

Until you stop riding
Until you find something else that works as well or better

So, let's say best case scenario, you need your cache of pumps to last 40 years (just picking a number).
Let's also say that you have enough pumps in your cache to last the 40 years (or whatever number you pick) due to normal wear. For example, if one pump wears out every five years you have 8 pumps.
Your question is - will that last pump still work 35 years from now when you pick it up having never been used before?
My bet (in the absence of actual data, which would take longer than I'm willing to wait) is yes, that 8th pump will still work.
Here's why
Your cache of pumps have two things going for them:

The seals are covered with a lubricant that protects them from oxidation
The seals and the lubricant are in a sealed environment

It may be that your real problem 35 years from now will be the oxidation of the lubricant rather than the oxidation of the seal.

Oxidation is the most predominant reaction of a lubricant in service. It is responsible for numerous lubricant problems - including viscosity increase, varnish, sludge and sediment formation, …
Oxidation - The Lubricant's Nemesis

You can increase the odds of in your favor by:

Storing the pumps in a temperature stable environment, as you describe in your question.
Re-lubricating the unused pumps after some period of time (check them at 5 years and set a schedule based on your observation) would assure that the lubrication does not break down and gum up the pump. Re-lubricate the pump with a lubricant that is similar as possible to what you find in the pump.