Why is there an altitude limitation on operating the slats/flaps on the A320?
Why would you want to deploy high lift devices at any juncture except for departure and arrival?
Holding is the predominant scenario.
Above 20,000ft ICAO holding speeds are higher thus allowing clean configuration.
Above 20,000ft, at slower speeds, the likelihood of encountering icing on the wings when moisture exists is also a very real consideration (upto about 28,000ft) and ice accretion is not what you want with high lift devices hanging out.
Extending slats/flaps in typical cruise or when we are flying in the Mach regime of flight, say about 29,000ft and above, is something that would cause much grief to the average Airliner, a possible jet upset scenario.
If other reasons already cited such as aircraft certification/cost of certification are a factor - most operators would not need a change from the 20,000ft figure.