How is 6-player Castle Panic (with wizard tower) supposed to work?
The point of reducing the number of cards is to keep the players from having too many resources at their disposal. If you have 5 players with 5 cards each, that's 25 cards for the players to work with. If you have 6 players with 4 cards each, that's 24. It's about the same number of resources in total.

Being able to trade twice is a huge advantage in resource management; I suspect the lower number of cards is actually to counteract the strength of being able to trade two cards instead of one. The extra trading allows the players to more precisely control who has access to which resources at a given time.

If the players had that amount of control and the extra 5 cards, the game would be too easy. The game is designed to be a challenge.