Gravel under vinyl fence post in clay soil?
The clay, as you state, will trap water with nowhere to go as well.  By digging the whole deeper and lining the bottom with gravel, what you do is provide a place for extra water to drain to until it can dissipate/drain.  

Without that extra space for water to drain, it can fill up the hole above the frost line and cause lateral displacement.