It's due to plant proteins in the pollen that cross-react with plant proteins in certain fruits and vegetables. This is why, when someone allergic to ragweed eats a banana, their immune system sees the two as related. This leads to an allergic reaction in your mouth, with itching or swelling as the most common symptom.