Most people who are allergic to coffee are allergic to the caffeine in it. Caffeine is a naturally occurring drug classified as a stimulant. It’s found in many different things including tea and coffee and is a natural byproduct of various plants.