Why does mouldy food make you sick?
One of the most common toxins in mouldy food/bread is aflatoxin. 

The exposure to high amount of aflatoxin can lead to acute aflatoxicosis with fulminant hepatic failure and even death.

While mild exposure to aflatoxin could be asymptomatic, it is better not to take a chance considering possible complications (citation from the link above): 


  Aflatoxins produce acute necrosis, cirrhosis, and carcinoma of the
  liver in a number of animal species; no animal species is resistant to
  the acute toxic effects of aflatoxins; hence it is logical to assume
  that humans may be similarly affected. A wide variation in LD50 values
  has been obtained in animal species tested with single doses of
  aflatoxins. For most species, the LD50 value ranges from 0.5 to 10
  mg/kg body weight. Animal species respond differently in their
  susceptibility to the chronic and acute toxicity of aflatoxins. The
  toxicity can be influenced by environmental factors, exposure level,
  and duration of exposure, age, health, and nutritional status of diet.
  Aflatoxin B1 is a very potent carcinogen in many species, including
  nonhuman primates, birds, fish, and rodents. In each species, the
  liver is the primary target organ of acute injury. Metabolism plays a
  major role in determining the toxicity of aflatoxin B1; studies show
  that this aflatoxion requires metabolic activation to exert its
  carcinogenic effect, and these effects can be modified by induction or
  inhibition of the mixed function oxidase system.