Why does potential energy relate to exothermic reactions in this way?
Energy is a useful concept for rationalising the capability of doing work or heating; other than that, it is purely a theoretical concept.
As Feynman brilliantly pointed out,


  It is important to realise that in physics today, we have no knowledge of what energy is.
  We do not have a picture that energy comes in little blobs of a definite amount.
  It is not that way.
  However, there are formulas for calculating some numerical quantity, and when we add it all together it gives “28”—always the same number.
  It is an abstract thing in that it does not tell us the mechanism or the reasons for the various formulas.


But let's get back to something answerable.

Energy is (a) conserved and (b) convertible into different forms.
If you got any system in a state $q_1$ with potential energy $U(q_1)$ and it changed into state $q_2$ with potential energy $U(q_2)$, then if $\Delta U = U(q_2) - U(q_1) &lt; 0 $, energy has been lost to the surroundings.
This loss could be lots of things, most of the time it is just heat (kinetic motion).
This loss, in the big picture, is just a transfer or conversion.



Let's get to a very simple example, hydrogen combustion:

$$\ce{2 H2 + O2 -&gt; 2 H2O}$$

Since energy can't be produced, we say this reaction releases large amounts of heat.
But, where does it come from?

All that the equation above tells us is the atomic rearrangement that has been taken place: in the left hand side we have two $\ce{H-H}$ bonds and a single $\ce{O=O}$ bond, in the right hand side we got four $\ce{O-H}$ bonds.
So quantifying the bond energies in each side gives us:

     |  Average bond energy (kJ/mol)
-----|-------------------------------
     |        |   Left    |   Right
Bond | Single | hand side | hand side
-----|--------|-----------|----------
H-H  |  -432  | 2 x -432  |    0
O=O  |  -495  |  -495     |    0
O-H  |  -467  |    0      | 4 x -467
--------------|-----------|----------
   Total      |   -1359   |  -1868


This means we get $-1868 - (-1359) = -509$ kJ for two moles of $\ce{H2}$, i.e., $-254.5$ kJ/mol of heat is released during the combustion of hydrogen.

Wikipedia says the correct value is $-242$ kJ/mol, so our calculation has an error of around 5%.



So heat energy has been released and it came from bond breaking and forming.
The energy stored in a bond could be called latent or potential, if you will.

It all boils down to conservation and convertibility.