What is the white substance left behind after boiling down water
These are minerals that are naturally present in "tap" water. Chemically, they are most likely a mixture of calcium carbonate and magnesium carbonate, both of which are only very lightly soluble in water.

As you boil away the water, these dissolved calcium/magnesium carbonates remain behind and their concentrations eventually become greater than their respective solubility limits. At this point they begin to precipitate out of the solution, forming the 'scale' you see on the pot.  Interestingly, while most salts become more soluble at higher temperatures, calcium carbonate is anomalous in that its solubility decreases as temperature increases (see the accepted answer here), which accelerates the precipitation process.

The easiest way to get rid of them is to add a cup of white vinegar and let it sit in the pot. You might need to agitate the scale from time to time. You can boil the vinegar in the pot if you want, but it will stink! Over time, the vinegar will dissolve the precipitate, and you will see carbon dioxide bubbles forming. You can then pour out the vinegar and flush the pot with lots of water.