Understanding the law of supply
As you noted correctly, it has something to do with the costs.  An important point here is a cost of producing an additional unit, and not average cost.


Let me give the following example.  Suppose that you are an owner of a bakery, and you have a single employee who works $8$ hours and produces $8$ breads a day ($1$ bread an hour). Assume you have to pay your employee $\$1$ an hour for the first $8$ hours and $\$1.50$ (overtime) for each additional hour (after $8$ hours), and you sell each bread at $\$1.20$.  

One day you think to yourself: why not to produce and sell $9$ breads (you come to this thought by the same reasoning as in your question)?  Then, you realize that in order to produce the $9$th bread, you need to keep your employee working for an additional hour and have to pay overtime, namely $\$1.5$ for the $9$th hour.  However, you will still sell the $9$th bread for $\$1.20$, which means you will lose $\$0.30$ from producing and selling the $9$th bread.

Suppose that the price of bread has increased to $\$1.60$.  Now, you surely want to keep your employee for the $9$th hour (or even more), since for each additional bread produced (after $8$ breads), you will make $\$0.10$ profit per bread.  So, you can see that an increase in price induced you to supply more.